The Kubermatic API lives inside the master cluster and therefore speaks to it via in-cluster communication, using the
kubermatic
service account. Communication with the seed clusters happens by providing the API with a kubeconfig that
has the required contexts and credentials for each seed cluster. The name of the context within the kubeconfig needs to
match an entry within the datacenters.yaml
(see below).
Also make sure your kubeconfig contains static, long-lived credentials. Some cloud providers use custom authentication
providers (like GKE using gcloud
and EKS using aws-iam-authenticator
). Those will not work in Kubermatic’s usecase
because the required tools are not installed.
In the example above, we define two possible seed identifiers for the datacenters.yml
: seed-1
and seed-2
.
There are 2 types of datacenters:
Both are defined in a file named datacenters.yaml
:
values.yaml
Installation of Kubermatic uses the Kubermatic Installer, which is
essentially a Kubernetes job with Helm and the required charts to install Kubermatic and its
associated resources. Customization of the cluster configuration is done using a cluster-specific values.yaml
, stored
as a secret within the cluster.
As a reference you can check out values.example.yaml.
For the kubermatic configuration you need to add base64
encoded configuration of datacenter.yaml
and kubeconfig
to
the values.yaml
file. You can do this with following command:
base64 kubeconfig | tr -d '\n'
Access to Kibana, Grafana and all other 3rd-party services included with Kubermatic is secured by running them behind Keycloak-Proxy and using Dex as the authentication provider. Dex can then be configured to use external authentication sources like GitHub’s or Google’s OAuth endpoint, LDAP or OpenID Connect. Kubermatic itself makes use of Dex as well, but since it supports OAuth natively does not make use of Keycloak-Proxy.
For this to work you have to configure both Dex and Keycloak-Proxy (called “IAP”, Identity-Aware Proxy) in your
values.yaml
.
oauth
Helm chart.For each service that is supposed to use Dex as an authentication provider, configure a client
. The callback URL is
called after authentication has been completed and must point to https://<domain>/oauth/callback
. Remember that this
will point to Keycloak and is therefore independent of the actual underlying application. Generate a secure random
secret for each client as well.
A sample configuration for Prometheus could look like this:
dex:
clients:
- id: prometheus # a unique identifier
name: Prometheus
secret: very-very-very-secret # clientSecret
# list of allowed redirect URIs
# (which one is used is determined by what Keycloak-Proxy decides)
RedirectURIs:
- https://kubermatic.initech.com/oauth/callback
Each service should have its own credentials. See the dex
section in the example
values.yaml.
Now that you have setup your “virtual OAuth provider” in the form of Dex, you need to configure Keycloak-Proxy to sit in
front of the 3rd-party services and use it for authentication. For each client that we configured in Dex, add a
deployment
to the IAP configuration. Use the client’s secret as the client_secret
and generate another random,
secure encryption key to encrypt the client state with (which is then stored as a cookie in the user’s browser).
A sample deployment for Prometheus could look like this:
iap:
deployments:
prometheus:
name: prometheus # will be used to create kubernetes Deployment object
# OAuth configuration from Dex
client_id: prometheus
client_secret: very-very-very-secret
# encryption key for cookie storage
encryption_key: ultra-secret-random-value
## see https://github.com/gambol99/keycloak-proxy#configuration
## example configuration allowing access only to the mygroup from
## mygithuborg organization
config:
scopes:
- "groups"
resources:
- uri: "/*"
groups:
- "mygithuborg:mygroup"
upstream_service: prometheus.monitoring.svc.cluster.local
upstream_port: 9999
ingress:
host: prometheus.kubermatic.initech.com
See the iap
section in the example
values.yaml for more
information.
Clone the installer repository to your disk and make sure to
checkout the appropriate release branch (release/vX.Y
). The latest stable release is already the default branch,
so in most cases there should be no need to switch. Alternatively you can also download a ZIP version from GitHub.
git clone https://github.com/kubermatic/kubermatic-installer
cd kubermatic-installer
A storage class with the name kubermatic-fast
needs to exist within the cluster. Also, make sure to either have a
default storage class defined or configure Minio in your values.yaml
to use a specific one:
minio:
storageClass: hdd-disk
Store the above YAML snippet in a file and then apply it using kubectl
:
kubectl apply -f storageclass.yaml
Before applying the Helm charts, ensure that Kubermatic’s CRDs are installed in your cluster by applying the provided manifests:
kubectl apply -f charts/kubermatic/crd
Install Helm on you local system and setup Tiller within the cluster.
Create a service account for Tiller and bind it to the cluster-admin
role:
kubectl create namespace kubermatic
kubectl create serviceaccount -n kubermatic tiller-sa
kubectl create clusterrolebinding tiller-cluster-role --clusterrole=cluster-admin --serviceaccount=kubermatic:tiller-sa
Afterwards install Tiller with the correct service account:
helm --service-account tiller-sa --tiller-namespace kubermatic init
Now you’re ready to deploy Kubermatic and its charts. It’s generally advisable to postpone installing the final certs
chart until you acquired LoadBalancer IPs/hostnames and can update your DNS zone to point to your new installation. This
ensure that the cert-manager
can quickly acquire TLS certificates instead of running into DNS issues.
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace nginx-ingress-controller nginx-ingress-controller charts/nginx-ingress-controller/
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace cert-manager cert-manager charts/cert-manager/
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace oauth oauth charts/oauth/
# Used for storing etcd snapshots
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace minio minio charts/minio/
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace kubermatic kubermatic charts/kubermatic/
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace iap iap charts/iap/
# When running on a cloud Provider like GCP, AWS or Azure with LB support also install the nodeport-proxy
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace nodeport-proxy nodeport-proxy charts/nodeport-proxy/
# For logging stack, ensure that all charts are deployed within the logging namespace
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace logging elasticsearch charts/logging/elasticsearch/
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace logging fluentbit charts/logging/fluentbit/
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace logging kibana charts/logging/kibana/
# For monitoring stack
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace monitoring prometheus charts/monitoring/prometheus/
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace monitoring node-exporter charts/monitoring/node-exporter/
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace monitoring kube-state-metrics charts/monitoring/kube-state-metrics/
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace monitoring grafana charts/monitoring/grafana/
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace monitoring alertmanager charts/monitoring/alertmanager/
After all charts have been deployed, update your DNS accordingly. See the last section on this page for more details. Once that is done, wait a bit and then install the final Helm chart:
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace default certs charts/certs/
We run an individual cronjob every 20 minutes for each customer cluster to backup the etcd-ring. Snapshots will be
stored by default to an internal S3 bucket provided by minio, though this can be changed by modifying the
storeContainer
& cleanupContainer
in the values.yaml
to your needs.
The cronjobs will be executed in the kube-system
namespace. Therefore if a container needs credentials, a secret must
be created in the kube-system namespace.
The workflow:
storeContainer
takes the snapshot and stores it somewhereThe storeContainer
will be executed on each backup process after a snapshot has been created and stored on a shared
volume accessible by the container. By default only the last 20 revisions will be kept. Older snapshots will be deleted.
By default the container will store the snapshot to minio.
The cleanupContainer
will delete all snapshots in S3 after a cluster has been deleted.
If the default container will be used, a secret in the kube-system
namespace must be created:
apiVersion: v1
kind: Secret
type: Opaque
metadata:
name: s3-credentials
namespace: kube-system
data:
ACCESS_KEY_ID: "SOME_BASE64_ENCODED_ACCESS_KEY"
SECRET_ACCESS_KEY: "SOME_BASE64_ENCODED_SECRET_KEY"
Kubermatic needs to have at least 2 DNS entries set.
The frontend of Kubermatic needs a single, simple DNS entry. Let’s assume it is being installed to serve
kubermatic.initech.com
. For the 3rd-party services like Prometheus or Grafana, you will also want to create a wildcard
DNS record *.kubermatic.initech.com
pointing to the same IP/hostname.
When running on a cloud provider which supports services of type LoadBalancer, the nginx-ingress chart should be configured to create such a service. The load balancer’s IP can then be fetched via:
kubectl -n nginx-ingress-controller get service nginx-ingress-controller -o wide
#NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
#nginx-ingress-controller LoadBalancer 10.47.242.69 35.198.146.37 80:30822/TCP,443:31412/TCP 160d app=ingress-nginx
The EXTERNAL-IP
field shows the correct IP used for the DNS entry. Depending on your provider, this might also be a
hostname, in which case you should set a CNAME record instead of an A record.
Without a LoadBalancer nginx will run as DaemonSet & allocate 2 ports on the host (80 & 443). Configuration of nginx happens via the values.yaml. The DNS entry needs to be configured to point to one or more of the cluster nodes.
For each seed cluster a single wildcard DNS entry must be configured. All apiservers of all customer clusters are being exposed via either NodePorts or a single LoadBalancer.
The domain will be based on the name of the seed-cluster as defined in the datacenters.yaml and the domain under which the frontend is available.
For example, when the base domain is kubermatic.initech.com
and a seed cluster in your datacenters.yaml
is called
europe-west1
, then
europe-west1.kubermatic.initech.com
*.europe-west1.kubermatic.initech.com
A customer cluster created in this seed cluster would get the domain [cluster ID].europe-west1.kubermatic.example.com
.
Get the IP from the nodeport-lb
service:
kubectl -n nodeport-proxy get service nodeport-lb
#NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
#nodeport-lb LoadBalancer 10.47.242.236 35.198.93.90 32493:30195/TCP,31434:30739/TCP,30825:32503/TCP,30659:30324/TCP 93d
Take one or more of the seed cluster worker nodes IPs.