Clone the installer repository to your disk and make sure to
checkout the appropriate release branch (release/vX.Y). The latest stable release is already the default branch,
so in most cases there should be no need to switch. Alternatively you can also download a ZIP version from GitHub.
git clone https://github.com/kubermatic/kubermatic-installer
cd kubermatic-installer
The Kubermatic API lives inside the master cluster and therefore speaks to it via in-cluster communication, using the
kubermatic service account. Communication with the seed clusters happens by providing the API with a kubeconfig that
has the required contexts and credentials for each seed cluster. The name of the context within the kubeconfig needs to
match an entry within the datacenters.yaml (see below) and needs to be a valid DNS part. Kubermatic uses DNS records
like *.[seed-name].[main-domain], so context names like user@seed1 would not work. Make sure the seed names are
alphanumeric.
Also make sure your kubeconfig contains static, long-lived credentials. Some cloud providers use custom authentication
providers (like GKE using gcloud and EKS using aws-iam-authenticator). Those will not work in Kubermatic’s usecase
because the required tools are not installed. You can use the kubeconfig-serviceaccounts.sh script from the
kubermatic-installer repository to automatically create proper service accounts inside each cluster and update the
kubeconfig file.
In the example above, we define two possible seed identifiers for the datacenters.yml: seed-1 and seed-2.
There are 2 types of datacenters:
Both are defined in a file named datacenters.yaml:
values.yamlInstallation of Kubermatic uses the Kubermatic Installer, which is
essentially a Kubernetes job with Helm and the required charts to install Kubermatic and its
associated resources. Customization of the cluster configuration is done using a cluster-specific values.yaml, stored
as a secret within the cluster.
As a reference you can check out values.example.yaml.
For the kubermatic configuration you need to add base64 encoded configuration of datacenter.yaml and kubeconfig to
the values.yaml file. You can do this with following command:
base64 kubeconfig | tr -d '\n'
Access to Kibana, Grafana and all other system services included with Kubermatic is secured by running them behind Keycloak-Gatekeeper and using Dex as the authentication provider. Dex can then be configured to use external authentication sources like GitHub’s or Google’s OAuth endpoint, LDAP or OpenID Connect. Kubermatic itself makes use of Dex as well, but since it supports OAuth natively does not make use of Keycloak-Gatekeeper.
For this to work you have to configure both Dex and Keycloak-Gatekeeper (called “IAP”, Identity-Aware Proxy) in your
values.yaml.
It is still possible to access the system services by using kubectl port-forward and thereby circumventing the
authentication entirely.
Please note that despite its name, Dex is part of the oauth Helm chart.
For each service that is supposed to use Dex as an authentication provider, configure a client. The callback URL is
called after authentication has been completed and must point to https://<domain>/oauth/callback. Remember that this
will point to Keycloak-Gatekeeper and is therefore independent of the actual underlying application. Generate a secure
random secret for each client as well.
A sample configuration for Prometheus could look like this:
dex:
clients:
- id: prometheus # a unique identifier
name: Prometheus
secret: very-very-very-secret # clientSecret
# list of allowed redirect URIs
# (which one is used is determined by what Keycloak-Proxy decides)
RedirectURIs:
- https://kubermatic.initech.com/oauth/callback
Each service should have its own credentials. See the dex section in the example
values.yaml.
Now that you have setup your “virtual OAuth provider” in the form of Dex, you need to configure Keycloak-Gatekeeper
to sit in front of the system services and use it for authentication. For each client that we configured in Dex,
add a deployment to the IAP configuration. Use the client’s secret (from Dex) as the client_secret and generate
another random, secure encryption key to encrypt the client state with (which is then stored as a cookie in the user’s
browser).
A sample deployment for Prometheus could look like this:
iap:
deployments:
prometheus:
name: prometheus # will be used to create kubernetes Deployment object
# OAuth configuration from Dex
client_id: prometheus
client_secret: very-very-very-secret
# encryption key for cookie storage
encryption_key: ultra-secret-random-value
## see https://github.com/gambol99/keycloak-proxy#configuration
## example configuration allowing access only to the mygroup from
## mygithuborg organization
config:
scopes:
- "groups"
resources:
- uri: "/*"
groups:
- "mygithuborg:mygroup"
upstream_service: prometheus.monitoring.svc.cluster.local
upstream_port: 9999
ingress:
host: prometheus.kubermatic.initech.com
See the iap section in the example
values.yaml for more
information.
A storage class with the name kubermatic-fast needs to exist within the cluster. Also, make sure to either have a
default storage class defined or configure Minio in your values.yaml to use a specific one:
minio:
storageClass: hdd-disk
An explicit storage class to AWS using SSDs can look like this:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: kubermatic-fast
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
Store the above YAML snippet in a file and then apply it using kubectl:
kubectl apply -f aws-storageclass.yaml
Please consult the Kubernetes documentation for more information about the possible parameters for your storage backend.
Before applying the Helm charts, ensure that Kubermatic’s CRDs are installed in your cluster by applying the provided manifests:
kubectl apply -f charts/kubermatic/crd
To properly install the cert-manager, you need to manually label its namespace or else the included webhook will not function correctly and your cluster will not be able to request certificates. Make sure to create your desired namespace and then label it like so:
kubectl create namespace cert-manager
kubectl label namespace cert-manager certmanager.k8s.io/disable-validation=true
Install Helm on you local system and setup Tiller within the cluster.
Create a service account for Tiller and bind it to the cluster-admin role:
kubectl create namespace kubermatic
kubectl create serviceaccount -n kubermatic tiller
kubectl create clusterrolebinding tiller-cluster-role --clusterrole=cluster-admin --serviceaccount=kubermatic:tiller
Afterwards install Tiller with the correct service account:
helm --service-account tiller --tiller-namespace kubermatic init
Now you’re ready to deploy Kubermatic and its charts. It’s generally advisable to postpone installing the final certs
chart until you acquired LoadBalancer IPs/hostnames and can update your DNS zone to point to your new installation. This
ensures that the cert-manager can quickly acquire TLS certificates instead of running into DNS issues.
On your very first installation, you must install the Kubermatic chart with kubermatic.checks.crd.disable=true. This
disables a check that was put in place for upgrading older Kubermatic versions to the current one.
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace nginx-ingress-controller nginx-ingress-controller charts/nginx-ingress-controller/
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace cert-manager cert-manager charts/cert-manager/
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace oauth oauth charts/oauth/
# Used for storing etcd snapshots
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace minio minio charts/minio/
# if you deployed Minio in another namespace than "minio", make sure to adjust s3Exporter.endpoint in the values.yaml
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace kube-system s3-exporter charts/s3-exporter/
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace kubermatic --set kubermatic.checks.crd.disable=true kubermatic charts/kubermatic/
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace iap iap charts/iap/
# When running on a cloud Provider like GCP, AWS or Azure with LB support also install the nodeport-proxy
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace nodeport-proxy nodeport-proxy charts/nodeport-proxy/
# For logging stack, ensure that all charts are deployed within the logging namespace
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace logging elasticsearch charts/logging/elasticsearch/
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace logging fluentbit charts/logging/fluentbit/
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace logging kibana charts/logging/kibana/
# For monitoring stack
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace monitoring prometheus charts/monitoring/prometheus/
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace monitoring node-exporter charts/monitoring/node-exporter/
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace monitoring kube-state-metrics charts/monitoring/kube-state-metrics/
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace monitoring grafana charts/monitoring/grafana/
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace monitoring alertmanager charts/monitoring/alertmanager/
# optional part of the monitoring stack, only used if you explicitly define scraping rules to monitoring arbitrary targets
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace monitoring blackbox-exporter charts/monitoring/blackbox-exporter/
# cluster backups (etcd and volumes) (requires proper configuration in the chart's values.yaml)
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace velero velero charts/backup/velero/
After all charts have been deployed, update your DNS accordingly. See the last section on this page for more details. Once that is done, wait a bit and then install the final Helm chart:
helm upgrade --tiller-namespace kubermatic --install --wait --timeout 300 --values values.yaml --namespace default certs charts/certs/
We run an individual cronjob every 20 minutes for each customer cluster to backup the etcd-ring. Snapshots will be
stored by default to an internal S3 bucket provided by minio, though this can be changed by modifying the
storeContainer & cleanupContainer in the values.yaml to your needs.
The cronjobs will be executed in the kube-system namespace. Therefore if a container needs credentials, a secret must
be created in the kube-system namespace.
The workflow:
storeContainer takes the snapshot and stores it somewhereThe storeContainer will be executed on each backup process after a snapshot has been created and stored on a shared
volume accessible by the container. By default only the last 20 revisions will be kept. Older snapshots will be deleted.
By default the container will store the snapshot to minio.
The cleanupContainer will delete all snapshots in S3 after a cluster has been deleted.
If the default container will be used, a secret named s3-credentials in the kube-system namespace must be created:
apiVersion: v1
kind: Secret
type: Opaque
metadata:
name: s3-credentials
namespace: kube-system
data:
ACCESS_KEY_ID: "SOME_BASE64_ENCODED_ACCESS_KEY"
SECRET_ACCESS_KEY: "SOME_BASE64_ENCODED_SECRET_KEY"
Kubermatic needs to have at least 2 DNS entries set.
The frontend of Kubermatic needs a single, simple DNS entry. Let’s assume it is being installed to serve
kubermatic.initech.com. For the system services like Prometheus or Grafana, you will also want to create a wildcard
DNS record *.kubermatic.initech.com pointing to the same IP/hostname.
When running on a cloud provider which supports services of type LoadBalancer, the nginx-ingress chart should be configured to create such a service. The load balancer’s IP can then be fetched via:
kubectl -n nginx-ingress-controller get service nginx-ingress-controller -o wide
#NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
#nginx-ingress-controller LoadBalancer 10.47.242.69 35.198.146.37 80:30822/TCP,443:31412/TCP 160d app=ingress-nginx
The EXTERNAL-IP field shows the correct IP used for the DNS entry. Depending on your provider, this might also be a
hostname, in which case you should set a CNAME record instead of an A record.
Without a LoadBalancer nginx will run as DaemonSet & allocate 2 ports on the host (80 & 443). Configuration of nginx happens via the values.yaml. The DNS entry needs to be configured to point to one or more of the cluster nodes.
For each seed cluster a single wildcard DNS entry must be configured. All apiservers of all customer clusters are being exposed via either NodePorts or a single LoadBalancer.
The domain will be based on the name of the seed-cluster as defined in the datacenters.yaml and the domain under which the frontend is available.
For example, when the base domain is kubermatic.initech.com and a seed cluster in your datacenters.yaml is called
europe-west1, then
europe-west1.kubermatic.initech.com*.europe-west1.kubermatic.initech.comA customer cluster created in this seed cluster would get the domain [cluster ID].europe-west1.kubermatic.example.com.
Get the IP from the nodeport-lb service:
kubectl -n nodeport-proxy get service nodeport-lb
#NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
#nodeport-lb LoadBalancer 10.47.242.236 35.198.93.90 32493:30195/TCP,31434:30739/TCP,30825:32503/TCP,30659:30324/TCP 93d
Take one or more of the seed cluster worker nodes IPs.