Working with KubeLB
Starting with KKP v2.24, KubeLB Enterprise Edition is integrated into the Kubermatic Kubernetes Platform (KKP). This means that you can use KubeLB to provision load balancers for your KKP clusters. KKP will take care of configurations and deployments for you in the user cluster. Admins mainly need to create the KubeLB manager cluster and configure KKP to use it.
For KubeLB Enterprise offering, please contact sales for more information.
For usage outside of KKP please follow the guide along.
Usage
This guide assumes that the KubeLB manager cluster has been configured by following the installation guide.
KubeLB Manager configuration
Each cluster that wants load balancer services is treated as a unique tenant by KubeLB. This means that the KubeLB manager needs to be aware of the tenant clusters. This is done by registering the tenant clusters in the KubeLB manager cluster. This is done by creating a namespace with the unique name of tenant and labelling it with kubelb.k8c.io/managed-by: kubelb
.
We then create a restricted service account in the tenant cluster that will be used by the KubeLB CCM to communicate with the KubeLB manager cluster. Eventually, we need a kubeconfig
that can be configured in the KubeLB CCM to communicate with the KubeLB manager cluster.
This script can be used for creating the required RBAC and generating the kubeconfig:
#!/usr/bin/env bash
set -euox pipefail
if [ $# -ne 1 ] ; then
echo 'No cluster ID provided'
exit 1
fi
clusterId=$1
namespace=$clusterId
kubectl create namespace "$namespace"
kubectl label namespace "$namespace" kubelb.k8c.io/managed-by=kubelb
cat <<EOF | kubectl apply -n "$namespace" -f -
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: kubelb-agent
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: kubelb-agent-role
rules:
- apiGroups:
- kubelb.k8c.io
resources:
- loadbalancers
verbs:
- create
- delete
- get
- list
- patch
- update
- watch
- apiGroups:
- kubelb.k8c.io
resources:
- loadbalancers/status
verbs:
- get
- patch
- update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: kubelb-agent-rolebinding
subjects:
- kind: ServiceAccount
name: kubelb-agent
roleRef:
kind: Role
name: kubelb-agent-role
apiGroup: rbac.authorization.k8s.io
EOF
# your server name goes here
server=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
token_name=$(kubectl -n $namespace get sa kubelb-agent -o jsonpath='{.secrets[0].name}')
ca=$(kubectl -n $namespace get secret/$token_name -o jsonpath='{.data.ca\.crt}')
token=$(kubectl -n $namespace get secret/$token_name -o jsonpath='{.data.token}' | base64 --decode)
echo "
apiVersion: v1
kind: Config
clusters:
- name: kubelb-cluster
cluster:
certificate-authority-data: ${ca}
server: ${server}
contexts:
- name: default-context
context:
cluster: kubelb-cluster
namespace: $namespace
user: default-user
current-context: default-context
users:
- name: default-user
user:
token: ${token}"
Manager Config
We have a dedicated CRD config
that can be used to manage configuration for KubeLB manager. The following is an example of a config
CRD:
apiVersion: kubelb.k8c.io/v1alpha1
kind: Config
metadata:
name: default
namespace: kubelb
spec:
envoyProxy:
replicas: 3
topology: shared
Users can skip creation of config
CRD via helm by setting kubelb.skipConfigGeneration
to true
in the values.yaml. This will de-couple the config
CRD from the helm chart and users can manage it separately.
NOTE: The config
CR named default
is mandatory for KubeLB manager to work.
KubeLB CCM configuration
For CCM, during installation we need to provide the kubeconfig
that we generated in the previous step. Also, the tenantName
field in the values.yaml should be set to the name of the tenant cluster.
Propagate annotations from services to LoadBalancer
KubeLB can propagate annotations from services in the consumer cluster to the LoadBalancers in the management cluster. This is useful for setting annotations that are required by the cloud provider to configure the LoadBalancers. For example, the service.beta.kubernetes.io/aws-load-balancer-internal
annotation is used to create an internal LoadBalancer in AWS.
Annotations are not propagated by default since tenants can make unwanted changes to the LoadBalancer configuration. Since each tenant is treated as a separate entity, the KubeLB manager cluster needs to be configured to allow the propagation of specific annotations.
This can be achieved in the following ways:
Propagate all annotations
This can be done by setting the kubelb.propagateAllAnnotations
field to true
in the config
CRD. This will propagate all annotations from the service to the LoadBalancer.
apiVersion: kubelb.k8c.io/v1alpha1
kind: Config
metadata:
name: default
namespace: kubelb
spec:
propagateAllAnnotations: true
Propagate specific annotations
This can be done by setting the kubelb.propagatedAnnotations
field in the config
CRD. This field is a map of annotations that are allowed to be propagated. The key is the annotation name and the value is the annotation value. If the value is empty, any value is allowed.
apiVersion: kubelb.k8c.io/v1alpha1
kind: Config
metadata:
name: default
namespace: kubelb
spec:
propagatedAnnotations:
metalb.universe.tf/allow-shared-ip: ""
metallb.universe.tf/loadBalancerIPs: "8.8.8.8"
Propagate annotations from tenant namespace
This is done by adding the kubelb.k8c.io/propagate-annotation
annotation to the tenant namespace in the management cluster. For multiple annotations, the suffix can be incremented like kubelb.k8c.io/propagate-annotation-1
. The suffix can be any arbitrary string, it’s just for uniqueness.
Here is a basic example, where optionally kubelb allows to set a values filter:
annotations:
kubelb.k8c.io/propagate-annotation: "metallb.universe.tf/address-pool"
kubelb.k8c.io/propagate-annotation-1: "metallb.universe.tf/loadBalancerIPs=192.168.1.100,192.168.1.102"
The first configured annotation allows propagating any value for metallb.universe.tf/address-pool
and the second one restricts the values to be either 192.168.1.100
or 192.168.1.102
.