Load-balancing & Service Discovery
This tutorial will guide you to perform load-balancing and service discovery across multiple Kubernetes clusters when using Cilium.
Prerequisites
You need to have a functioning Cluster Mesh setup, please follow the guide Setting up Cluster Mesh to set it up.
Load-balancing with Global Services
Establishing load-balancing between clusters is achieved by defining a
Kubernetes service with identical name and namespace in each cluster and adding
the annotation service.cilium.io/global: "true"
to declare it global.
Cilium will automatically perform load-balancing to pods in both clusters.
apiVersion: v1
kind: Service
metadata:
name: rebel-base
annotations:
service.cilium.io/global: "true"
spec:
type: ClusterIP
ports:
- port: 80
selector:
name: rebel-base
Disabling Global Service Sharing
By default, a Global Service will load-balance across backends in multiple clusters.
This implicitly configures service.cilium.io/shared: "true"
. To prevent service
backends from being shared to other clusters, this option should be disabled.
Below example will expose remote endpoint without sharing local endpoints.
apiVersion: v1
kind: Service
metadata:
name: rebel-base
annotations:
service.cilium.io/global: "true"
service.cilium.io/shared: "false"
spec:
type: ClusterIP
ports:
- port: 80
selector:
name: rebel-base
Synchronizing Kubernetes EndpointSlice (Beta)
Note
This is a beta feature. Please provide feedback and file a GitHub issue if you experience any problems.
By default Kubernetes EndpointSlice synchronization is disabled on non Headless Global services.
To have Cilium discover remote clusters endpoints of a Global Service
from DNS or any third party controllers, enable synchronization by adding
the annotation service.cilium.io/global-sync-endpoint-slices: "true"
.
This will allow Cilium to create Kubernetes EndpointSlices belonging to a
remote cluster for services that have that annotation.
Regarding Global Headless services this option is enabled by default unless
explicitly opted-out by adding the annotation service.cilium.io/global-sync-endpoint-slices: "false"
.
Note that this feature does not complement/is not required by any other Cilium features and is only required if you need to discover EndpointSlice from remote cluster on third party controllers. For instance, the Cilium ingress controller works in a Cluster Mesh without enabling this feature, although if you use any other ingress controller you may need to enable this.
This feature is currently disabled by default via a feature flag. To install Cilium with EndpointSlice Cluster Mesh synchronization, run:
helm install cilium ./cilium \ --namespace kube-system \ --set clustermesh.enableEndpointSliceSynchronization=true
To enable EndpointSlice Cluster Mesh synchronization on an existing Cilium installation, run:
helm upgrade cilium ./cilium \ --namespace kube-system \ --reuse-values \ --set clustermesh.enableEndpointSliceSynchronization=true kubectl -n kube-system rollout restart deployment/cilium-operator
Known Limitations
This is a beta feature, you may experience bugs or shortcomings.
Hostnames are synchronized as is without any form of conflict resolution mechanisms. This means that multiple StatefulSets with a single governing Service that synchronize EndpointSlices across multiple clusters should have different names. For instance, you can add the cluster name to the StatefulSet name (
cluster1-my-statefulset
instead ofmy-statefulset
).
Deploying a Simple Example Service
In cluster 1, deploy:
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/kubernetes/clustermesh/global-service-example/cluster1.yaml
In cluster 2, deploy:
kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/kubernetes/clustermesh/global-service-example/cluster2.yaml
From either cluster, access the global service:
kubectl exec -ti deployment/x-wing -- curl rebel-base
You will see replies from pods in both clusters.
In cluster 1, add
service.cilium.io/shared="false"
to existing global servicekubectl annotate service rebel-base service.cilium.io/shared="false" --overwrite
From cluster 1, access the global service one more time:
kubectl exec -ti deployment/x-wing -- curl rebel-base
You will still see replies from pods in both clusters.
From cluster 2, access the global service again:
kubectl exec -ti deployment/x-wing -- curl rebel-base
You will see replies from pods only from cluster 2, as the global service in cluster 1 is no longer shared.
In cluster 1, remove
service.cilium.io/shared
annotation of existing global servicekubectl annotate service rebel-base service.cilium.io/shared-
From either cluster, access the global service:
kubectl exec -ti deployment/x-wing -- curl rebel-base
You will see replies from pods in both clusters again.
Limitations
Global NodePort services load balance across both local and remote backends only if Cilium is configured to replace kube-proxy (either
kubeProxyReplacement=true
ornodePort.enabled=true
). Otherwise, only local backends are eligible for load balancing when accessed through the NodePort.Global services accessed by a Node, or a Pod running in host network, load balance across both local and remote backends only if Cilium is configured to replace kube-proxy (
kubeProxyReplacement=true
). This limitation can be overcome enabling SocketLB in the host namespace:socketLB.enabled=true
,socketLB.hostNamespaceOnly=true
. Otherwise, only local backends are eligible for load balancing.