Setting up Cilium in AWS ENI mode

Create an AWS cluster

Setup a Kubernetes on AWS. You can use any method you prefer, bu for the simplicity of this tutorial, we are going to use eksctl. For more details on how to set up an EKS cluster using eksctl, see the section Installation on AWS EKS.

eksctl create cluster -n eni-cluster -N 0

Disable the aws-node DaemonSet (EKS only)

If you are running an EKS cluster, disable the aws-node DaemonSet so it does not interfere with the ENIs managed by Cilium:

kubectl -n kube-system set image daemonset/aws-node aws-node=docker.io/spaster/alpine-sleep

Prepare & Deploy Cilium

Download the Cilium release tarball and change to the kubernetes install directory:

curl -LO https://github.com/cilium/cilium/archive/1.6.1.tar.gz
tar xzvf 1.6.1.tar.gz
cd cilium-1.6.1/install/kubernetes

Install Helm to prepare generating the deployment artifacts based on the Helm templates.

Generate the required YAML file and deploy it:

helm template cilium \
  --namespace kube-system \
  --set global.eni=true \
  --set global.egressMasqueradeInterfaces=eth0 \
  --set global.tunnel=disabled \
  --set global.nodeinit.enabled=true \
  > cilium.yaml
kubectl create -f cilium.yaml

Note

The above options are assuming that masquerading is desired and that the VM is connected to the VPC using eth0. It will route all traffic that does not stay in the VPC via eth0 and masquerade it.

If you want to avoid masquerading, set global.masquerade=false. You must ensure that the security groups associated with the ENIs (eth1, eth2, …) allow for egress traffic to outside of the VPC. By default, the security groups for pod ENIs are derived from the primary ENI (eth0).

Scale up the cluster

eksctl get nodegroup --cluster eni-cluster
CLUSTER                     NODEGROUP       CREATED                 MIN SIZE        MAX SIZE        DESIRED CAPACITY        INSTANCE TYPE   IMAGE ID
test-cluster                ng-25560078     2019-07-23T06:05:35Z    0               2               0                       m5.large        ami-0923e4b35a30a5f53
eksctl scale nodegroup --cluster eni-cluster -n ng-25560078 -N 2
[]  scaling nodegroup stack "eksctl-test-cluster-nodegroup-ng-25560078" in cluster eksctl-test-cluster-cluster
[]  scaling nodegroup, desired capacity from 0 to 2

Validate the Installation

You can monitor as Cilium and all required components are being installed:

kubectl -n kube-system get pods --watch
NAME                                    READY   STATUS              RESTARTS   AGE
cilium-operator-cb4578bc5-q52qk         0/1     Pending             0          8s
cilium-s8w5m                            0/1     PodInitializing     0          7s
coredns-86c58d9df4-4g7dd                0/1     ContainerCreating   0          8m57s
coredns-86c58d9df4-4l6b2                0/1     ContainerCreating   0          8m57s

It may take a couple of minutes for all components to come up:

cilium-operator-cb4578bc5-q52qk         1/1     Running   0          4m13s
cilium-s8w5m                            1/1     Running   0          4m12s
coredns-86c58d9df4-4g7dd                1/1     Running   0          13m
coredns-86c58d9df4-4l6b2                1/1     Running   0          13m

Deploy the connectivity test

You can deploy the “connectivity-check” to test connectivity between pods.

kubectl apply -f https://raw.githubusercontent.com/cilium/cilium/1.6.1/examples/kubernetes/connectivity-check/connectivity-check.yaml

It will deploy a simple probe and echo server running with multiple replicas. The probe will only report readiness while it can successfully reach the echo server:

kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
echo-585798dd9d-ck5xc    1/1     Running   0          75s
echo-585798dd9d-jkdjx    1/1     Running   0          75s
echo-585798dd9d-mk5q8    1/1     Running   0          75s
echo-585798dd9d-tn9t4    1/1     Running   0          75s
echo-585798dd9d-xmr4p    1/1     Running   0          75s
probe-866bb6f696-9lhfw   1/1     Running   0          75s
probe-866bb6f696-br4dr   1/1     Running   0          75s
probe-866bb6f696-gv5kf   1/1     Running   0          75s
probe-866bb6f696-qg2b7   1/1     Running   0          75s
probe-866bb6f696-tb926   1/1     Running   0          75s