IPVLAN based Networking (beta)

This guide explains how to configure Cilium to set up an ipvlan-based datapath instead of the default veth-based one.

Note

This is a beta feature. Please provide feedback and file a GitHub issue if you experience any problems.

The feature lacks support of the following, which will be resolved in upcoming Cilium releases:

  • IPVLAN L2 mode
  • L7 policy enforcement
  • NAT64
  • IPVLAN with tunneling

Note

The ipvlan-based datapath in L3 mode requires v4.12 or more recent Linux kernel, while L3S mode, in addition, requires a stable kernel with the fix mentioned in this document (see below).

First step is to download the Cilium Kubernetes descriptor:

curl -LO https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/kubernetes/1.14/cilium.yaml
curl -LO https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/kubernetes/1.13/cilium.yaml
curl -LO https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/kubernetes/1.12/cilium.yaml
curl -LO https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/kubernetes/1.11/cilium.yaml
curl -LO https://raw.githubusercontent.com/cilium/cilium/HEAD/examples/kubernetes/1.10/cilium.yaml

Edit the cilium-config ConfigMap in that file with the etcd server that is running in your cluster and set the option datapath-mode to "ipvlan".

It is also required to specify the ipvlan master device which typically points to a networking device that is facing the external network. This is done through setting ipvlan-master-device to the name of the networking device such as "eth0" or "bond0", for example. Be aware this ConfigMap will be used by all nodes, so it is required this device name is consistent on all nodes where you are going to deploy Cilium.

The ipvlan datapath only supports direct routing mode right now, therefore tunneling must be disabled through setting tunnel to "disabled".

To make ipvlan work between hosts, routes on each host have to be installed either manually or automatically by Cilium. The latter can be enabled through setting auto-direct-node-routes to "true".

The --install-iptables-rules parameter is optional and if set to "false" then Cilium will not install any iptables rules which are mainly for interaction with kube-proxy, and additionally it will trigger ipvlan setup in L3 mode. For the default case where the latter is "true", ipvlan is operated in L3S mode such that netfilter in host namespace is not bypassed. Optionally, the agent can also be set up for masquerading all traffic leaving the ipvlan master device if masquerade is set to "true". Note that in order for L3S mode to work correctly, a kernel with the following fix is required: d5256083f62e . This fix is included in stable kernels v4.9.155, 4.14.98, 4.19.20, 4.20.6 or higher. Without this kernel fix, ipvlan in L3S mode cannot connect to kube-apiserver.

Masquerading with iptables in L3-only mode is not possible since netfilter hooks are bypassed in the kernel in this mode, hence L3S (symmetric) had to be introduced in the kernel at the cost of performance. However, Cilium supports its own BPF-based masquerading which does not rely in any way on iptables masquerading. If the --install-iptables-rules parameter is set to "false" and --masquerade set to "true", then Cilium will use the more efficient BPF-based masquerading where ipvlan can remain in L3 mode as well (instead of L3S). A Linux kernel v4.16 or higher would be required for BPF-based masquerading.

Example ConfigMap extract for ipvlan in pure L3 mode:

datapath-mode: "ipvlan"
ipvlan-master-device: "bond0"
tunnel: "disabled"
install-iptables-rules: "false"
auto-direct-node-routes: "true"

Example ConfigMap extract for ipvlan in L3S mode with iptables masquerading all traffic leaving the node:

datapath-mode: "ipvlan"
ipvlan-master-device: "bond0"
tunnel: "disabled"
masquerade: "true"
auto-direct-node-routes: "true"

Example ConfigMap extract for ipvlan in L3 mode with more efficient BPF-based masquerading instead of iptables-based:

datapath-mode: "ipvlan"
ipvlan-master-device: "bond0"
tunnel: "disabled"
masquerade: "true"
install-iptables-rules: "false"
auto-direct-node-routes: "true"

Apply the DaemonSet file to deploy Cilium and verify that it has come up correctly:

kubectl create -f ./cilium.yaml
kubectl -n kube-system get pods -l k8s-app=cilium
NAME                READY     STATUS    RESTARTS   AGE
cilium-crf7f        1/1       Running   0          10m

For further information on Cilium’s ipvlan datapath mode, see Architecture.