Upgrade Guide¶

Upgrading from >=1.2.5 to 1.4.y¶

1. Follow the standard procedures to perform the upgrade as described in Upgrading Minor Versions.

Changes that may require action¶

• The --serve option was removed from cilium-bugtool in favor of a much reduced binary size. If you want to continue using the option, please use an older version of the cilium-bugtool binary.
• The DaemonSet now uses dnsPolicy: ClusterFirstWithHostNet in order for Cilium to look up Kubernetes service names via DNS. This in turn requires the cluster to run kube-dns. If you are not running kube-dns, remove the dnsPolicy field. This will mean that you cannot use the etcd-operator. More details can be found in the kube-dns section.

New ConfigMap Options¶

• enable-ipv4: If true, all endpoints are allocated an IPv4 address.
• enable-ipv6: If true, all endpoints are allocated an IPv6 address.
• preallocate-bpf-maps: If true, reduce per-packet latency at the expense of up-front memory allocation for entries in BPF maps. If this value is modified, then during the next Cilium startup the restore of existing endpoints and tracking of ongoing connections may be disrupted. This may lead to policy drops or a change in loadbalancing decisions for a connection for some time. Endpoints may need to be recreated to restore connectivity. If this option is set to false during an upgrade to 1.4.0 or later, then it may cause one-time disruptions during the upgrade.
• auto-direct-node-routes: If true, then enable automatic L2 routing between nodes. This is useful when running in direct routing mode and can be used as an alternative to running a routing daemon. Routes to other Cilium managed nodes will then be installed on automatically.
• install-iptables-rules: If set to false then Cilium will not install any iptables rules which are mainly for interaction with kube-proxy. By default it is set to true.
• masquerade: The agent can optionally be set up for masquerading all network traffic leaving the main networking device if masquerade is set to true. By default it is set to false.
• datapath-mode: Cilium can operate in two different datapath modes, that is, either based upon veth devices (default) or ipvlan devices (beta). The latter requires an additional setting to specify the ipvlan master device.
• New ipvlan-specific CNI integration mode options (beta):
  • ipvlan-master-device: When running Cilium in ipvlan datapath mode, an ipvlan master device must be specified. This is typically pointing to a networking device that is facing the external network. Be aware that this will be used by all nodes, so it is required that the device name is consistent on all nodes where this is going to be deployed.
• New flannel-specific CNI integration mode options (beta):
  • flannel-master-device: When running Cilium with policy enforcement enabled on top of Flannel, the BPF programs will be installed on the network interface specified in this option and on each network interface belonging to a pod.
  • flannel-uninstall-on-exit: If flannel-master-device is specified, this determines whether Cilium should remove BPF programs from the master device and interfaces belonging to pods when the Cilium DaemonSet is deleted. If true, Cilium will remove programs from the pods.
  • flannel-manage-existing-containers: On startup, install a BPF programs to allow for policy enforcement on pods that are currently managed by Flannel. This also requires the Cilium DaemonSet to be running with hostPID: true, which is not enabled by default.

Deprecated ConfigMap Options¶

• disable-ipv4: Superseded by enable-ipv4, with the logic reversed.
• legacy-host-allows-world: This option allowed users to specify Cilium 1.0-style policies that treated traffic that is masqueraded from the outside world as though it arrived from the local host. As of Cilium 1.4, the option is disabled by default if not specified in the ConfigMap, and the option is scheduled to be removed in Cilium 1.5 or later. For more information, see Traffic from world to endpoints is classified as from host.

Upgrading from 1.2.x to 1.3.y¶

1. If you are running Cilium 1.0.x or 1.1.x, please upgrade to 1.2.x first. It is also possible to upgrade from 1.0 or 1.1 directly to 1.3 by combining the upgrade instructions for each minor release. See 1.2 Upgrade Notes.
2. Upgrade to Cilium 1.2.4 or later using the guide Upgrading Micro Versions.
3. Follow the standard procedures to perform the upgrade as described in Upgrading Minor Versions.

New ConfigMap Options¶

• ct-global-max-entries-tcp/ct-global-max-entries-other: Specifies the maximum number of connections supported across all endpoints, split by protocol: tcp or other. One pair of maps uses these values for IPv4 connections, and another pair of maps use these values for IPv6 connections. If these values are modified, then during the next Cilium startup the tracking of ongoing connections may be disrupted. This may lead to brief policy drops or a change in loadbalancing decisions for a connection.
• clean-cilium-bpf-state: Similar to clean-cilium-state but only cleans the BPF state while preserving all other state. Endpoints will still be restored and IP allocations will prevail but all datapath state is cleaned when Cilium starts up. Not required for normal operation.

New ConfigMap Options¶

• cluster-name: Name of the cluster. Only relevant when building a mesh of clusters.
• cluster-id: Unique ID of the cluster. Must be unique across all connected clusters and in the range of 1 and 255. Only relevant when building a mesh of clusters.
• monitor-aggregation-level: If you want cilium monitor to aggregate tracing for packets, set this level to “low”, “medium”, or “maximum”. The higher the level, the less packets that will be seen in monitor output.

Upgrade Impact¶

Upgrading to 1.2.x from 1.1.y¶

1. If you are running Cilium 1.0.x. Please consider upgrading to 1.1.x first. At the very least, be aware that all 1.1 Upgrade Notes will apply as well.
2. Upgrade to Cilium 1.1.4 or later using the guide Upgrading Micro Versions.
3. Consider the new ConfigMap options described in section New ConfigMap Options.
4. Follow the standard procedures to perform the upgrade as described in Upgrading Minor Versions.

New ConfigMap Options¶

• legacy-host-allows-world: In Cilium 1.0, all traffic from the host, including from local processes and traffic that is masqueraded from the outside world to the host IP, would be classified as from the host entity (reserved:host label). Furthermore, to allow Kubernetes agents to perform health checks over IP into the endpoints, the host is allowed by default. This means that all traffic from the outside world is also allowed by default, regardless of security policy. This behavior is continued to maintain backwards compatible but it can be disabled (recommended) by setting legacy-host-allows-world to false. See Traffic from world to endpoints is classified as from host for more details.
• sidecar-istio-proxy-image: Regular expression matching compatible Istio sidecar istio-proxy container image names.

Deprecated Options¶

• sidecar-http-proxy

Upgrading to Cilium 1.1.x from Cilium 1.0.y¶

1. Consider the new ConfigMap options described in section New ConfigMap Options.
2. Follow the guide in MTU handling behavior change in Cilium 1.1 to update the MTU of existing endpoints to the new improved model. This step can also be performed after the upgrade but performing it before the upgrade will guarantee that no packet loss occurs during the upgrade phase.
3. Follow the standard procedures to perform the upgrade as described in Upgrading Minor Versions.

Downgrading to Cilium 1.1.x from Cilium 1.2.y¶

When downgrading from Cilium 1.2, the target version must be Cilium 1.1.4 or later.

1. Check whether you have any DNS policy rules installed:

   $ kubectl get cnp --all-namespaces -o yaml | grep "fqdn"
   

   If any DNS rules exist, these must be removed prior to downgrade as these rules are not supported by Cilium 1.1.

MTU handling behavior change in Cilium 1.1¶

Cilium 1.0 by default configured the MTU of all Cilium-related devices and endpoint devices to 1450 bytes, to guarantee that packets sent from an endpoint would remain below the MTU of a tunnel. This had the side-effect that when a Cilium-managed pod made a request to an outside (world) IP, if the response came back in 1500B chunks, then it would be fragmented when transmitted to the cilium_host device. These fragments then pass through the Cilium policy logic. Latter IP fragments would not contain L4 ports, so if any L4 or L4+L7 policy was applied to the destination endpoint, then the fragments would be dropped. This could cause disruption to network traffic.

$ kubectl create -f https://raw.githubusercontent.com/cilium/mtu-update/v1.1/mtu-update.yaml

$ kubectl get ds mtu-update -n kube-system
NAME         DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
mtu-update   1         1         1         1            1           <none>          18s

$ kubectl delete -f https://raw.githubusercontent.com/cilium/mtu-update/v1.1/mtu-update.yaml

Upgrading to Cilium 1.0.x from older versions¶

Versions of Cilium older than 1.0.0 are unsupported for upgrade. The Upgrading Minor Versions may work, however it may be more reliable to start again from the Installation

Downgrading to Cilium 1.0.x from Cilium 1.1.y¶

1. Check whether you have any CIDR policy rules not compatible with 1.0.x:

   $ kubectl get cnp --all-namespaces -o yaml
   

   • Default prefix: If any CIDR rules match on the CIDR prefix /0, these must be removed prior to downgrade as these rules are not supported by Cilium 1.0. (PR 4458)
   • CIDR-dependent L4 policies: If any CIDR rules also specify a toPorts section, these must be removed prior to downgrade as these rules are not supported by Cilium 1.0. (PR 3835)
   • IPv6 CIDR matching: Technically supported since 1.0.2, officially supported since 1.1.0. (PR 4004)

   Any rules that are not compatible with 1.0.x must be removed before downgrade.

2. Add or update the option clean-cilium-bpf-state to the ConfigMap and set to true. This will cause BPF maps to be removed during the downgrade, which avoids bugs such as Issue 5070. As a side effect, any loadbalancing decisions for active connections will be disrupted during downgrade. For more information on changing ConfigMap options, see Upgrading a ConfigMap.

3. Follow the instructions in the section Upgrading Minor Versions to perform the downgrade to the latest micro release of the 1.0 series.

4. Set the clean-cilium-bpf-state ConfigMap option back to false.

Export the current ConfigMap¶

$ kubectl get configmap -n kube-system cilium-config -o yaml --export > cilium-cm-old.yaml
$ cat ./cilium-cm-old.yaml
apiVersion: v1
data:
  clean-cilium-state: "false"
  debug: "true"
  disable-ipv4: "false"
  etcd-config: |-
    ---
    endpoints:
    - https://192.168.33.11:2379
    #
    # In case you want to use TLS in etcd, uncomment the 'ca-file' line
    # and create a kubernetes secret by following the tutorial in
    # https://cilium.link/etcd-config
    ca-file: '/var/lib/etcd-secrets/etcd-ca'
    #
    # In case you want client to server authentication, uncomment the following
    # lines and add the certificate and key in cilium-etcd-secrets below
    key-file: '/var/lib/etcd-secrets/etcd-client-key'
    cert-file: '/var/lib/etcd-secrets/etcd-client-crt'
kind: ConfigMap
metadata:
  creationTimestamp: null
  name: cilium-config
  selfLink: /api/v1/namespaces/kube-system/configmaps/cilium-config

In the ConfigMap above, we can verify that Cilium is using debug with true, it has a etcd endpoint running with TLS, and the etcd is set up to have client to server authentication.

Download the ConfigMap with the changes for 1.4.0¶

$ wget https://raw.githubusercontent.com/cilium/cilium/1.4.0/examples/kubernetes/1.8/cilium-cm.yaml

$ wget https://raw.githubusercontent.com/cilium/cilium/1.4.0/examples/kubernetes/1.9/cilium-cm.yaml

$ wget https://raw.githubusercontent.com/cilium/cilium/1.4.0/examples/kubernetes/1.10/cilium-cm.yaml

$ wget https://raw.githubusercontent.com/cilium/cilium/1.4.0/examples/kubernetes/1.11/cilium-cm.yaml

$ wget https://raw.githubusercontent.com/cilium/cilium/1.4.0/examples/kubernetes/1.12/cilium-cm.yaml

$ wget https://raw.githubusercontent.com/cilium/cilium/1.4.0/examples/kubernetes/1.13/cilium-cm.yaml

Verify its contents:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: cilium-config
  namespace: kube-system
data:
  # This etcd-config contains the etcd endpoints of your cluster. If you use
  # TLS please make sure you follow the tutorial in https://cilium.link/etcd-config
  etcd-config: |-
    ---
    endpoints:
      - https://cilium-etcd-client.kube-system.svc:2379
    #
    # In case you want to use TLS in etcd, uncomment the 'ca-file' line
    # and create a kubernetes secret by following the tutorial in
    # https://cilium.link/etcd-config
    ca-file: '/var/lib/etcd-secrets/etcd-client-ca.crt'
    #
    # In case you want client to server authentication, uncomment the following
    # lines and create a kubernetes secret by following the tutorial in
    # https://cilium.link/etcd-config
    key-file: '/var/lib/etcd-secrets/etcd-client.key'
    cert-file: '/var/lib/etcd-secrets/etcd-client.crt'

  # If you want to run cilium in debug mode change this value to true
  debug: "false"

  # Enable IPv4 addressing. If enabled, all endpoints are allocated an IPv4
  # address.
  enable-ipv4: "true"

  # Enable IPv6 addressing. If enabled, all endpoints are allocated an IPv6
  # address.
  enable-ipv6: "false"

  # If a serious issue occurs during Cilium startup, this
  # invasive option may be set to true to remove all persistent
  # state. Endpoints will not be restored using knowledge from a
  # prior Cilium run, so they may receive new IP addresses upon
  # restart. This also triggers clean-cilium-bpf-state.
  clean-cilium-state: "false"
  # If you want to clean cilium BPF state, set this to true;
  # Removes all BPF maps from the filesystem. Upon restart,
  # endpoints are restored with the same IP addresses, however
  # any ongoing connections may be disrupted briefly.
  # Loadbalancing decisions will be reset, so any ongoing
  # connections via a service may be loadbalanced to a different
  # backend after restart.
  clean-cilium-bpf-state: "false"

  # In Cilium 1.0, all traffic from the host, including from local processes
  # and traffic that is masqueraded from the outside world to the host IP,
  # would be classified as from the host entity (reserved:host label).
  # Furthermore, to allow Kubernetes agents to perform health checks over IP
  # into the endpoints, the host is allowed by default. This means that all
  # traffic from the outside world is also allowed by default, regardless of
  # security policy.
  #
  # This option was introduced in Cilium 1.1 to disable this behaviour. It must
  # be explicitly set to "false" to take effect on Cilium 1.3 or earlier.
  # Cilium 1.4 sets this to "false" by default if it is not specified in the
  # ConfigMap.
  #
  # This option has been deprecated, it will be removed in Cilium 1.5 or later.
  #
  # For more information, see https://cilium.link/host-vs-world
  #legacy-host-allows-world: "false"

  # If you want cilium monitor to aggregate tracing for packets, set this level
  # to "low", "medium", or "maximum". The higher the level, the less packets
  # that will be seen in monitor output.
  monitor-aggregation-level: "none"

  # ct-global-max-entries-* specifies the maximum number of connections
  # supported across all endpoints, split by protocol: tcp or other. One pair
  # of maps uses these values for IPv4 connections, and another pair of maps
  # use these values for IPv6 connections.
  #
  # If these values are modified, then during the next Cilium startup the
  # tracking of ongoing connections may be disrupted. This may lead to brief
  # policy drops or a change in loadbalancing decisions for a connection.
  #
  # For users upgrading from Cilium 1.2 or earlier, to minimize disruption
  # during the upgrade process, comment out these options.
  ct-global-max-entries-tcp: "524288"
  ct-global-max-entries-other: "262144"

  # Pre-allocation of map entries allows per-packet latency to be reduced, at
  # the expense of up-front memory allocation for the entries in the maps. The
  # default value below will minimize memory usage in the default installation;
  # users who are sensitive to latency may consider setting this to "true".
  #
  # This option was introduced in Cilium 1.4. Cilium 1.3 and earlier ignore
  # this option and behave as though it is set to "true".
  #
  # If this value is modified, then during the next Cilium startup the restore
  # of existing endpoints and tracking of ongoing connections may be disrupted.
  # This may lead to policy drops or a change in loadbalancing decisions for a
  # connection for some time. Endpoints may need to be recreated to restore
  # connectivity.
  #
  # If this option is set to "false" during an upgrade from 1.3 or earlier to
  # 1.4 or later, then it may cause one-time disruptions during the upgrade.
  preallocate-bpf-maps: "false"

  # Regular expression matching compatible Istio sidecar istio-proxy
  # container image names
  sidecar-istio-proxy-image: "cilium/istio_proxy"

  # Encapsulation mode for communication between nodes
  # Possible values:
  #   - disabled
  #   - vxlan (default)
  #   - geneve
  tunnel: "vxlan"

  # Name of the cluster. Only relevant when building a mesh of clusters.
  cluster-name: default

  # Unique ID of the cluster. Must be unique across all conneted clusters and
  # in the range of 1 and 255. Only relevant when building a mesh of clusters.
  #cluster-id: 1

  # Interface to be used when running Cilium on top of a CNI plugin.
  # For flannel, use "cni0"
  flannel-master-device: ""
  # When running Cilium with policy enforcement enabled on top of a CNI plugin
  # the BPF programs will be installed on the network interface specified in
  # 'flannel-master-device' and on all network interfaces belonging to
  # a container. When the Cilium DaemonSet is removed, the BPF programs will
  # be kept in the interfaces unless this option is set to "true".
  flannel-uninstall-on-exit: "false"
  # Installs a BPF program to allow for policy enforcement in already running
  # containers managed by Flannel.
  # NOTE: This requires Cilium DaemonSet to be running in the hostPID.
  # To run in this mode in Kubernetes change the value of the hostPID from
  # false to true. Can be found under the path `spec.spec.hostPID`
  flannel-manage-existing-containers: "false"

  # DNS Polling periodically issues a DNS lookup for each `matchName` from
  # cilium-agent. The result is used to regenerate endpoint policy.
  # DNS lookups are repeated with an interval of 5 seconds, and are made for
  # A(IPv4) and AAAA(IPv6) addresses. Should a lookup fail, the most recent IP
  # data is used instead. An IP change will trigger a regeneration of the Cilium
  # policy for each endpoint and increment the per cilium-agent policy
  # repository revision.
  #
  # This option is disabled by default starting from version 1.4.x in favor
  # of a more powerful DNS proxy-based implementation, see [0] for details.
  # Enable this option if you want to use FQDN policies but do not want to use
  # the DNS proxy.
  #
  # To ease upgrade, users may opt to set this option to "true".
  # Otherwise please refer to the Upgrade Guide [1] which explains how to
  # prepare policy rules for upgrade.
  #
  # [0] http://docs.cilium.io/en/stable/policy/language/#dns-based
  # [1] http://docs.cilium.io/en/stable/install/upgrade/#changes-that-may-require-action
  tofqdns-enable-poller: "false"

Add new options¶

Add the new options manually to your old ConfigMap, and make the necessary changes.

In this example, the debug option is meant to be kept with true, the etcd-config is kept unchanged, and legacy-host-allows-world is a new option, but after reading the Version Specific Notes the value was kept unchanged from the default value.

After making the necessary changes, the old ConfigMap was migrated with the new options while keeping the configuration that we wanted:

$ cat ./cilium-cm-old.yaml
apiVersion: v1
data:
  debug: "true"
  disable-ipv4: "false"
  # If you want to clean cilium state; change this value to true
  clean-cilium-state: "false"
  legacy-host-allows-world: "false"
  etcd-config: |-
    ---
    endpoints:
    - https://192.168.33.11:2379
    #
    # In case you want to use TLS in etcd, uncomment the 'ca-file' line
    # and create a kubernetes secret by following the tutorial in
    # https://cilium.link/etcd-config
    ca-file: '/var/lib/etcd-secrets/etcd-ca'
    #
    # In case you want client to server authentication, uncomment the following
    # lines and add the certificate and key in cilium-etcd-secrets below
    key-file: '/var/lib/etcd-secrets/etcd-client-key'
    cert-file: '/var/lib/etcd-secrets/etcd-client-crt'
kind: ConfigMap
metadata:
  creationTimestamp: null
  name: cilium-config
  selfLink: /api/v1/namespaces/kube-system/configmaps/cilium-config

Apply new ConfigMap¶

After adding the options, manually save the file with your changes and install the ConfigMap in the kube-system namespace of your cluster.

$ kubectl apply -n kube-system -f ./cilium-cm-old.yaml

As the ConfigMap is successfully upgraded we can start upgrading Cilium DaemonSet and RBAC which will pick up the latest configuration from the ConfigMap.

Affected versions¶

• Any version of Cilium running on Linux 4.10 or earlier

When a CIDR policy with too many unique prefix lengths is imported, Cilium will reject the policy with a message like the following:

$ cilium policy import too_many_cidrs.json
Error: Cannot import policy: [PUT /policy][500] putPolicyFailure  too many
egress CIDR prefix lengths (128/40)

$ cilium policy import too_many_cidrs.json
Error: Cannot import policy: [PUT /policy][500] putPolicyFailure  Adding
specified prefixes would result in too many prefix lengths (current: 3,
result: 32, max: 18)

The supported count of unique prefix lengths may differ between Cilium minor releases, for example Cilium 1.1 supports 20 unique prefix lengths on Linux 4.10 or older, while Cilium 1.2 only supports 18 (for IPv4) or 4 (for IPv6).

Mitigation¶

Users may construct CIDR policies that use fewer unique prefix lengths. This can be achieved by composing or decomposing adjacent prefixes.

Solution¶

Upgrade the host Linux version to 4.11 or later. This step is beyond the scope of the Cilium guide.

Affected versions¶

• Cilium 1.0 or earlier deployed using the DaemonSet and ConfigMap YAMLs provided with that release.

Affected environments will see no output for one or more of the below commands:

$ kubectl get ds cilium -n kube-system -o yaml | grep -B 3 -A 2 -i legacy-host-allows-world
$ kubectl get cm cilium-config -n kube-system -o yaml | grep -i legacy-host-allows-world

Unaffected environments will see the following output (note the configMapKeyRef key in the Cilium DaemonSet and the legacy-host-allows-world: "false" setting of the ConfigMap):

$ kubectl get ds cilium -n kube-system -o yaml | grep -B 3 -A 2 -i legacy-host-allows-world
          - name: CILIUM_LEGACY_HOST_ALLOWS_WORLD
            valueFrom:
              configMapKeyRef:
                name: cilium-config
                optional: true
                key: legacy-host-allows-world
$ kubectl get cm cilium-config -n kube-system -o yaml | grep -i legacy-host-allows-world
  legacy-host-allows-world: "false"

Mitigation¶

Users who are not reliant upon IP-based health checks for their kubernetes pods may mitigate this issue on earlier versions of Cilium by adding the argument --allow-localhost=policy to the Cilium DaemonSet for the Cilium container. This prevents the automatic insertion of L3 allow policy in kubernetes environments. Note however that with this option, if the Cilium Network Policy allows traffic from the host, then it will still allow access from the outside world.

$ kubectl edit ds cilium -n kube-system
(Edit the "args" section to add the option "--allow-localhost=policy")
$ kubectl rollout status daemonset/cilium -n kube-system
(Wait for kubernetes to redeploy Cilium with the new options)

Solution¶

Cilium 1.1 and later only classify traffic from a process on the local host as from the host entity; other traffic that is masqueraded to the host IP is now classified as from the world entity (reserved:world label). Fresh deployments using the Cilium 1.1 YAMLs are not affected.

Affected users are recommended to upgrade using the steps below.

Upgrade steps¶

1. Redeploy the Cilium DaemonSet with the YAMLs provided with the Cilium 1.1 or later release. The instructions for this are found at the top of the Upgrade Guide.

2. Add the config option legacy-host-allows-world: "false" to the Cilium ConfigMap under the “data” paragraph.

   $ kubectl edit configmap cilium-config -n kube-system
   (Add a new line with the config option above in the "data" paragraph)
   

3. (Optional) Update the Cilium Network Policies to allow specific traffic from the outside world. For more information, see Network Policy.