The Home SDDC Lab is fully online and operational now, so I’ve begun deploying different Kubernetes clusters, leveraging different CNA projects. The plan is to get better exposure to what the community projects offer before I dive deeper into the VMware Tanzu offerings.
After deploying a rather vanilla Kubernetes 1.17.4 cluster with Flannel, I embarked on the journey of leveraging Cilium as the CNI and their cluster-wide network policies with eBPF.
All of my Kubernetes VMs are running Ubuntu 18.04 with 2 vCPU, 4Gb of RAM and 160Gb of storage. The networking for these VMs is simply a VLAN-backed network on my Cisco Catalyst 3960g switch — not leveraging NSX-T for these clusters yet. Finally, the installation of Docker and Kubernetes is done through the official app repos and automated using Ansible.
Cilium Installation
Note: Guide assumes Kubernetes software has already been deployed onto the bare metal or virtual machines you intend to leverage Cilium with.
The official documentation is available here. As I am finding with most Kubernetes projects, leveraging the official documentation is the first step, however there are usually other things required before things run smoothly.
Cilium relies upon BPF and needs to have a filesystem mounted local on the Kubernetes master node(s) and all the minion nodes.
/etc/systemd/system/sys-fs-bpf.mount [Unit] Description=Cilium BPF mounts Documentation=http://docs.cilium.io/ DefaultDependencies=no Before=local-fs.target umount.target After=swap.target [Mount] What=bpffs Where=/sys/fs/bpf Type=bpf [Install] WantedBy=multi-user.target
After creating the systemd unit file, the service must be enabled and started on each node.
$ sudo systemctl enable sys-fs-bpf.mount $ sudo systemctl start sys-fs-bpf.mount
From there, we can validate the service is running and the filesystem is mounted properly.
$ sudo systemctl status sys-fs-bpf.mount ● sys-fs-bpf.mount - Cilium BPF mounts Loaded: loaded (/etc/systemd/system/sys-fs-bpf.mount; enabled; vendor preset: enabled) Active: active (mounted) since Tue 2020-03-24 13:20:21 UTC; 1h 59min ago Where: /sys/fs/bpf What: bpffs Docs: http://docs.cilium.io/ Process: 5744 ExecMount=/bin/mount bpffs /sys/fs/bpf -t bpf (code=exited, status=0/SUCCESS) Tasks: 0 (limit: 4660) CGroup: /system.slice/sys-fs-bpf.mount Mar 24 13:20:21 cilium-master01 systemd[1]: Mounting Cilium BPF mounts... Mar 24 13:20:21 cilium-master01 systemd[1]: Mounted Cilium BPF mounts. $ mount | grep bpffs bpffs on /sys/fs/bpf type bpf (rw,relatime)
If you haven’t already, initialize the Kubernetes cluster and then deploy the Cilium layer.
$ sudo kubeadm init --pod-network-cidr 10.244.0.0/16 $ mkdir -p $HOME/.kube $ sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config $ sudo chown $(id -u):$(id -g) $HOME/.kube/config $ kubectl create -f https://raw.githubusercontent.com/cilium/cilium/1.7.1/install/kubernetes/quick-install.yaml serviceaccount/cilium created serviceaccount/cilium-operator created configmap/cilium-config created clusterrole.rbac.authorization.k8s.io/cilium created clusterrole.rbac.authorization.k8s.io/cilium-operator created clusterrolebinding.rbac.authorization.k8s.io/cilium created clusterrolebinding.rbac.authorization.k8s.io/cilium-operator created daemonset.apps/cilium created deployment.apps/cilium-operator created
Check the status of the Cilium deployment.
$ kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system cilium-operator-6547f48966-wd7hl 0/1 Pending 0 2m39s kube-system cilium-pjdvr 1/1 Running 0 2m39s kube-system coredns-6955765f44-qn5vq 1/1 Running 0 10m kube-system coredns-6955765f44-r9872 1/1 Running 0 10m kube-system etcd-cilium-master01 1/1 Running 0 10m kube-system kube-apiserver-cilium-master01 1/1 Running 0 10m kube-system kube-controller-manager-cilium-master01 1/1 Running 0 10m kube-system kube-proxy-wwbzg 1/1 Running 0 10m kube-system kube-scheduler-cilium-master01 1/1 Running 0 10m
Proceed to add in the Kubernetes minions to the cluster and watch their respective Cilium pieces deploy and come online.
$ kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system cilium-j5q2h 0/1 Init:0/1 0 28s kube-system cilium-operator-6547f48966-wd7hl 0/1 Pending 0 5m44s kube-system cilium-pjdvr 1/1 Running 0 5m44s kube-system coredns-6955765f44-qn5vq 1/1 Running 0 13m kube-system coredns-6955765f44-r9872 1/1 Running 0 13m kube-system etcd-cilium-master01 1/1 Running 0 13m kube-system kube-apiserver-cilium-master01 1/1 Running 0 13m kube-system kube-controller-manager-cilium-master01 1/1 Running 0 13m kube-system kube-proxy-wwbzg 1/1 Running 0 13m kube-system kube-proxy-zrwsn 0/1 ContainerCreating 0 28s kube-system kube-scheduler-cilium-master01 1/1 Running 0 13m $ kubectl get nodes NAME STATUS ROLES AGE VERSION cilium-master01 Ready master 15m v1.17.4 cilium-node01 Ready <none> 109s v1.17.4
After adding all of the nodes, you are all set to start deploying applications to the Cilium Kubernetes cluster.
$ kubectl get pods -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-system cilium-5hrfc 1/1 Running 0 33m kube-system cilium-d8ztv 1/1 Running 0 29m kube-system cilium-hl9dx 1/1 Running 0 8m28s kube-system cilium-j5q2h 1/1 Running 0 65m kube-system cilium-jqj86 1/1 Running 0 62m kube-system cilium-operator-6547f48966-wd7hl 1/1 Running 0 70m kube-system cilium-pjdvr 1/1 Running 0 70m kube-system coredns-6955765f44-qn5vq 1/1 Running 0 79m kube-system coredns-6955765f44-r9872 1/1 Running 0 79m kube-system etcd-cilium-master01 1/1 Running 0 79m kube-system kube-apiserver-cilium-master01 1/1 Running 0 79m kube-system kube-controller-manager-cilium-master01 1/1 Running 0 79m kube-system kube-proxy-5c8wz 1/1 Running 0 29m kube-system kube-proxy-m6wzc 1/1 Running 0 62m kube-system kube-proxy-mwndd 1/1 Running 0 8m28s kube-system kube-proxy-rhk9x 1/1 Running 0 33m kube-system kube-proxy-wwbzg 1/1 Running 0 79m kube-system kube-proxy-zrwsn 1/1 Running 0 65m kube-system kube-scheduler-cilium-master01 1/1 Running 0 79m $ kubectl get nodes NAME STATUS ROLES AGE VERSION cilium-master01 Ready master 95m v1.17.4 cilium-node01 Ready <none> 81m v1.17.4 cilium-node02 Ready <none> 78m v1.17.4 cilium-node03 Ready <none> 49m v1.17.4 cilium-node04 Ready <none> 45m v1.17.4 cilium-node05 Ready <none> 24m v1.17.4
Next post will be showing how I deployed Hubble and validated it was functioning properly in the cluster.
Enjoy!
