Building a Kubernetes cluster you would actually run in production

Managed Kubernetes is the right default. But some clusters have to live in your own data centre — sovereignty, GPU economics, or a Nutanix estate you already paid for — and then someone has to build the thing.

This is the build we do, with the reasoning left in. It uses kubeadm, because the alternative distributions are mostly opinions layered on top of exactly these steps, and understanding the steps is what lets you debug the distribution later.

The target: three control-plane nodes with stacked etcd, three or more workers, Cilium for networking, and a cluster that survives losing any single machine.

Prerequisites nobody writes down

Most failed cluster builds fail here, before kubeadm is ever invoked, and the error messages are unhelpful because the failure happens two layers down.

Kernel and networking

cat <<'EOF' | sudo tee /etc/modules-load.d/k8s.conf
overlay
br_netfilter
EOF
sudo modprobe overlay && sudo modprobe br_netfilter
 
cat <<'EOF' | sudo tee /etc/sysctl.d/99-k8s.conf
net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1
fs.inotify.max_user_instances       = 8192
fs.inotify.max_user_watches         = 524288
EOF
sudo sysctl --system

The two inotify values are the ones people leave out. Default limits are fine for a laptop and far too low for a node running a few hundred containers; you find out when the kubelet starts logging too many open files and pods stop getting config map updates, which looks like anything except an inotify problem.

Swap off, and mean it

sudo swapoff -a
sudo sed -i '/[[:space:]]swap[[:space:]]/ s/^/#/' /etc/fstab

Recent kubelet versions can tolerate swap under a feature gate. Do not use it for a first production cluster. The memory accounting that the scheduler and the OOM killer rely on becomes approximate, and approximate memory accounting produces incidents that are extremely hard to attribute.

Container runtime

containerd, with the systemd cgroup driver. This single setting causes more mysterious kubelet crash loops than anything else on the list, because a cgroup driver mismatch between kubelet and containerd produces nodes that register and then die under load.

sudo apt-get install -y containerd
containerd config default | sudo tee /etc/containerd/config.toml >/dev/null
sudo sed -i 's/SystemdCgroup = false/SystemdCgroup = true/' /etc/containerd/config.toml
sudo systemctl restart containerd

Verify rather than assume:

sudo containerd config dump | grep SystemdCgroup
#             SystemdCgroup = true

Control plane endpoint

Before kubeadm init, decide how clients reach the API server. This is a one-way door: it is baked into every certificate and every kubeconfig, and changing it later means regenerating both across the fleet.

Use a DNS name pointing at a virtual IP or load balancer, never a node address:

k8s-api.internal.example.com → 10.20.0.10 (VIP)
                             → 10.20.0.11:6443 (cp-1)
                             → 10.20.0.12:6443 (cp-2)
                             → 10.20.0.13:6443 (cp-3)

On bare metal, kube-vip running as a static pod on each control-plane node is the least moving parts. On Nutanix or a cloud, use the platform load balancer — health-check TCP 6443 rather than an HTTP path, because the API server rejects unauthenticated HTTP probes and your load balancer will conclude the whole cluster is down.

Initialising the first control plane

Use a configuration file, not flags. Flags are undocumented six months later; a file is reviewable, diffable, and reusable when you rebuild.

apiVersion: kubeadm.k8s.io/v1beta4
kind: ClusterConfiguration
kubernetesVersion: v1.34.0
clusterName: prod-eu-1
controlPlaneEndpoint: "k8s-api.internal.example.com:6443"
networking:
  podSubnet: "10.244.0.0/16"
  serviceSubnet: "10.96.0.0/12"
apiServer:
  extraArgs:
    - name: audit-log-path
      value: /var/log/kubernetes/audit.log
    - name: audit-log-maxage
      value: "30"
    - name: audit-policy-file
      value: /etc/kubernetes/audit-policy.yaml
  extraVolumes:
    - name: audit-policy
      hostPath: /etc/kubernetes/audit-policy.yaml
      mountPath: /etc/kubernetes/audit-policy.yaml
      readOnly: true
      pathType: File
    - name: audit-log
      hostPath: /var/log/kubernetes
      mountPath: /var/log/kubernetes
      pathType: DirectoryOrCreate
controllerManager:
  extraArgs:
    - name: bind-address
      value: 0.0.0.0
scheduler:
  extraArgs:
    - name: bind-address
      value: 0.0.0.0
etcd:
  local:
    dataDir: /var/lib/etcd
    extraArgs:
      - name: quota-backend-bytes
        value: "8589934592"
---
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
cgroupDriver: systemd
serializeImagePulls: false
maxPods: 150
evictionHard:
  memory.available: "500Mi"
  nodefs.available: "10%"
  imagefs.available: "15%"
systemReserved:
  cpu: "500m"
  memory: "1Gi"
kubeReserved:
  cpu: "500m"
  memory: "1Gi"

Four of those are the difference between a lab cluster and a production one.

Audit logging configured at init. Adding it later means editing static pod manifests on every control-plane node under time pressure, usually during the incident that made you want audit logs.

bind-address: 0.0.0.0 on controller-manager and scheduler. They bind to localhost by default, which means Prometheus cannot scrape them and you have no visibility into scheduling latency or controller queue depth.

quota-backend-bytes at 8 GiB. etcd's default 2 GiB limit is generous until it isn't, and a full etcd puts the entire cluster into read-only mode. Raising it early is free.

systemReserved and kubeReserved. Without these, the scheduler believes it can allocate 100% of node memory to pods. It cannot — the kubelet, containerd and sshd need some — so the node hits system OOM instead of graceful pod eviction, and you lose the kubelet along with the workload.

Now initialise:

sudo kubeadm init \
  --config kubeadm-config.yaml \
  --upload-certs \
  --skip-phases=addon/kube-proxy

--upload-certs stages the control-plane certificates in a Secret with a two-hour TTL so the other two nodes can join without you copying PEM files around.

--skip-phases=addon/kube-proxy is deliberate. We are about to install Cilium in kube-proxy replacement mode, and installing kube-proxy first means uninstalling it later, which leaves stale iptables rules that produce genuinely baffling connectivity bugs.

Networking: Cilium without kube-proxy

helm repo add cilium https://helm.cilium.io && helm repo update
 
helm install cilium cilium/cilium \
  --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=k8s-api.internal.example.com \
  --set k8sServicePort=6443 \
  --set ipam.mode=kubernetes \
  --set routingMode=native \
  --set ipv4NativeRoutingCIDR=10.244.0.0/16 \
  --set autoDirectNodeRoutes=true \
  --set bpf.masquerade=true \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true

routingMode=native with autoDirectNodeRoutes=true skips encapsulation entirely, so pod traffic is plain routed IP rather than VXLAN. Lower latency, no MTU arithmetic, and packet captures you can actually read. The requirement is that all nodes share a layer 2 domain — true in most single-rack and single-subnet deployments, and worth confirming with your network team before you commit.

kubeProxyReplacement=true replaces iptables service routing with eBPF. On a cluster with a few thousand services, iptables rule evaluation becomes a measurable latency cost; eBPF map lookups do not degrade the same way.

Confirm before proceeding:

cilium status --wait
kubectl get nodes
# NAME    STATUS   ROLES           AGE   VERSION
# cp-1    Ready    control-plane   3m    v1.34.0

A node stuck in NotReady after CNI installation is almost always a CIDR overlap. Your podSubnet must not collide with anything routable on the physical network — check before you build, not after.

Joining the rest

sudo kubeadm join k8s-api.internal.example.com:6443 \
  --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash> \
  --control-plane --certificate-key <key>

sudo kubeadm join k8s-api.internal.example.com:6443 \
  --token <token> \
  --discovery-token-ca-cert-hash sha256:<hash>

Three control-plane nodes, not two. etcd needs a quorum, and a two-member cluster has a quorum of two — so losing either node loses the cluster. Two control-plane nodes are strictly less available than one.

Verify it survives a failure

Before anything real runs here, prove the cluster does what you believe it does.

# etcd is healthy and has a leader
kubectl -n kube-system exec etcd-cp-1 -- etcdctl \
  --cacert /etc/kubernetes/pki/etcd/ca.crt \
  --cert /etc/kubernetes/pki/etcd/server.crt \
  --key /etc/kubernetes/pki/etcd/server.key \
  endpoint status --cluster -w table
 
# DNS resolves from inside a pod
kubectl run dnstest --image=busybox:1.36 --rm -it --restart=Never \
  -- nslookup kubernetes.default
 
# Pod-to-pod across nodes actually works
kubectl create deployment nginx --image=nginx --replicas=6
kubectl expose deployment nginx --port 80
kubectl run curl --image=curlimages/curl --rm -it --restart=Never \
  -- curl -s -o /dev/null -w '%{http_code}\n' http://nginx

Then take a control-plane node away — sudo systemctl stop kubelet && sudo reboot — and watch the API stay up. If it does not, you have found a load balancer health check problem, and you have found it on a Tuesday morning rather than during an incident.

Seven things before you hand it over

A cluster that passes the checks above is running. It is not yet ready for other people.

1. Storage. No CSI driver means no PersistentVolumeClaim. Nutanix CSI on Nutanix, ebs-csi-driver on AWS, Rook-Ceph or Longhorn on generic bare metal. Mark exactly one StorageClass as default.
2. Ingress. An ingress controller plus a way to get an external IP. On bare metal that means MetalLB or Cilium's own LB-IPAM; without it every LoadBalancer service sits in Pending forever.
3. etcd backups, off the cluster. etcdctl snapshot save on a timer, shipped to object storage, with a restore rehearsed at least once. An etcd snapshot on the same node as etcd is not a backup.
4. Certificate expiry. kubeadm certificates last one year. kubeadm certs check-expiration in a monitored cron job, because the failure mode is total and arrives without warning.
5. Resource governance. A LimitRange and a ResourceQuota per namespace. Without them one team's unbounded deployment evicts another team's production pods, and the scheduler was doing its job correctly.
6. NetworkPolicy default deny. Flat pod networking means every pod can reach every other pod, including your database. A default-deny policy per namespace plus explicit allows is an afternoon of work that closes most of the lateral movement surface.
7. An upgrade plan. Kubernetes ships three releases a year and supports roughly fourteen months. Decide now whether you upgrade quarterly, and write down who does it — otherwise you will be four versions behind before anyone notices, and the catch-up path is much worse than the routine one.

The honest summary

kubeadm init takes about four minutes. Everything in this post that is not kubeadm init is why the cluster is still standing in eighteen months.

If you take one thing: decide the control-plane endpoint and the pod CIDR before you type the first command. Both are baked into certificates and routing tables, both are genuinely painful to change, and both are decided in thirty seconds if you ask the right person early.