Skip to content
RRCHNM Infra

k0s networking

The cluster runs Cilium as CNI with kube-proxy replacement and Gateway API ingress (HTTPRoute/GRPCRoute, never Ingress resources). LoadBalancer IPs come from Cilium L2 announcement pools scoped per trust tier — aq114-public (the single address fronting the public edge), aq114-internal, and RH461 local for the devl-vdi tier; the ranges live in k0s/kube-system/cilium-lb-pool.yaml. Cross-subnet behavior of the L2 announcement model is its own story: cross-subnet.

Why 10.24.0.0/16 for the pod CIDR

Cilium uses ipv4NativeRoutingCIDR as a policy boundary, not just an address pool — it answers “is this destination in-cluster?” Traffic to a destination inside the CIDR is treated as intra-cluster and not SNAT’d; traffic to a destination outside gets masqueraded to the node’s LAN IP.

If any node’s LAN IP falls inside the pod CIDR, that SNAT decision flips wrong: pod→node traffic is classified as in-cluster, not SNAT’d, and the reply from a hostNetwork process has no route back to the pod IP — the handshake never completes. (Cilium’s default 10.0.0.0/8 contains every fleet LAN, so it does exactly this.)

10.24.0.0/16 is disjoint from everything on the fleet:

RangeWhat it isOverlaps 10.24.0.0/16?
10.96.0.0/12 (10.96–10.111)k0s service CIDRno
10.112.12.64/26AQ114 node LAN (theia + hyperion VMs)no
10.112.113.0/25 + 10.112.113.128/25RH461 client + reservation LANsno
10.114.0.0/16TF-managed tfbr0 bridges on AQ114 hostsno
10.164.0.0/16TF-managed tfbr0 bridges on RH461 hostsno

When adding a new bridge or subnet anywhere in the fleet, don’t pick anything in 10.24.0.0/16 or 10.96.0.0/12 — that keeps the pod/service boundary clean.

Node-local load balancing (konnectivity HA)

Enabled via spec.network.nodeLocalLoadBalancing: { enabled: true, type: EnvoyProxy } in the cluster setup playbook. Each worker runs a local Envoy static pod, and k0s repoints the node’s worker components at it — kubelet → 127.0.0.1:7443, konnectivity-agent → 127.0.0.1:7132 — while Envoy holds connections open to all controllers’ apiservers (:6443) and konnectivity-servers (:8132).

Why — konnectivity exec on an HA control plane. With three controllers, each runs its own konnectivity-server advertising serverCount=3, and each controller’s apiserver proxies exec/logs/port-forward through its own local konnectivity-server. Unless every agent reaches every server, an agent attaches to only one (clientsCount=1) and churns retrying for the rest — so an exec that lands on a controller whose server has no attached agent fails with Internal error occurred: … No agent available. This is latent on any multi-controller k0s; it surfaced here the first time a workload needed apiserver→pod exec — the OpenBao bootstrap (bao status / bao operator unseal over kubectl exec). NLLB fixes it: each agent dials its node-local Envoy, which fans out to all three servers, so every server holds every agent (clientsCount=3, no churn) and exec works through any controller.

Why NLLB and not CPLB or an external LB. k0s offers three HA approaches, and they’re mutually exclusive (the other two set spec.api.externalAddress, which NLLB forbids):

  • CPLB (controlPlaneLoadBalancing, keepalived VIP) and an external LB balance external clients → control plane (a VIP for kubectl from outside) — and CPLB’s IPVS only fronts the apiserver port 6443, not konnectivity 8132. Neither helps the konnectivity-agent, which is a worker→control-plane connection.
  • NLLB is purpose-built for that worker-side path and explicitly covers the konnectivity-agent. Being self-contained per node (no VIP, no VRRP) it also sidesteps a topology trap: the controllers straddle two subnets — odin/zagreus on AQ114, erwin on RH461 — and a VRRP VIP can’t cross subnets, but a per-node Envoy just opens L3 connections to all three controller IPs.

Trade-off. NLLB requires spec.api.externalAddress to be unset, so the admin kubeconfig (k0s kubeconfig admin) and Cilium’s k8sServiceHost point at the primary controller rather than a VIP — i.e. no external-access HA for kubectl/Cilium. That’s the same posture as a plain non-LB cluster and is acceptable here (there was never an external control-plane LB; the break was purely worker-side).

Verify: the konnectivity-agent DaemonSet shows --proxy-server-host=localhost --proxy-server-port=7132, and an agent’s log settles at clientsCount=<#controllers> instead of looping on duplicate server. Functionally, consistent kubectl exec into any pod is the check.

Stack-applier kube-router / kube-proxy gotcha

After restarting k0scontroller, k0s’s stack applier may transiently (re)deploy the default kube-router and kube-proxy DaemonSets despite network.provider: custom and kubeProxy.disabled: true. If they appear (labels k0s.k0sproject.io/stack=kuberouter / =kubeproxy), delete them — kubectl delete daemonset -n kube-system kube-router kube-proxy — and subsequent reconciles correctly skip them.