k0s across subnets

The k0s cluster in this repo spans nodes on multiple host LANs:

RH461 (10.112.113.128/25): controller erwin, workers levi, hange on kyojin; workers naota, canti on atomsk (plus haruko, a reserved-but-unbuilt 5th controller). Node IPs per fleet.yaml.
AQ114 (10.112.12.64/26): controller zagreus, workers megaera, thanatos on theia. Hyperion’s odin/thor/loki live. Future 4th AQ114 host’s trio reserved as osiris/isis/horus (inventory-only until racked).

The two subnets are L3-routable to each other (campus routing) but they are separate L2 broadcast domains. This document captures what works, what can break, and which tests to run before relying on the cluster.

What works without intervention

Control plane joins. etcd peering (2379/2380), API server (6443), kubelet (10250) are all TCP unicast over routed paths. Subnet-spanning is fine for these.

Pod-to-pod traffic over the CNI overlay. Cilium (our CNI) encapsulates pod traffic in geneve UDP tunnels between nodes by default. The underlay just sees node-to-node UDP packets; pod IPs are hidden inside encapsulation. Pod CIDRs don’t need to be routable in the underlay — the overlay handles it. This is exactly the pattern designed for cross-subnet clusters.

Service VIPs. Intercepted locally per node by Cilium’s eBPF datapath. Independent of underlay topology.

What can break

Concern	Likely impact	Symptom
Campus firewall between subnets	Most likely failure mode. If campus blocks arbitrary UDP between subnets, the geneve tunnel (UDP 6081) won’t form.	Control plane joins fine, kubectl works, but pods on different nodes can’t reach each other. DNS resolution within cluster fails intermittently (CoreDNS pods on far-side nodes unreachable).
MTU mismatch	Overlay encapsulation adds ~50 bytes. Effective pod MTU drops from 1500 to ~1450.	Hangs on flows that don’t honor PMTUD — TLS to specific endpoints, VPN-tunneled apps, large HTTP responses. Smaller flows look fine.
NodePort / external access asymmetry	Campus firewall may treat 10.112.113.x and 10.112.12.x asymmetrically for ingress on the NodePort range (30000–32767).	Some external clients can reach NodePort on one subnet but not the other.
Cilium L2 announce (LoadBalancer / Gateway IPs)	L2 announce relies on ARP within a single broadcast domain. A pool announced from RH461 nodes is unreachable as L2 from AQ114, and vice versa — packets get there by L3 routing only.	Service `EXTERNAL-IP` reachable from clients on the same subnet via direct ARP; clients on the other subnet route via campus L3. Falls over when both the client and a backend are on the wrong side and the inter-site link is down. Per-site pool design + service classification covered in `k0s-gateway-placement`.
etcd quorum latency	etcd is sensitive to ~10ms+ latency. Campus subnet hop is usually sub-millisecond.	Probably fine, but worth checking. Manifests as etcd leader churn, slow API server response.

Tests to run after k0s comes online

1. Cilium tunnel formation

KUBECONFIG=/configs/configs/k0s/config kubectl -n kube-system exec ds/cilium -- cilium status --verbose

Look for Cluster health: N/N reachable. If any nodes are unreachable, that’s the geneve UDP being blocked between their subnets.

TODO (still open as of 2026-09): confirm Cilium is actually in tunnel mode (not native routing) — ansible/playbooks/k0s/setup-cluster.yaml sets ipv4NativeRoutingCIDR: 10.24.0.0/16 but does NOT set routingMode / tunnel explicitly, so we’re relying on the chart default. With nodes split across RH461 (10.112.113.128/25) and AQ114 (10.112.12.64/26) L3-separated segments, native routing would silently break pod-to-pod across sites (campus routers don’t know 10.24.0.0/16). Verify:

KUBECONFIG=/configs/configs/k0s/config kubectl -n kube-system get cm cilium-config -o yaml | grep -E 'tunnel|routing-mode'

Expect routing-mode: tunnel and tunnel-protocol: vxlan (or geneve). If routing-mode: native appears, that’s the bug — pin routingMode: tunnel explicitly in the Helm values.

2. Cross-host pod ping (smoke test)

Place one pod on a kyojin worker, one on a theia worker; ping by pod IP:

kubectl run nshoot-k --image=nicolaka/netshoot \
  --overrides='{"spec":{"nodeSelector":{"kubernetes.io/hostname":"levi"}}}' \
  -- sleep 600

kubectl run nshoot-t --image=nicolaka/netshoot \
  --overrides='{"spec":{"nodeSelector":{"kubernetes.io/hostname":"megaera"}}}' \
  -- sleep 600

# Wait for both Running, then:
kubectl exec nshoot-k -- ping -c 3 $(kubectl get pod nshoot-t -o jsonpath='{.status.podIP}')

If this fails: overlay isn’t traversing the inter-subnet path. Check Cilium status (test 1), then check campus firewall.

3. CoreDNS resolution from each subnet

kubectl exec nshoot-k -- nslookup kubernetes.default.svc.cluster.local
kubectl exec nshoot-t -- nslookup kubernetes.default.svc.cluster.local

Both should resolve. If one fails, CoreDNS pods on the other subnet aren’t reachable — same root cause as test 2 (overlay broken).

4. NodePort reachability from outside

If you’ve deployed any service as type: NodePort:

# From this jump box or another external client:
curl http://<kyojin-worker-IP>:<nodeport>/  # e.g. levi — IP per fleet.yaml
curl http://<theia-worker-IP>:<nodeport>/   # e.g. megaera — IP per fleet.yaml

If one subnet works and the other doesn’t, you’ve got the firewall asymmetry described above.

5. etcd cluster health

kubectl -n kube-system exec etcd-erwin -- etcdctl \
  --cacert=/var/lib/k0s/pki/etcd/ca.crt \
  --cert=/var/lib/k0s/pki/etcd/server.crt \
  --key=/var/lib/k0s/pki/etcd/server.key \
  endpoint status --cluster -w table

(Path may vary; this is a sketch.) Look for all controllers healthy with similar RAFT INDEX values. If a far-side controller’s index lags consistently (e.g. zagreus on theia from kyojin’s perspective), latency between subnets is too high — unlikely on campus but possible.

Mitigations if tests fail

Overlay UDP blocked (most likely root cause)

Three options, in order of preference:

Get the campus firewall to allow UDP 6081 between 10.112.113.0/25 and 10.112.12.64/26. Cleanest. Cilium’s default geneve transport works as designed.
Switch Cilium to VXLAN transport (UDP 8472). If campus already permits some UDP between subnets but not 6081, VXLAN’s port may be allowed. Configurable via Cilium HelmRelease — tunnelProtocol: vxlan.
Switch Cilium to native routing (no overlay). Requires pod CIDRs to be routable in the underlay — typically via BGP between routers and nodes. Heaviest option; only worth it if overlay UDP is permanently blocked and BGP infrastructure exists.

MTU issues

Set the cluster MTU explicitly to be safe:

Cilium HelmRelease: MTU: 1450 (or whatever the path supports minus encap overhead)
This forces pods to negotiate smaller packets and avoids fragmentation issues

NodePort asymmetry

Two paths:

Use a Service LoadBalancer (with MetalLB BGP, or external LB) instead of NodePort — gives you a single VIP routable from anywhere.
Document which NodePort URLs are reachable from which client networks; live with the asymmetry.

When NOT to redesign preemptively

For a homelab with controlled routing between subnets and Cilium overlay, k0s usually works without manual intervention. The ugliness is in the underlay: overlay UDP must traverse the inter-subnet path. Don’t switch CNIs or rebuild networking before testing. Run tests 1–3 above first; if they pass, you’re done.

If the tests fail, the failure mode points directly at the fix (firewall change vs. CNI config change), so debug-then-fix is faster than guess-and-rebuild.