k0s gateway placement

Consolidated ingress design + debug doc covering: how Cilium L2 announce works in this cluster, the three layered bugs we fixed to make it actually announce, where Gateway IPs live and how that interacts with site reliability, what the campus L3 router enforces against cross-subnet LB IPs (and why AQ114 is permissive while RH461 is not), DNS integration, and what remains for production HA. Companion notes:

cilium-gateway-l7 — what happens to the packet after it reaches the worker (TPROXY + nftables inet filter). The L7 layer above this note’s L2/L3 concerns.
k0s-cross-subnet — CNI overlay (geneve UDP) for pod traffic across LAN segments. Different layer from ingress.
k0s-cluster-rebalance — controller/worker distribution; AQ114 reliability bias.
internal-dns — azelf/mesprit/uxie split (all on-prem since 2026-05), rrchnm.internal zone authority on arceus.

TL;DR (state as of 2026-05-18)

L2 announce wasn’t actually announcing before 2026-05-17. Three layered fixes plus one campus-policy workaround got the Phase 2 ingress stack operational end-to-end:

Cilium feature off. l2announcements.enabled was missing from the Helm values — CRDs were inert. Fixed in setup-cluster.yaml.
Wrong-subnet announcer. Without a nodeSelector an AQ114 worker could win the lease for an RH461 pool IP and ARP on its own subnet. Fixed with rrchnm.org/site=<rh461|aq114> labels (sourced from rrchnm_site host vars via incus_remote) and a nodeSelector on the announcement policy. Each per-Gateway policy also needs a serviceSelector so both pool/policy pairs don’t match every Service.
Campus L3 router drops cross-subnet data plane to L2-announced IPs on RH461. Empirically confirmed via two-ended tcpdump 2026-05-17. The router ARPs cleanly but drops the SYN. Same router forwards hundreds of packets/sec to the announcing worker’s DHCP-bound primary IP in the same window. The differential is destination-IP-binding-table membership (RH461 is DHCP-snooped; LB IPs are phantom). AQ114 does NOT enforce the same check (confirmed 2026-05-18) — its static-IP subnet has no DHCP transactions to populate a snooping table, so cross-subnet to Cilium-L2 IPs on AQ114 works in both directions. The platform Gateway moved to AQ114 (10.112.12.121) 2026-05-18; other per-namespace Gateways stay on RH461 (acceptable — only RH461-side operators use them today, migration is opportunistic).

The remaining production-HA gap is storage: backend pods + their local-path PVs are still pinned to RH461 workers. Moving Gateway IPs alone is theater for RH461-survival HA until storage migrates too.

Background — how Cilium L2 + Gateway API actually expose services

Two independent CRDs, intentionally decoupled:

CiliumLoadBalancerIPPool — “here are IPs Cilium can hand out as Service EXTERNAL-IPs.” An address book; doesn’t make any IP reachable on its own.

CiliumL2AnnouncementPolicy — “from these nodes, on these interfaces, broadcast ARP claiming the IPs of services matching this selector.” The megaphone — who shouts about which IPs on which LAN.

A Gateway resource (Gateway API v1) materializes as an auto-created cilium-gateway-<name> LoadBalancer Service in the same namespace. That service draws an IP from a pool and is announced by whichever node the policy elects.

Why pool ≠ policy. The same pool can be announced from different node subsets; the same nodes can announce IPs from different pools. Multi-LAN clusters need this decoupling because L2 announce is per-broadcast-domain — only nodes physically on a LAN can ARP-respond for IPs in that LAN’s subnet.

The Cilium-agent on the elected node uses an AF_PACKET socket on the announce interface to respond to ARP requests for the LB IP. It does not add the IP to the kernel’s interface address list. Cilium’s BPF program on tc-ingress catches incoming packets to the LB IP and DNATs them to the backing pod via the Service translation table.

Current state (2026-05-18, post-AQ114-move)

Two pools, two policies, scoped by serviceSelector:

Resource	Spec
`CiliumLoadBalancerIPPool/local`	`10.112.113.150–199` (RH461)
`CiliumL2AnnouncementPolicy/local`	iface `enp5s0`, nodeSelector `rrchnm.org/site=rh461`, serviceSelector `io.cilium.gateway/owning-gateway NotIn [platform]`
`CiliumLoadBalancerIPPool/aq114`	`10.112.12.121–125` (AQ114, 5 IPs)
`CiliumL2AnnouncementPolicy/aq114`	iface `enp5s0`, nodeSelector `rrchnm.org/site=aq114`, serviceSelector `io.cilium.gateway/owning-gateway In [platform]`

Mirror the In/NotIn lists when moving additional Gateways to AQ114.

Live Gateways:

IP	Gateway	Namespace	Site	Hostnames
`10.112.12.121`	platform	platform	AQ114	forge, bao, zot, headlamp, heimdall, ca
`10.112.113.152`	observability	observability	RH461	grafana, hubble
`10.112.113.153`	alpine	vdi	RH461	alpine
`10.112.113.154`	arch	vdi	RH461	arch
`10.112.113.155`	debian	vdi	RH461	debian
`10.112.113.156`	kali	vdi	RH461	kali
`10.112.113.150`	owasp-juice-shop	devl	RH461	juice

The 10.112.113.150–199 RH461 range is reserved for our use at the campus DHCP layer (excluded from the lease pool). The 10.112.12.121–125 AQ114 range is carved from the static-IP block per inventory (.120 is baird on legacy XCP-ng, .126 reserved for a template VM).

Why AQ114 placement matters: site reliability

Per k0s-cluster-rebalance’s AQ114 reliability bias section, AQ114 is the resilient side — datacenter-grade power/cooling/network vs. RH461’s research-hall environment. The control plane is deliberately AQ114-biased:

Etcd quorum is 2-of-3 on AQ114 (zagreus on theia, odin on hyperion; jean on RH461). RH461 outage → cluster keeps quorum and stays read-write.
Worker majority is on AQ114 (4 of 6: megaera/thanatos on theia, thor/loki on hyperion).
DNS resolvers split: azelf on AQ114, mesprit + uxie on RH461 (mesprit on kyojin, uxie on ibuypwr).

For a service to survive RH461 outage, all three coupled layers (Gateway IP, backend pods, backend data) must live on AQ114. Moving just the IP is theater — see What “moving to AQ114” actually requires below.

Service-classification rule

Class	Site	Rationale
Critical / production — must survive RH461 outage	AQ114	Gateway IP, backend pods, and (where possible) data all on the resilient side
Internal tools / dev / scratch — VDI, juice-shop, dev workloads	RH461	Acceptable to lose during RH461 maintenance; no need to consume scarce AQ114 IPs

Current classification:

Service	Namespace	Class	Site
forgejo / openbao / zot / headlamp / heimdall	platform	Critical	AQ114 (Gateway moved 2026-05-18)
grafana / hubble	observability	Critical	RH461 (TBD — migrate opportunistically)
owasp-juice-shop	devl	Internal/dev	RH461 (stay)
alpine / arch / debian / kali (VDI)	vdi	Internal	RH461 (stay)

L2 announce — Issues 1 & 2 (in-cluster Cilium config)

Issue 1 — the feature was disabled

The two CRDs were defined in k0s/kube-system/cilium-lb-pool.yaml since the cluster was first built, but the Cilium Helm chart does not enable L2 announce by default. Without l2announcements.enabled: true in the values, cilium-agent doesn’t register the AF_PACKET listener and the CRDs are inert.

How to spot it:

kubectl get cm -n kube-system cilium-config -o yaml | grep enable-l2
# Healthy: enable-l2-announcements: "true"
# Broken: only `enable-l2-neigh-discovery: "true"` (different feature — kernel
#         neighbor discovery, NOT ARP announce)

kubectl get leases -n kube-system | grep cilium-l2announce
# Healthy: one lease per LB Service, each with a holder name
# Broken: empty (no agent is trying to win the election)

Fix — ansible/playbooks/k0s/setup-cluster.yaml Cilium Helm values:

l2announcements:
  enabled: true
k8sClientRateLimit:
  qps: 50
  burst: 100

The k8sClientRateLimit bump is upstream’s recommendation — the leader-election loop makes ~10 API calls/s/node and the default rate limit (5 qps) throttles it into a slow apply.

Issue 2 — wrong-subnet announcer

With the feature enabled but no node selector on the policy, leader election picks any worker. The RH461 pool could be won by an AQ114 worker, who’d then ARP for the RH461 IP on its own subnet. Nothing listens; nothing routes the IP cross-subnet.

How to spot it:

kubectl get leases -n kube-system | grep cilium-l2announce
# Compare each lease's holder against `kubectl get nodes -o wide` — the
# holder's primary IP must be in the same /N as the announced LB IP.

incus exec <kyojin>:<rh461-controller> -- /bin/bash -c \
  "ip neigh flush dev enp5s0 && curl -m 3 http://<rh461-lb-ip>/ -o /dev/null; \
   ip neigh show <rh461-lb-ip>"
# Healthy: REACHABLE <mac>
# Broken: INCOMPLETE

Fix — site labels sourced from inventory. ansible/inventory.yaml — each IncusOS host carries rrchnm_site: rh461|aq114. VMs inherit through incus_remote:

kyojin:   { ..., rrchnm_site: rh461 }
atomsk:   { ..., rrchnm_site: rh461 }
ibuypwr:  { ..., rrchnm_site: rh461 }
theia:    { ..., rrchnm_site: aq114 }
hyperion: { ..., rrchnm_site: aq114 }

ansible/playbooks/k0s/setup-cluster.yaml adds a “Label k0s workers with site” play after worker readiness, then k0s/kube-system/cilium-lb-pool.yaml matches the label on each policy’s nodeSelector (see Current state above for the full spec).

Two-policy gotcha — serviceSelector is mandatory

Once both local (RH461) and aq114 policies exist, without a serviceSelector they both match every LB Service. The union of candidate workers includes both sites, and an RH461 worker can win the lease for an AQ114-pool IP (or vice versa) that it can’t announce on its subnet.

Scope each policy with serviceSelector.matchExpressions on io.cilium.gateway/owning-gateway (auto-applied by Cilium to Gateway-owned Services):

# local policy — RH461 workers, all Gateway Services EXCEPT platform
serviceSelector:
  matchExpressions:
    - key: io.cilium.gateway/owning-gateway
      operator: NotIn
      values: [platform]

# aq114 policy — AQ114 workers, ONLY the platform Gateway
serviceSelector:
  matchExpressions:
    - key: io.cilium.gateway/owning-gateway
      operator: In
      values: [platform]

Mirror the In/NotIn lists when moving additional Gateways to AQ114.

Stale-lease gotcha — Cilium doesn’t auto-reset on Service IP change

When the platform Gateway’s IP moved from .157 (RH461) to .121 (AQ114), the old L2-announce lease stayed held by an RH461 worker (connie) for 11h+ until manually deleted:

kubectl delete lease -n kube-system cilium-l2announce-<ns>-<svc>

Forces a fresh election. Worth adding to pin-gateway-ips.yaml if Gateway IP moves become routine.

Resolver-cache gotcha — systemd-resolved holds negative cache

After DNS rewires, clients may still get NXDOMAIN for the old name until their resolver’s negative cache TTL expires. resolvectl flush-caches on the client clears it.

Issue 3 — cross-subnet LB IPs are dropped at the router data plane (not access-switch ARP)

After Issues 1+2 are fixed, L2 announce works within the announcing subnet. Cross-subnet, the diagnosis is more specific than the original “no route to host” framing suggested:

ARP plane: ✅ works. The campus L3 router does issue ARP requests for the LB IP and gets correct replies from the announcing worker.
Data plane: ❌ blocked on RH461. The router never forwards the original packet, even after learning the MAC.

Verified with (2026-05-17 — observability Gateway live at .150, `connie` as announcer)

Two-ended capture during a single curl odin → http://10.112.113.150/ (4s timeout, 3 SYN retries):

odin’s enp5s0 outbound:

23:18:07.102772  IP 10.112.12.114.46460 > 10.112.113.150.80: Flags [S], ...
23:18:08.109864  IP 10.112.12.114.46460 > 10.112.113.150.80: Flags [S], ...
23:18:09.133825  IP 10.112.12.114.46460 > 10.112.113.150.80: Flags [S], ...

SYN packets confirmed leaving odin. ✓

connie’s enp5s0 inbound (same window):

23:18:08.086157  ARP, Request who-has 10.112.113.150 tell 10.112.113.130
23:18:08.086175  ARP, Reply 10.112.113.150 is-at aa:fb:9f:4e:2a:30
23:18:09.100062  ARP, Request who-has 10.112.113.150 tell 10.112.113.130
23:18:09.100077  ARP, Reply 10.112.113.150 is-at aa:fb:9f:4e:2a:30
23:18:10.120387  ARP, Request who-has 10.112.113.150 tell 10.112.113.130
23:18:10.120506  ARP, Reply 10.112.113.150 is-at aa:fb:9f:4e:2a:30

.130 is the campus L3 router’s interface on the RH461 subnet. Router ARPed three times (matching odin’s three SYN retries), connie replied in ~15µs each. Zero TCP SYN packets from 10.112.12.114:46460 landed on connie. Data plane dropped between router and worker.

Meanwhile in the same window the capture recorded 462 packets of unrelated cross-subnet traffic — including a continuous TCP session 10.112.12.114:6443 ↔ 10.112.113.215:* (odin apiserver ↔ connie kubelet). Same router, same NIC, simultaneously forwarding hundreds of packets/sec to connie’s primary IP .215 while dropping every SYN to its L2-announced IP .150.

A/B summary

Destination	DHCP-bound?	Router behavior
`10.112.113.215` (connie primary)	Yes — bound to MAC-connie	Forwards 462+ packets/window ✓
`10.112.113.150` (Cilium L2-announced)	No — phantom IP	ARPs, gets reply, drops data plane ✗

Re-tested 2026-05-18 with two candidate LB IPs (.197 and .199) repointed via Cilium Gateway+Envoy. Same-subnet from jean: HTTP 200. Cross-subnet from odin: connect: no route to host for both. .199 is documented in inventory as “the only confirmed-coordinated MAC in the .150-.199 dummy pool” but behaved identically to .197. Repurposing existing reservation IPs as Cilium LB IPs doesn’t get past the cross-subnet drop. Two possible mechanisms (we can’t disambiguate without admin disclosure):

Coordination for .199 was removed when kyogre relocated to .133 (binding gone stale).
Strict ARP/MAC-binding validation: even if the IP is in the snooping table, ARP responder MAC must match the original reservation MAC.

The likely security mechanism

Campus is almost certainly running an enterprise-managed-network stack typical of security-as-a-service offerings:

Layer	What it does	What it protects against
DHCP snooping	Switch watches DHCP transactions per VLAN; builds binding table `(port, MAC, IP, lease-time)`	Establishes ground truth for legitimate `(MAC, IP)`
IP Source Guard (IPSG)	At access-port ingress, drops packets whose `(src-MAC, src-IP)` mismatches binding	Source-IP spoofing from compromised hosts
Dynamic ARP Inspection (DAI)	Validates ARP against binding table, drops mismatches	ARP spoofing / MITM
L3 router destination-IP egress filter	At router egress into a VLAN, drops packets whose destination IP isn’t in the binding table	Routes to phantom/unowned IPs — this is what blocks our LB pool

The first three are standard defense-in-depth. The fourth is more aggressive — a destination-side counterpart to IPSG. Cilium L2 announce ARPs for an IP with no DHCP provenance, which the binding table is designed to flag.

2026-05-18 — AQ114 cross-subnet confirmed working

Tested the symmetric case: move the platform Gateway to a Cilium L2-announced IP in the AQ114 pool (10.112.12.121), then curl from RH461.

incus exec hyperion:odin -- curl --cacert ... https://forge.rrchnm.internal/ → HTTP/1.1 200 OK, full chain verifies. ✓ (Same-subnet AQ114→AQ114.)
incus exec kyojin:jean -- curl --cacert ... https://forge.rrchnm.internal/ → HTTP/1.1 200 OK, full chain verifies. ✓ Cross-subnet RH461→AQ114 — works.
openssl s_client ... -CAfile /usr/local/share/ca-certificates/rrchnm-internal-ca.crt → Verify return code: 0 (ok).

This proves the snooping/destination-binding policy is per-VLAN, not symmetric. AQ114 doesn’t enforce it (no DHCP transactions to populate the binding table, so no enforcement to apply). Reshapes the recommendation tree below — Flavor 6 (LB on AQ114) becomes the first-line option.

(The analysis of what a scoped campus-filter exemption would loosen — and the drafted request wording further below — stay in the operator note, not published here.)

Alternatives to the exemption ask

Flavor 1 — Macvlan/Multus CNI for k0s pods

Add Multus alongside Cilium; pod gets a secondary NIC on the LAN with a DHCP-bound IP. Each service consumes a MAC reservation. Burns IPs fast; per-pod TLS or sidecar required; doesn’t multiplex hostnames per IP. Not recommended.

Flavor 2 — NodePort + DNS pointing at worker primary IPs

Services via type: NodePort (30000–32767, already in workers’ nftables allow-range). DNS A records → worker primary IPs. Cross-subnet works because primaries are DHCP-bound. Ugly ports in URLs unless we run a proxy in front. No TLS terminator help from this layer. Half-measure.

Flavor 3 — Caddy reverse-proxy on the existing Incus VM fleet

We already operate 5 caddy_proxies (kyogre/groudon on kyojin/RH461; articuno/zapdos on hyperion; moltres on theia/AQ114). All have DHCP-bound primary IPs, ca_root_cert in trust store, egress to internal CoreDNS + heimdall through the existing nftables egress lockdown. Adding reverse-proxy stanzas reuses an existing pattern (caddies already proxy LAMP backends). Caddy has built-in ACME — can fetch certs directly from heimdall.

Net effect: cross-subnet works (primary IPs); TLS terminates at Caddy with auto-renewing certs; Gateway-API + Cilium-L2 + TPROXY + site-labels stack becomes irrelevant for external ingress; cert-manager becomes optional. Was the recommended path before the AQ114 test; still viable if you want ingress entirely outside k8s.

Flavor 4 — Gateway API impl on hostNetwork DaemonSet

Different Gateway controller (Envoy Gateway, NGINX Gateway Fabric, Istio Gateway) as DaemonSet with hostNetwork: true, exposing on each worker’s primary IP. DNS round-robins. Same cross-subnet property as Flavor 3; keeps Gateway-API ergonomics. Worth considering if staying k8s-native but unsure about Cilium Gateway specifically.

Flavor 5 — Status quo + campus exemption

The original ask path. Real trade-off; “no longer needed” given Flavor 6 works.

Flavor 6 — Cilium L2 LB on AQ114 (the path we took 2026-05-18)

Keep Cilium Gateway, keep the per-namespace Gateway shape, just put the LB pool on the AQ114 subnet. AQ114’s router doesn’t enforce the destination-IP binding check that RH461 does (no DHCP = no snooping table), so Cilium-L2-announced IPs in this range are reachable cross-subnet from both sites without any campus-side change.

Net effect: cross-subnet works both directions; Gateway API + Cilium L2 + TPROXY + site-labels stack stays in place; minimal IaC delta; per-namespace Gateways preserve standard ergonomics; no security-posture conversation with network admin.

IaC delta (from the 2026-05-18 session):

New AQ114 pool + policy in k0s/kube-system/cilium-lb-pool.yaml (with serviceSelector on each policy — see two-policy gotcha above)
Per-Gateway move: spec.addresses in k0s/<ns>/gateway.yaml to an AQ114 IP; update the matching gateways entry in ansible/vars/k0s-gateways.yaml in lockstep
pin-gateway-ips.yaml writes the lbipam annotation; CoreDNS template renders new A records

Limitation worth flagging: Flavor 6 works because AQ114’s static-IP subnet apparently doesn’t enforce the destination-IP egress filter. If campus networking ever tightens AQ114 to match RH461 (e.g. via explicit ACL rather than DHCP snooping), Flavor 6 reverts to the same problem. Worth checking in a year. Caddy-on-VM (Flavor 3) doesn’t have this risk — primary IPs are inside whatever enforcement model the campus uses.

Recommendation

If you need…	Do this
External access to k0s services from anywhere on campus, minimum IaC delta	Flavor 6 (LB pool on AQ114). Tested working. Per-namespace Gateways stay; move them to AQ114 IPs one-by-one.
Same, but with the Caddy fleet as single ingress story (less k8s, more existing-infra reuse)	Flavor 3 (Caddy reverse-proxy). Bigger pivot, strictly less infrastructure to operate.
To stay k8s-native but de-risk dependence on Cilium L2 announce	Flavor 4 (Gateway-on-hostNetwork DaemonSet).
To eliminate the cross-subnet limitation universally, any future LB consumer in either subnet	Flavor 5 (exemption ask) — only if Flavor 6 stops working AND Flavor 3 unacceptable.
To experiment with reservation re-use as an LB shortcut	Don’t. Tested 2026-05-18; doesn’t work.

The current cluster is functional. cert-manager Phase 2 is complete; certs issue end-to-end. The platform Gateway moved to AQ114 (10.112.12.121) on 2026-05-18 and is reachable from both sites. Remaining per-namespace Gateways (observability, devl, vdi) are still on RH461 — migrate opportunistically.

What “moving to AQ114” actually requires

Three coupled layers — Gateway IP, backend pods, backend data. Moving only one is theater for RH461-survival HA.

1. Gateway IP — done for platform

The 2026-05-18 sequence (replicable for other Gateways):

Confirm the AQ114 pool + policy + serviceSelector pattern is in place (see Current state).
Append the Gateway name to the aq114 policy’s In [...] list AND remove from the local policy’s NotIn [...] list (or add to NotIn if it wasn’t there).
Edit k0s/<ns>/gateway.yaml — spec.addresses to the new AQ114 IP.
Edit ansible/vars/k0s-gateways.yaml — gateways: entry’s IP in lockstep.
kubectl apply -k k0s/<ns>/ and kubectl apply -k k0s/kube-system/.
Delete the stale L2 lease: kubectl delete lease -n kube-system cilium-l2announce-<ns>-<svc> (see stale-lease gotcha above).
Re-run pin-gateway-ips.yaml to set the lbipam annotation on the regenerated Service.
Re-run setup-dns-servers.yaml --tags coredns to publish the new A records.
Smoke test from both sites: incus exec hyperion:odin -- curl ... and incus exec kyojin:jean -- curl ....

2. Backend pod placement — soft, easy to fix

nodeAffinity preferred on each Deployment/StatefulSet to favor site=aq114. Soft preference, not hard requirement — pods can still schedule on RH461 if AQ114 is full or all AQ114 nodes are draining.

3. Backend data placement — hard, the actual bottleneck

local-path is single-node, no replication. PVs are physical directories on whichever node first mounted the PVC. Affinity hints can’t move them — only manual data migration can.

Current platform-namespace state (snapshot 2026-05-14):

Pod	Pinned to	Site	Data
forgejo	sasha	RH461	gitea-shared-storage PV
zot	sasha	RH461	zot-pvc-zot-0 PV
openbao	connie	RH461	data-openbao-0 PV (sealed)
openbao-injector	sasha	RH461	(stateless)
headlamp	sasha	RH461	(stateless)

Implication: the platform Gateway IP move to AQ114 alone buys ~zero HA. RH461 dies → all platform services die regardless of where the gateway IP lives, because backend pods can’t reschedule to AQ114 (data is on RH461 disks).

Migration paths per stateful service

Service	Path	Effort	Notes
headlamp	Delete pod, let it reschedule with `nodeAffinity: site=aq114`	trivial	Stateless
openbao	Stop pod → `rsync` data dir from connie to an AQ114 node’s local-path → recreate PVC pinned to AQ114	medium	Sealed vault data intact across rsync; unseal flow unchanged
zot	Same rsync approach	medium	OCI blobs; quiesce pushes during copy
forgejo	Same rsync; OR migrate to external postgres on a LAMP backend + S3 attachments	medium → high	Postgres path is the right long-term move; sqlite is a smell at this scale

Or: step back from `local-path` entirely

If RH461-vs-AQ114 HA is a real goal:

App-native externalization — forgejo → external postgres (LAMP fleet) + S3 attachments; zot → S3 backend. Skips the PV problem entirely. Most surgical.
NFS from a fileserver on AQ114 — single SPOF, but on the resilient side. Easy to wire, easy to back up.
Replicated block storage (Longhorn / OpenEBS Mayastor / Rook-Ceph) — proper HA, no SPOF. Heavyweight; defer until simpler options exhausted.

OpenBao is awkward — file storage backend is simplest but not replicable. The HA story is “switch to integrated Raft storage” (built-in, multi-replica) or “use external storage like Postgres” — values-file change of medium scope.

DNS implications

One A record per hostname, pointing at whichever Gateway IP the service ends up on. No dual records, no split-horizon. Clients on the “wrong” site reach the IP via campus L3 routing transparently.

forge.rrchnm.internal     A  10.112.12.121     ; AQ114 platform
bao.rrchnm.internal       A  10.112.12.121
zot.rrchnm.internal       A  10.112.12.121
headlamp.rrchnm.internal  A  10.112.12.121
grafana.rrchnm.internal   A  10.112.113.152    ; RH461 observability (today)
juice.rrchnm.internal     A  10.112.113.150    ; RH461 devl (stays)
alpine.rrchnm.internal    A  10.112.113.153    ; RH461 vdi (stays)

Resolver placement is already cross-site (azelf on AQ114, mesprit + uxie on RH461 on kyojin + ibuypwr respectively) so name resolution survives an AQ114 outage; an RH461 outage drops to a single resolver on AQ114.

IP pinning + Ansible-driven DNS A records — committed plan

The Gateway resource’s auto-allocated IP isn’t a stable thing to write A records against. Pinning via spec.addresses makes them deterministic, which lets Ansible generate the matching *.rrchnm.internal A records from a parallel map. Two coordinated edits per gateway.

Cilium spec.addresses caveat — ignored, annotation workaround

Empirically confirmed 2026-05-17 against Cilium 1.16.5; still active on the current 1.19.4 pin (not re-tested for upstream fix): Cilium’s Gateway controller does NOT translate Gateway.spec.addresses to the underlying LoadBalancer Service (cilium-gateway-<gateway>). The Service has no loadBalancerIP or externalIPs; LB-IPAM then assigns IPs sequentially from the pool, drifting the Gateway away from its pin. Annotations on the Gateway resource are also NOT propagated.

What does work: lbipam.cilium.io/ips: <ip> annotation on the Service itself. LB-IPAM honors it immediately.

Because the Service is generated by Cilium (not by our kustomize tree), the annotation has to be applied post-create. The workaround:

ansible/vars/k0s-gateways.yaml — gateways: list (namespace + gateway + ip + hostnames per Gateway). Single source of truth, loaded by both setup-dns-servers.yaml (renders zone A records) and pin-gateway-ips.yaml (writes lbipam annotations) — keeps DNS records and Service annotations from drifting.
ansible/playbooks/k0s/pin-gateway-ips.yaml — waits for each Cilium-generated Service to exist, then sets lbipam.cilium.io/ips.
Makefile target k0s-pin-gateway-ips — runs the playbook. Chained at the end of make k0s.

Until Cilium upstream lands spec.addresses translation, the gateways list is the third source of truth alongside spec.addresses in gateway.yaml. When the bug is fixed, delete the gateways ip field (or the whole pin playbook + Make target) — spec.addresses becomes authoritative again.

Source-of-truth model

The IP appears in two places (three while the Cilium workaround is active):

k0s/<ns>/gateway.yaml — spec.addresses: [{ type: IPAddress, value: <ip> }] on each Gateway.
ansible/vars/k0s-gateways.yaml — gateways: list with { namespace, gateway, ip, hostnames } per entry.

Each location has a header comment pointing at the other. Drift shows up as a bad DNS lookup (zone has new IP but gateway still serves the old, or vice versa) — visible during smoke testing, not silent.

IP allocation plan

Pool	Range	Use
RH461 stable	`10.112.113.150–.169`	Pinned gateways with A records
RH461 ephemeral	`10.112.113.170–.199`	Auto-allocated; no DNS
AQ114 stable	`10.112.12.121–.123`	Pinned gateways with A records
AQ114 ephemeral	`10.112.12.124–.125`	Auto-allocated; no DNS

The AQ114 stable sub-range is small (3 IPs) by design — only Critical-class services should consume it. Expand if more services move to AQ114.

gateway.yaml convention

# Pinned IP — matches the entry for this gateway in `gateways`
# (ansible/vars/k0s-gateways.yaml). Update both together.
# IP allocation plan: k0s-gateway-placement.md
---
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: platform
  namespace: platform
spec:
  gatewayClassName: cilium
  addresses:
    - type: IPAddress
      value: 10.112.12.121
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: Same

If Cilium can’t honor the request (IP outside any pool, already allocated), the Gateway status surfaces Addresses: Unavailable — fail-loud.

Drift detection

# What the zone says:
dig +short @10.112.12.69 forge.rrchnm.internal      # → 10.112.12.121

# What the cluster says:
kubectl -n platform get gateway platform -o jsonpath='{.status.addresses[0].value}'
                                                     # → 10.112.12.121

If those don’t match, one of the two source-of-truth files was edited without the other. make lint could grow a lint-gateway-ips subcommand if drift becomes recurring; not worth pre-building.

Why not Option B (single source of truth via templating)

Templating gateway YAMLs from a single Ansible map eliminates drift but costs more than it’s worth:

Breaks kubectl apply -k k0s/<ns> for dev iteration — gateways become generated artifacts requiring a render step.
Forces all gateway resources through the Ansible playbook, which doesn’t render manifests for Kustomize-applied paths.
Creates a layered system where gateway.yaml files are placeholders, not the real manifest — confusing for new readers.
Real drift cost is small: ~7 entries, slow churn, lookup failures fail loud.

Trigger to revisit: gateway count past ~20, or drift incidents more than once or twice a year.

Tests to run after future cluster rebuilds

# 1. Feature flag landed.
kubectl get cm -n kube-system cilium-config -o yaml | grep enable-l2-announcements
# Expect: "true"

# 2. Workers labeled.
kubectl get nodes -L rrchnm.org/site
# Expect each worker labeled rh461 or aq114 per inventory.

# 3. Lease holder is in the right subnet for its pool.
kubectl get leases -n kube-system | grep cilium-l2announce
# Holder name must be a worker whose primary IP is in the same /N as the LB IP.

# 4a. Same-subnet client reaches the LB IP.
incus exec kyojin:jean -- /bin/bash -c \
  "ip neigh flush dev enp5s0 && curl -m 3 http://<rh461-lb-ip>/ -o /dev/null && \
   ip neigh show <rh461-lb-ip>"
# Expect REACHABLE + a valid HTTP response.

# 4b. Cross-subnet client reaches an AQ114 LB IP (Flavor 6 working).
incus exec kyojin:jean -- bash -c \
  "curl --connect-timeout 4 -o /dev/null -sw 'http=%{http_code}\n' \
     http://<aq114-lb-ip>/"
# Expect 200 (or service-specific 404 for unknown Host). 000 = Flavor 6 broke.

# 4c. Cross-subnet from AQ114 to RH461 LB IP — STILL EXPECTED TO FAIL until
#     campus snooping bindings or BGP. Documented limitation, not a regression.
incus exec hyperion:odin -- bash -c \
  "curl --connect-timeout 4 -o /dev/null -sw 'http=%{http_code}\n' \
     http://<rh461-lb-ip>/"
# Expect 000.

# 5. cert-manager admission webhook is disabled (see cilium-gateway-l7.md).
kubectl get deploy -n cert-manager cert-manager-webhook \
  -o jsonpath='{.spec.replicas}{"\n"}'
# Expect: 0
kubectl get validatingwebhookconfiguration cert-manager-webhook \
  -o jsonpath='{.webhooks[*].failurePolicy}{"\n"}'
# Expect: Ignore

# 6. End-to-end stack health — Gateway → Envoy → cert-manager → TLS chain.
kubectl get certificate -n platform platform-tls \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}{"\n"}'
# Expect: True

Failure-mode matrix

For a service classified Critical (gateway on AQ114, pods on AQ114, data on AQ114):

Failure	Gateway	Backend	Client experience
RH461 host dies	up (envoy on AQ114 node)	up (pods on AQ114 nodes)	service works, all clients reach via campus L3
AQ114 host dies	down (envoy can’t ARP from RH461)	down (pods can’t reschedule, data on AQ114 disks)	service down — but cluster also degraded (etcd lost quorum)
Inter-site link dies	up	up	RH461 clients can’t reach AQ114 IP; AQ114 clients fine
Single AQ114 node dies	re-elects another AQ114 node	reschedules if PVC isn’t on dead node	brief blip, then fine

For a service classified Internal (gateway on RH461, pods anywhere):

Failure	Gateway	Backend	Client experience
RH461 host dies	down	depends on pod placement	service down; acceptable per classification
AQ114 host dies	up	up if pods on RH461	service works
Inter-site link dies	up	up	AQ114 clients can’t reach RH461 IP; RH461 clients fine

Migration sequence (recommended)

Cheap, no-behavior-change steps first.

IP pinning + Ansible-driven DNS landed. ✓ Done.
Coordinate AQ114 IP range with network admin. ✓ Done — 10.112.12.121–.125 carved from inventory.
Label nodes — kubectl label node ... site=.... ✓ Done via setup-cluster.yaml.
Apply AQ114 pool + policy (CiliumLoadBalancerIPPool + CiliumL2AnnouncementPolicy with serviceSelector). ✓ Done.
Migrate one critical service end-to-end as a proof — recommend headlamp first. ✓ Done — platform Gateway moved end-to-end 2026-05-18; cert chain verifies from both sites.
Decide storage strategy for the three stateful platform services before moving them. PENDING — migrating their Gateways without resolving storage placement is theater for RH461-survival.
Migrate stateful services per the chosen storage path. PENDING — long pole.
Update DNS A records as each service migrates. Done in lockstep with each move via the gateways map.

Optional, opportunistic: migrate observability/devl/vdi Gateways to AQ114 IPs if cross-subnet reach becomes needed for them too. Today only RH461-side operators use them, so not urgent.

What NOT to do:

Don’t migrate gateway IPs ahead of backend pods + data. The intermediate state (gateway on AQ114, backend on RH461) is no better than current state — both die when RH461 dies.
Don’t promote mabase (atomsk’s anticipated controller) to a voting controller without first deciding whether to preserve the AQ114 etcd bias. Adding a 4th controller on RH461 dilutes 2-of-3 AQ114 quorum to 2-of-4 (still works, but RH461 outage now risks losing 2 controllers if jean+mabase are both there).

Open questions

Konnectivity for cluster egress. k0s normally wires the apiserver’s EgressSelectorConfiguration so webhook traffic (the cluster flow type) tunnels through konnectivity-agent on workers, which dial Service IPs from the pod-network side where Cilium BPF works. In this cluster the only DIAL_REQs on konnectivity-agents are for kubelet (:10250), never webhooks. If konnectivity-routed webhooks could be made to work, Issue 3 would no longer matter for admission webhooks. Worth a session digging into k0s’s actual EgressSelectorConfiguration content.
Switching to BGP would eliminate Issue 3 entirely if campus networking would peer with us. Out of scope for current Phase 2.
AQ114 enforcement parity risk. Flavor 6 works because AQ114 doesn’t enforce the destination-IP egress filter. If campus tightens AQ114 (explicit ACL rather than DHCP snooping), Flavor 6 reverts to the same problem as RH461. Worth re-testing annually.
In-subnet TCP probe failure during initial L2 verification. During the 2026-05-17 diagnostic, curl jean → http://10.112.113.150/ returned http=000 even though ARP resolved REACHABLE. The observability Gateway had been live for ~3 min with no HTTPRoute backend programmed. Worth retesting after a real backend is deployed to confirm whether this was transient envoy programming or actual misconfiguration; the cross-subnet diagnosis held regardless because we were looking at SYN arrival on the wire.