Cilium Gateway L7 — TPROXY vs nftables

Companion to k0s-gateway-placement. That note covers how a packet gets to a Cilium-elected worker (ARP propagation, DHCP snooping, cross-subnet router policy). This note covers what happens to the packet after it arrives — and why our Gateway listened, our Envoy was healthy, and our HTTPRoutes were Accepted, yet the Gateway returned nothing for two hours of bring-up.

TL;DR

Cilium implements Gateway API L7 LB by redirecting LB-IP traffic to a per-node Envoy process via Linux TPROXY. The redirect lives in the kernel ip filter table (installed by Cilium via the iptables-nft compatibility shim), but the redirected packets ALSO traverse our inet filter table (installed by the nftables role) — and our input chain default-drops anything not explicitly allowed. Result: every SYN that arrived at a worker for an LB IP got marked by Cilium, TPROXY-targeted at the local Envoy port, then dropped by our nftables before reaching the listening socket. Envoy never saw the connection, no SYN-ACK came back, every consumer (apiserver self-checks, browsers, curl, cert-manager solver callbacks) hit connection timed out.

Fix: one line in ansible/roles/nftables/templates/nftables.conf.j2 under the k0s_workers block:

meta mark & 0x0f00 == 0x0200 accept comment "Cilium L7 proxy (TPROXY)"

That matches Cilium’s tproxy-mark (0x200), accepting redirected packets in the inet filter table so they actually reach Envoy.

This was the last invisible piece of cert-manager Phase 2 — once it landed, the certificate issued cleanly. Prior to 2026-05-17, no Cilium Gateway in this cluster had ever served traffic. Every HTTPRoute apply returned Accepted=True, the Gateway showed Programmed=True, but the data plane was silently going nowhere.

How Cilium’s L7 LB actually routes a packet

The data path from curl http://10.112.113.157/ → an HTTPRoute backend pod:

L2 announce. Some RH461 worker (sasha or connie, the lease holder) ARPs claiming 10.112.113.157. The campus access switch learns the IP→MAC mapping by snooping the GARP. The campus L3 router, if it has the IP in its DHCP/snooping binding table, will forward to that MAC. (See k0s-gateway-placement for the dance and what can break here.)
Packet arrives at the elected worker’s enp5s0. Destination IP 10.112.113.157. Source IP whatever (could be a host on the LAN, could be a pod-network IP if the pod’s egress SNATs).
Cilium’s BPF tc-ingress program runs. It sees the destination is a Service VIP marked as L7 LB. It sets a packet mark of the form 0x{port_hex}0200 where port_hex is the local Envoy listener port for this specific Gateway. Example: for the platform Gateway with Envoy on 127.0.0.1:18303, mark = 0x7f470200 (0x7f47 = 32583… wait, 18303 = 0x477f, so the mark is 0x{port}0200 byte-reversed, i.e. 0x7f470200 — the proxy port encodes as a high-bytes hex in the mark).

The packet enters the kernel’s mangle PREROUTING. Cilium has installed (via iptables-nft):

-A CILIUM_PRE_mangle ! -o lo -m socket --transparent -m mark ! ...
  -j MARK --set-xmark 0x200/0xffffffff
-A CILIUM_PRE_mangle -p tcp -m mark --mark 0x7f470200
  -j TPROXY --on-port 18303 --on-ip 127.0.0.1 --tproxy-mark 0x200/0xffffffff

The TPROXY action rewrites the destination to 127.0.0.1:18303 AND sets the tproxy-mark 0x200.

Kernel routing decision. ip rule (also Cilium-installed) says from all fwmark 0x200/0xf00 lookup 2004. Table 2004 contains local default dev lo. The kernel concludes the packet is destined for a local socket.
Packet enters the filter input chain. This is the failure point in our cluster — see next section.
Envoy accepts the connection. Its listening socket was opened with IP_TRANSPARENT, so it accepts even though the original destination IP wasn’t local. Envoy applies HTTPRoute matching (Host header → backend Service), proxies to a backend pod.

Steps 1–5 work without our touching anything. Step 7 works (verified by direct curl 127.0.0.1:18303 -H "Host: forge.rrchnm.internal" returning the backend response). Step 6 is where everything fell apart.

How our `input` chain killed it

ansible/roles/nftables/templates/nftables.conf.j2 defines a default-drop inet filter input chain. The k0s-worker-specific allowlist permits:

tcp dport 10250                    # kubelet
tcp dport 30000-32767              # NodePort range
udp dport 30000-32767              # NodePort (incl. neko WebRTC)
tcp dport 4240, 4244               # Cilium agent health, Hubble
udp dport 8472                     # Cilium VXLAN

…and that’s it. When the TPROXY’d packet arrives at INPUT with destination 127.0.0.1:18303 (a Cilium-chosen ephemeral proxy port that lives in neither the kubelet nor the NodePort range), nothing matches, and the chain’s default policy drop discards it.

Cilium’s own iptables-nft rules DO accept this kind of packet — CILIUM_INPUT has:

-A CILIUM_INPUT -m mark --mark 0x200/0xf00
  -j ACCEPT  -m comment "cilium: ACCEPT for proxy traffic"

But that lives in the ip filter table. Linux netfilter evaluates both the ip filter AND inet filter tables for the same packet. If either drops, the packet is dropped. Our inet filter drops.

The fix is to add an equivalent mark-match accept to our inet filter input chain. The mark mask 0x0f00 == 0x0200 matches every TPROXY-redirected packet regardless of which Gateway’s Envoy port it’s headed for.

How to spot this failure mode

The symptoms looked exactly like a network-layer drop, which is why it took a few hours to land on the right layer:

# 1. ARP for the LB IP resolves cleanly (rule out k0s-gateway-placement Issues 1+2):
incus exec kyojin:jean -- /bin/bash -c \
  "ip neigh flush dev enp5s0 && ping -c 1 -W 2 10.112.113.157; \
   ip neigh show 10.112.113.157"
# → 1 packets transmitted, 0 received (TCP/UDP fails, ICMP also fails)
# → 10.112.113.157 lladdr <some MAC> REACHABLE  ← ARP works

# 2. SYN packets arrive at the announcing worker's enp5s0:
incus exec kyojin:<announcing-worker> -- timeout 6 tcpdump -i enp5s0 -nn 'tcp and port 80 and host 10.112.113.157'
# → captures: 10.112.113.X.PORT > 10.112.113.157.80: Flags [S]
# → NO SYN-ACK in reply

# 3. Cilium's TPROXY rule counter is incrementing (rule out "Cilium not redirecting"):
CILIUM=$(kubectl get pods -n kube-system -l k8s-app=cilium --field-selector spec.nodeName=<announcing-worker> -o name | head -1)
kubectl exec -n kube-system $CILIUM -c cilium-agent -- iptables-save -t mangle \
  | grep "platform/cilium-gateway-platform/listener"
# → rule packets counter > 0  ← redirect IS firing

# 4. Envoy is healthy and would serve if reached:
incus exec kyojin:<announcing-worker> -- curl -m 3 -H "Host: forge.rrchnm.internal" \
  http://127.0.0.1:18303/
# → 200 OK  ← Envoy works fine on its own

Steps 2 + 3 + 4 together = the redirect is being installed AND firing AND the destination is healthy, but the connection never completes. That’s the TPROXY-into-default-drop signature.

nft list ruleset on the worker confirms the default-drop policy and the absence of any mark-match accept:

incus exec kyojin:<worker> -- nft list ruleset | grep -A 2 "chain input"
# → type filter hook input priority filter; policy drop;
#   iif "lo" accept
#   ct state established,related accept
#   …no mark match…

Why this stayed invisible for so long

The Gateway API was added to this cluster months ago. cilium-lb-pool.yaml, gateway.yaml, and every HTTPRoute reported clean state. No one had actually tried to load a Gateway-fronted hostname end-to-end until cert-manager’s admission webhook started trying to validate Certificate resources whose ACME flow depended on the Gateway working.

Three subsystems silently broken at the same time, each masked by the previous:

Subsystem	Broken because	Visible symptom
L2 announce	`l2announcements.enabled` missing from Cilium values	`kubectl get leases` empty
L2 announce site	No `nodeSelector` on the L2 policy	AQ114 worker won RH461 IP’s lease, ARP went nowhere
Gateway L7 LB	`inet filter` input drops TPROXY-redirected packets	Anyone hitting the LB IP got `connect: timeout`

Fixing L2 announce exposed the L2 site mismatch. Fixing the site exposed the inet-filter drop. Each fix made the cluster “more visibly broken” because consumers actually started trying to use the data path.

Permanent fix in IaC

ansible/roles/nftables/templates/nftables.conf.j2, under {% if 'k0s_workers' in group_names %}:

meta mark & 0x0f00 == 0x0200 accept comment "Cilium L7 proxy (TPROXY)"

The mask 0x0f00 matches just the Cilium proxy-mark nibble; we don’t pin to a specific port (Cilium picks ephemeral ones per Gateway). Comment intentionally name-checks Cilium L7 proxy (TPROXY) so the next person grepping for the rule finds it.

Applied via the same role-apply that lays down the rest of nftables: re-running any setup-* playbook with --tags init,nftables against k0s_workers re-renders + reloads.

Retiring the cert-manager hostNetwork pin (same-day follow-up)

The TPROXY fix above unblocked the L7 data plane for everything routing through the Gateway. But the cert-manager admission webhook sits above the Gateway in the architecture: it’s called directly by the apiserver during kubectl apply of cert-manager CRDs, and that call has to work from every apiserver — including the AQ114 controllers that can’t reach RH461 LB IPs cross-subnet (Issue 3 in k0s-gateway-placement).

The original workaround was to run the webhook on hostNetwork: true pinned to sasha, with the apiserver dialing https://sasha.rrchnm.internal:30260/validate. That worked, but cost a single-point-of-failure on sasha and a tight coupling between cert-manager and a specific node name. After the TPROXY fix landed and platform-tls issued cleanly, we revisited the webhook setup and disabled the admission webhook entirely instead.

The decision shape:

Approach	Net behavior	Operational cost
Webhook on ClusterIP (default) + `failurePolicy: Fail`	Every cert-manager CRD apply fails	Doesn’t work for k0s+Cilium
Webhook on hostNetwork + pin to `sasha`	Validation works from every apiserver	sasha SPoF; tight coupling to node name
Webhook on ClusterIP + `failurePolicy: Ignore`	Apiserver tries the call, times out in `timeoutSeconds`, proceeds	Idle pod cost + ~1s per apply latency
`webhook.replicaCount: 0` + `failurePolicy: Ignore`	No pod, apiserver tries, gives up immediately, proceeds	Zero

We picked row 4. The webhook’s three jobs (validation, mutation, conversion) all degrade gracefully:

Admission validation — gone, but covered by static kubeconform against cert-manager’s OpenAPI schema at IaC-review time. Catches typos in PR diff rather than at apply.
Default-field mutation — cert-manager’s controller has fallback defaults that kick in at reconcile time. Our platform-tls-cert.yaml already specifies every field explicitly, so the mutating webhook was a no-op for us anyway.
CRD version conversion — we only write v1 resources, and the cert-manager v1.20 CRDs don’t serve older versions by default.

What the controller still does (everything that actually matters for issuing certs):

Watches Certificate/Order/Challenge resources
Runs the full ACME flow against heimdall
Creates challenge HTTPRoutes/Pods/Services via Gateway API
Renews automatically when renewBefore triggers
Writes Secrets, rotates keys

The webhook is purely an admission-time concern; disabling it doesn’t affect the runtime.

IaC shape (post-retirement)

k0s/platform/cert-manager-values.yaml:

webhook:
  replicaCount: 0
  timeoutSeconds: 1

ansible/playbooks/k0s/deploy-platform.yaml runs a post-Helm task that patches both webhook configurations to failurePolicy: Ignore (the chart doesn’t expose failurePolicy as a value, so it’s a kubectl patch against the VWC + MWC):

- name: Patch cert-manager VWC/MWC to failurePolicy=Ignore (webhook disabled)
  ansible.builtin.command:
    cmd: >-
      kubectl patch {{ item.kind }} {{ item.name }} --type=json
      -p='[{"op":"replace","path":"/webhooks/0/failurePolicy","value":"Ignore"}]'
  loop:
    - { kind: validatingwebhookconfiguration, name: cert-manager-webhook }
    - { kind: mutatingwebhookconfiguration, name: cert-manager-webhook }

apply-network-policies.yaml has a comment block where the allow-apiserver-webhook CNP used to live, documenting the choice and explaining how to restore it if you ever re-enable the webhook.

Verified end-to-end

After the change: force-deleted platform-tls Secret → cert-manager controller re-issued it via the full ACME flow → new Secret materialized, Ready=True — no webhook touched. Applied a smoke-test Certificate/webhook-disable-smoke-test separately, went Ready=True in ~70 seconds. Both apply paths succeeded with failurePolicy: Ignore doing its job (apiserver tries the webhook, times out, proceeds).

If you ever want validation back

Two paths:

Re-enable the webhook — restore the hostNetwork+sasha pin in cert-manager-values.yaml, restore the allow-apiserver-webhook CNP, change the VWC/MWC patch task from Ignore to Fail. The pattern is preserved in git history if needed.
Add CI-side validation — kubeconform --schema-location 'https://raw.githubusercontent.com/jetstack/cert-manager/master/deploy/crds/{{ .ResourceKind }}.yaml' (or vendored copies) in a pre-merge hook. Catches the same typos at PR-review time rather than apply time. Doesn’t depend on the cluster being reachable from CI.

Path 2 is the modern shape; path 1 is the safety net.

Open questions

Why Cilium installs to ip filter and not inet filter. Probably history — iptables-nft exists exactly to translate legacy iptables rules to the modern nftables backend, but it targets the ip filter table by name. Cilium’s manager hasn’t been rewritten to install native nftables rules in the inet family. Worth tracking; a future Cilium release might fix this and let us drop the workaround.
The 24-hour cert duration. platform-tls was requested with duration: 2160h (90 days, renewBefore 15 days), but the cert heimdall issued has a 24-hour notAfter. Heimdall’s ACME provisioner caps it via defaults.json (tls.maxDuration or the provisioner-level claim). cert-manager auto-renews so this works, but the renewal cadence is now every ~12 hours instead of every ~75 days. Followup item; tune heimdall provisioner.
bao 307 redirect. OpenBao at bao.rrchnm.internal returns 307 on GET / — fine for Gateway routing, but its UI/API may need a Host-header-aware redirect target. Not a Gateway bug.