Skip to content
RRCHNM Infra

k0s public edge

Status: proposed (2026-08-27). Greenfield for the dynamic-dev-URL half; the HAProxy + public-Gateway half reverses a documented default in public-ingress.md (see “Relationship to public-ingress.md”). Not built.

Goal

Two things the current design does not give us:

  1. Deployed k0s services become publicly reachable without re-running make haproxy per service. Today every public/HAProxy-fronted hostname is an Ansible-managed frontend/backend map entry — adding one is a fleet playbook run. We want “add a public k0s service” to be an HTTPRoute committed to git and Argo-synced, nothing else.
  2. Dynamic dev URLs <hash>.devl.rrchnm.org — wildcard public hostnames, one per ephemeral deployment, routed dynamically inside k0s.

The architecture

Make the HAProxy layer a thin, STATIC L4 pass-through whose only job is the F5-return hairpin. All routing lives in k0s.

F5  129.174.134.71/72/73  (DNAT, source-preserving)
  → dedicated HAProxy VIPs .141/.142/.143      [k0s-public tier — see "IP collision" below]
      STATIC config: forward :80/:443 → the public Gateway's LAN IP. ONE backend.
      No SNI map, no per-host entries → never edited when a service is added.
  → k0s Cilium `public` Gateway  (LAN-only Cilium LB IP)
      terminates TLS, owns ALL routing + the *.devl.rrchnm.org wildcard listener
  → HTTPRoutes (Argo-synced)   ← add a public service HERE, never in Ansible

The win: HAProxy config is set-once. make haproxy (for this tier) runs to stand it up and never again for routing. New public service = new HTTPRoute.

Why this shape (and why it resolves the two blockers)

public-gateway.yaml calls the Cilium Gateway “unsuitable as a direct F5 backend” for two reasons; the thin-HAProxy front resolves both by construction:

  • F5 asymmetric return. F5 does source-preserving DNAT, so the backend must route replies back through the F5 next-hop. The HAProxy fleet already has the mangle/fwmark/ip rule machinery for exactly this (acme-off-fleet-clients §6 — ruleset details operator-side). Keeping HAProxy in the path means F5-return stays on the HAProxy nodes (proven) and the k0s workers see only symmetric LAN traffic — sidestepping the “port fwmark onto Cilium eBPF nodes, unproven” risk that public-ingress.md:90-96 flags.
  • Source-IP allowlist. HAProxy enforces it persistently (campus CIDR for dev); Cilium reverts loadBalancerSourceRanges, so the Gateway can’t. Coarse gating on HAProxy, fine routing on the Gateway.

Relationship to public-ingress.md (the reversal, stated plainly)

public-ingress.md is the design-of-record and chose direct F5 → public Cilium Gateway for k0s services (plan-k0s-lamp-hybrid.md already routes LAMP that way). It lists “add HAProxy in front of the public Gateway” only under Re-evaluation triggers (:479-484): “Public service SLO tightens to ‘must survive k0s upgrade’” and “WAF / rate-limit / IP-deny becomes a requirement.” This plan elects that trigger and makes HAProxy-fronting the default for the public edge.

Rationale for electing it now rather than the direct path:

  • The direct path’s F5-return-on-k0s-workers is unwired and unproven (:90-96); the HAProxy front makes it a non-issue.
  • The “add a service without Ansible” goal is satisfied either way (routing is HTTPRoutes regardless) — but the HAProxy front also gives the always-up L4 layer that survives k0s upgrades, which is the first documented trigger.
  • One extra hop is the acknowledged cost (:56-67 argued a front “adds nothing” — true for open services where envoy already terminates; false once F5-return + allowlist matter, which the :69-108 2026-06 limitation section concedes).

→ Extend public-ingress.md with a decision-log entry pointing here (do not rewrite its log in place).

The dedicated HAProxy tier

A NEW HAProxy role, distinct from the executing plan-haproxy-incusos.md fleet (that fleet is SNI-passthrough → the Caddy pool for non-k0s static sites; 3/5 nodes live, agni/rudra reserved-but-unbuilt). This tier is thin-L4 → the k0s Gateway. Same haproxy role, different backend + purpose.

  • Nodes: 2–3 IncusOS instances for VRRP HA on .141–.143. Open question: reuse the reserved agni/rudra slots (earmarked for the Caddy fleet in plan-haproxy-incusos.md) or stand up fresh nodes. → decision-log entry in plan-haproxy-incusos.md.
  • Config: one static frontend per VIP → one backend (the Gateway LAN IP), mode tcp passthrough. Carries the acme-off-fleet-clients §6 fwmark/route drop-in (they need it; k0s workers don’t, under this design). fwmark must not collide with Cilium’s 0x200 TPROXY marks (cilium-gateway-l7).

The k0s public Gateway (activate it, LAN-only, behind HAProxy)

  • Give it a LAN-only Cilium LB IP that HAProxy targets (reconcile: live manifests have it dormant on .151; public-ingress.md/changelog.md reference .88/.89). This IP is separate from the .141–.143 HAProxy VIPs and is never F5-mapped.
  • Listeners: HTTPS for the public hostnames + a *.devl.rrchnm.org wildcard.
  • allowedRoutes: open to the namespace(s) hosting public services (label-gated), not just platform/cert-manager as today.
  • Certs: letsencrypt ClusterIssuer (exists; on LE staging at the time of writing per the CA design doc — since flipped to prod).

Dynamic dev URLs — *.devl.rrchnm.org

  • DNS (external/campus): *.devl.rrchnm.org → 129.174.134.71 in the campus rrchnm.org zone (rrchnm.internal must never render public names). Coordination task with the NS team.
  • Wildcard TLS (the linchpin — likely already solvable): cert-manager DNS-01 is required (HTTP-01 can’t do wildcards). Precedent exists: plan-popquiz-rrroster-k0s.md runs LE DNS-01 against the rrchnm.org authoritative NS via RFC2136 using the Caddy fleet’s TSIG key (caddy_proxies.sops.yaml). A cert-manager DNS-01 solver for *.devl.rrchnm.org reuses that access — no new devl.rrchnm.org delegation needed if the same RFC2136 key/zone works from cert-manager. Reconcile against internal-dns (documents only the arceus/internal RFC2136 path) and record the delegation-vs-RFC2136 decision there.
  • Dynamic dispatch (<hash> → deployment): app-layer above the Gateway. Either a wildcard HTTPRoute → a dispatcher service that maps hash→backend, or a small controller that mints a per-deployment HTTPRoute on create. Straightforward once the wildcard listener + cert exist.

Open conflicts / dependencies (load-bearing — resolve before building)

  1. Public-IP allocation — resolved: fleet.yaml is current, the notes are stale. fleet.yaml’s allocation map (the SoT) lists .141–.143 → F5 .71–.73 as free/reserved, and per the operator (2026-08-27) that map is up to date. public-ingress.md’s conflicting claims (.71 = XCP-ng VIP, .72 = hermes/postfix :25, .73 → the .88 Gateway leg) are stale — XCP-ng is retiring and the public-ingress allocation section predates the current fleet map. So no collision. Residual items, both light: (a) the F5 NAT entries .71–.73 → .141–.143 still need filing with the network team (fleet marks them reserved-not-yet-filed — the normal F5 coordination for any new VIP); (b) the stale allocation claims in public-ingress.md want correcting (docs hygiene).
  2. Public Gateway IP: live manifests have it dormant on .151; public-ingress.md/ changelog.md reference .88/.89 — also stale (the live .151 is current). Under this design the Gateway takes a LAN-only IP behind HAProxy — reuse .151 or a fresh AQ114 LB IP; not F5-mapped either way.
  3. Internal targets .141–.143 sit in the HAProxy .128/25 block between VIPs .134–.139 and hosts .144–.148; appear free but confirm against the inventory.
  4. F5 return mode: confirmed source-preserving DNAT (same as the existing VIPs) → the fwmark/route machinery is required on the new HAProxy nodes.

Phased plan

  • Phase 0 — file F5 NAT + confirm LAN targets (light; not a blocker): the IP question is resolved — fleet.yaml (SoT) has .141–.143 → F5 .71–.73 free. So Phase 0 is just: file the F5 NAT .71–.73 → .141–.143 with the network team, pick the Gateway’s LAN IP (.151 or new), and correct the stale public-ingress.md allocation + .88/.89 claims. No collision to untangle.
  • Phase 1 — public services (no wildcard). Build detail + status (2026-08-27):
    • Phase 1a — self-service Gateway + proof. ✅ DONE, LAN-testable.
      • k0s/platform/public-gateway.yaml: allowedRoutes → the rrchnm.org/public-ingress: "true" namespace label (any opted-in ns attaches a public HTTPRoute; no Gateway/HAProxy edit ever). Also moved the Gateway .151 → .150.
      • k0s/platform/public-demo.yaml: throwaway whoami (traefik/whoami) in a labeled public-demo ns + an HTTPRoute for demo.rrchnm.org. Removable.
      • Test once platform syncs: curl -H 'Host: demo.rrchnm.org' http://10.112.12.150/.
    • Gateway .151 → .150 move. ✅ DONE — public-gateway.yaml (addresses), ansible/vars/k0s-gateways.yaml (SoT ip + allocation comment), k0s/kube-system/cilium-lb-pool.yaml (aq114-public block .151-.153 → .150). Frees .151-.153 for the HAProxy hosts. kubeconform-clean.
    • The dedicated HAProxy tier. ✅ DEPLOYED & VERIFIED (2026-08-28) — public path live end to end.
      • janus / portunus / terminus — hosts .151 / .152 / .153, VRRP VIPs .141 / .142 / .143 (F5 .71/.72/.73). Spread hyperion/theia/hyperion; MACs locally-administered aa:5c:3e:91:d7:51 / b6:2f:88:0c:41:52 / ca:74:1d:a3:6f:53.
      • opentofu/fleet.tf: class named k0s_haproxy_servers (NOT k0s_public_haproxy — chosen so it reads as a sibling of haproxy_servers). A distinct class is required because the class name IS the Ansible group here, and this tier needs its own group / VIPs / passthrough config; but it is NOT a new VM shape — a shared local.haproxy_spec (2 CPU / 4 GiB) is referenced by BOTH haproxy_servers and k0s_haproxy_servers, so the box is identical and DRY. tofu validate ✓.
      • fleet.yaml: the 3 instances + k0s_haproxy_servers in group_tree’s incus_instances children + .128/25 allocation map updated (.150 Gateway, .151-.153 hosts, .141-.143 VIPs). ansible-inventory --graph resolves the group.
      • group_vars/k0s_haproxy_servers.yaml: this tier’s config — VIPs .141-.143 (keepalived, router_ids 141/142/143, haproxy_internal_vips: []), the acme-off-fleet-clients.md §6 fwmark/F5-return + conntrack + rate-limit vars (duplicated — the role has NO defaults/, so every haproxy_* var must be present), and the trigger haproxy_gateway_backend: 10.112.12.150.
      • playbooks/setup-k0s-haproxy-servers.yaml: near-twin of setup-haproxy-servers.yaml, hosts: k0s_haproxy_servers, forces haproxy_sites: [] (no catalog), same keepalived VRRP-instance computation. --syntax-check ✓.
      • Shared-role change (was the RISK): roles/haproxy/templates/frontends.cfg.j2 now has a top-level {% if haproxy_gateway_backend is defined %} passthrough branch (:80 → gateway-http / :443 mode tcp → gateway-https, both server … {{ backend }}) {% else %} the entire original catalog/maps config {% endif %}. Verified byte-identical for the existing fleet: rendered the original vs modified templates with Ansible’s Jinja settings (trim_blocks=True, lstrip_blocks=False) and an empty catalog — orig == modified exactly. The sites/maps loops are no-ops under haproxy_sites: [].
      • mk/fleet.mk: new make k0s-haproxy target (tofu k0s_haproxy_servers VMs + the playbook), deliberately NOT in make up — one-time stand-up; publishing a k0s service afterward is a git HTTPRoute, never a make run.
    • Applied & verified (2026-08-27 → 28):
      • group_vars/k0s_haproxy_servers.sops.yaml created (vrrp_auth_password; recipients init+operator+cicd — a superset of the sibling’s, so the runtime keys decrypt it). Commit bbacb4d0.
      • make plan → make k0s-haproxy: 3 to add, 0 to change, 0 to destroy — the shared local.haproxy_spec refactor left the existing fleet untouched (proof the DRY change was transparent). VMs recorded in tofu state (serial 1321, commit 79922a04).
      • VIPs settled 1:1 after the nftables fix below: janus .141, portunus .142, terminus .143 (each MASTER of one, BACKUP of two). haproxy -c clean.
      • Public path proven end to end: curl http://demo.rrchnm.org → 200 whoami echo, round-robin across the tier (X-Forwarded-For .151/.152 both seen). Public DNS resolves to all three F5 IPs 129.174.134.71/72/73, and the F5 NAT .71-.73 → .141-.143 is live — which also validated the F5-return fwmark machinery in production (a public client’s VIP-sourced reply is marked → routed out the F5 next-hop → NAT’d back to the client).
      • Commits: cef78103 (fleet.tf/fleet.yaml/group_vars/playbook/frontends.cfg.j2/mk), f0ac5d74 (nftables fix, lesson 2), 94fc70a8 (Gateway self-service + .150), 6daf2ca7 (public-demo).
    • Lessons — bit us during bring-up; heed on any rebuild or new HAProxy-family tier:
      1. Evacuate the Gateway from .151 on the LIVE cluster BEFORE provisioning the .151-.153 hosts. The .151 → .150 move was committed but not applied, so the live cilium-gateway-public was still L2-announced on .151 when janus (.151) booted → ARP collision → janus intermittently lost SSH mid-playbook (.152/.153 were unclaimed pool IPs, hence fine). Correct order: make k0s-kube-system (pool → .150) + make k0s-pin-gateway-ips (annotation → .150) FIRST, then make k0s-haproxy.
      2. A new HAProxy-family group MUST be added to the nftables template’s HAProxy gates. roles/nftables/templates/nftables.conf.j2 gated the whole HAProxy block (:80/:443/:8181 accept, VRRP proto-112 accept, the ip raw notrack + ip mangle F5-return tables) on 'haproxy_servers' in group_names, so the tier got NONE of it: VRRP dropped → each node elected MASTER of all three VIPs (split-brain), and :80/:443 were firewalled shut. Fix (f0ac5d74): extend both gates to … or 'k0s_haproxy_servers' in group_names, and make HAPROXY_LB_SET select the host’s OWN fleet. Render-diff proved every non-tier host byte-identical. To recover a split-brain: re-apply nftables (--tags init) then restart keepalived fleet-wide.
      3. LAN-test via the HOST IP or the Gateway, NEVER the F5-NAT’d VIP. A VIP’s :80/:443 replies are fwmark-routed to the F5 next-hop, so a direct-LAN curl to .141-.143 black-holes (curl rc 000) — the return path only closes for a client arriving through the F5. Use http://<host-ip> (.151-.153) or http://10.112.12.150 (Gateway) for LAN checks.
    • Remaining — Phase D only: add the Gateway :443/HTTPS listener + LE cert. Until then http:// is fully live (public), https:// won’t serve and the gateway-https HAProxy backend health-check stays DOWN by design. (Public DNS + F5 NAT are DONE.)
  • Migrating existing public services onto this edge (popquiz / rrroster / analytics). Already k0s apps, but they reach the public via the OLD workaround: a per-app caddy sidecar (terminates TLS via its own DNS-01 cert, reverse-proxies to the Django app — NOT serving statics) + a dedicated LB IP (.155/.156) + a hand-maintained services_kubernetes_passthrough HAProxy entry (make haproxy). Target: an HTTPRoute on the public Gateway — drop the sidecars/PVCs, free .155/.156, delete the HAProxy entries.
    • Source-IP allowlist — SOLVED at the tier (implemented 2026-08-28). The client IP is preserved TO the tier (F5 asymmetric-return) but LOST past it — the Gateway/pod see the HAProxy node IP (proven: whoami echoed X-Forwarded-For: 10.112.12.151). So the allowlist is enforced HERE, exactly like the site fleet’s enforce_ip_allowlist — and driven from the SAME kind of catalog, with the SAME knob. Services opt in via vars/services-kubernetes.yaml → services_kubernetes_public with haproxy: { enforce_ip_allowlist: true } — the exact knob the site catalogs + the passthrough entries use; setup-k0s-haproxy-servers.yaml folds those into haproxy_gateway_sites (just as the site playbook computes haproxy_sites), and frontends.cfg.j2’s passthrough branch renders one shared acl ip_whitelist src {{ haproxy_ratelimit_whitelist }} (campus + internal, same var as the site fleet) then per host a :443 SNI reject + a :80 Host 403 gated on !ip_whitelist. Allowlisting is opt-in via the catalog (the self-service default applies otherwise — details operator-side). popquiz/rrroster/analytics are pre-seeded GMU-only, inert until their DNS + HTTPRoute route through the tier. Render-diff confirmed the site fleet stays byte-identical. Apply with setup-k0s-haproxy-servers.yaml --tags config (NOT make haproxy).
    • Cutover order (per service): (1) Phase D cert covers its FQDN; (2) commit the HTTPRoute on public + the app’s plain Service (drop the sidecar); (3) apply the tier allowlist FIRST (no open window); (4) cut public DNS from the .155/.156 path to F5 .71-.73; (5) decommission — delete the sidecar + edge-LB, remove the services_kubernetes_passthrough entry + re-run make haproxy on the SITE fleet, free the LB IP.
    • Regression to weigh: the app loses the real client IP (Envoy stamps the HAProxy node IP into XFF). Fine for popquiz/rrroster; for the analytics dashboard confirm it doesn’t need per-request client IPs before cutting over (or wait for PROXY-protocol below).
  • Deferred / future TODO — Cilium Gateway PROXY-protocol ingestion. The clean long-term fix for BOTH the allowlist and client-IP visibility is PROXY protocol threaded tier → Gateway → app (tier send-proxy-v2, Envoy reads it, enforce at L7 / restore XFF). Whether Cilium’s Gateway API accepts PROXY protocol on the listener (via CiliumGatewayClassConfig / annotation) is version-dependent and UNVERIFIED on our Cilium — needs a spike. If supported, it supersedes the tier-level SNI allowlist and hands apps the real client IP again. Until then, the tier-level catalog allowlist (services_kubernetes_public → haproxy.enforce_ip_allowlist, shared haproxy_ratelimit_whitelist) is the enforcement point.
  • Phase 2 — wildcard dev URLs: cert-manager DNS-01 solver for *.devl.rrchnm.org (reuse the Caddy-fleet RFC2136 key), campus wildcard DNS record, wildcard listener + cert, and the <hash> dispatcher/controller.
  • Phase N — docs: decision-log entries into public-ingress.md (trigger elected), plan-haproxy-incusos.md (agni/rudra role), internal-dns.md + internal-cert-authority.md (public-zone DNS-01 for wildcards).

Onboarding a new public *.rrchnm.org service — the RFC2136 DNS-01 grant (REQUIRED)

Every new public FQDN added to the public-tls cert SANs needs its ACME DNS-01 challenge record writable by cert-manager, or issuance stalls and the Gateway serves a cert missing that SAN (browser: “your connection is not private”; curl (60) no alternative certificate subject name matches). This bit us onboarding postiz.rrchnm.org and will bite every future service — capture it here.

  • Authority + key. rrchnm.org is served by our own ns1/ns2/ns3.rrchnm.org (BIND 9.18). ns1 = 34.227.140.85 is the SOA MNAME (primary) — RFC2136 updates go there. cert-manager’s letsencrypt ClusterIssuer signs with TSIG key rfc2136-key. (HMACSHA512, from public_dns_tsig_key). The client side is correct and unchanged per service.
  • The zone layout gotcha. Each public service is its own BIND zone (zone "svc.rrchnm.org" { type master; file …; update-policy { … }; }), NOT a record in a single rrchnm.org zone. So the grant lives in that service’s zone block.
  • The grant gotcha (the actual trap). DNS-01 writes _acme-challenge.svc.rrchnm.org — a child of the A-record name. BIND grant … name svc.rrchnm.org. TXT; is exact-match and does NOT cover the _acme-challenge. child → the update is REFUSED. Use the durable form instead, per service zone: 
    update-policy { grant rfc2136-key zonesub TXT; };   # any TXT at/below apex, TXT-only = safe
    Then named-checkconf && rndc reconfig — rndc reload re-reads zone files only, NOT named.conf, so an update-policy edit needs reconfig. (A silent named-checkconf error makes reconfig a no-op — always run it first.)
  • Verify server-side before trusting cert-manager (ends all guessing): 
    nsupdate -y 'hmac-sha512:rfc2136-key:<secret-from-named.conf>'
      server 34.227.140.85 ; zone svc.rrchnm.org
      update add _acme-challenge.svc.rrchnm.org 60 TXT "probe" ; send
    dig +short TXT _acme-challenge.svc.rrchnm.org @34.227.140.85   # "probe" = grant live
    REFUSED = grant not loaded (reconfig / checkconf / key-name mismatch vs the key "…" statement). NOTAUTH(BADSIG) = wrong secret/algorithm in the probe, not a policy problem.
  • cert-manager symptoms + recovery. A refused challenge shows state=pending reason="DNS update failed. Server replied: REFUSED", presented=false. Deleting such a Challenge hangs — its finalizer cleanup (an RFC2136 delete) is refused too; clear with kubectl -n platform patch challenge <c> --type=merge -p '{"metadata":{"finalizers":null}}' (safe when presented=false — nothing was published). If the CertificateRequest wedges (…not up to date: [spec.dnsNames]), delete the stale CR by name (public-tls-N; the CRs carry no labels, so -l selectors match nothing) — cert-manager rebuilds it.
  • Debt: popquiz / rrroster / dashboard.stats are coasting on pre-split certs and will hit this exact wall at renewal (≤90d). Retrofit each zone with zonesub TXT + rndc reconfig before then. A single grant rfc2136-key zonesub TXT; in each is the standing fix.

Case study — Postiz (engagement ns): the Temporal three-gotcha

First self-hosted stateful app on the public edge (social scheduler, github.com/gitroomhq/postiz-app): one app container + Postgres + Redis + Temporal + Elasticsearch, translated from their docker-compose.dev.yaml into k0s/engagement/postiz. Overlay is Argo-managed; ns/quota/cert/allowlist are Ansible. The app came up easily; Temporal was the entire fight — three sequential failures, each masked as a 502 from the in-pod nginx (:5000 serves the frontend and reverse-proxies /api → the NestJS backend on :3000). Because pod readiness only checks the frontend, the pod reads 1/1 Running the whole time while /api 502s. Diagnose from logs, not pod status.

  1. Search-attribute cap → add Elasticsearch. A lone temporalio/auto-setup uses standard SQL visibility, which caps custom search attributes at 3 per type; Postiz registers more, so the backend’s TemporalRegister.onModuleInit throws “cannot have more than 3 search attribute of type Text”, never binds :3000. Fix = advanced (ES) visibility, matching their compose: elasticsearch:7.17.27 single-node + ENABLE_ES=true / ES_SEEDS / ES_VERSION=v7. Under PSA baseline we can’t set the vm.max_map_count sysctl (no privileged init), so node.store.allow_mmap=false sidesteps it. ES is ephemeral (emptyDir) — visibility index only, rebuilt on boot; workflows live in Postgres.
  2. Phantom dynamic-config path → don’t point at a missing file. Their compose sets DYNAMIC_CONFIG_FILE_PATH=config/dynamicconfig/development-sql.yaml, but that file ships only in their mounted copy, not in the auto-setup image → the server crash-loops “no such file or directory” (after schema setup succeeds — misleading).
  3. Search-attribute cache hang → supply the config properly. Simply dropping the env stops the crash but the backend then hangs ~60s in (registers attributes, never binds :3000, log freezes, CPU spins) — the newly-registered attrs aren’t readable yet. The cure is that file’s system.forceSearchAttributesCacheRefreshOnRead: true. Supply it via a ConfigMap mounted read-only (subPath) at the ABSOLUTE path /etc/temporal/config/dynamicconfig/development-sql.yaml
    • set DYNAMIC_CONFIG_FILE_PATH to it. (New kind → whitelist ConfigMap in the AppProject and make k0s-gitops-apps before merge, else Argo: “synchronization tasks are not valid”.)

Operational notes: order = Temporal healthy (with the config) → then restart the app (kubectl -n engagement delete pod -l app=postiz) so the backend re-inits against a correct Temporal; PM2 gives up on the crashed backend, so a stale pool pod never self-heals. The ES image is ~600MB — watch the canti containerd-unpack bug on the pull. Pilot debt to productionize: Postgres/ES on local-path/emptyDir (→ baird Postgres, an ES PVC), local uploads (→ Garage S3), and the ghcr image is digest-pinned but the stack is otherwise dev-grade. Registration is closed (DISABLE_REGISTRATION=true) — Postiz has no env-seeded admin, so the first registrant is owner; we opened it briefly (campus-gated at the tier) to create the account, then flipped it.

Cross-references

public-ingress.md (master plan; the direct-vs-fronted decision + triggers) · plan-haproxy-incusos.md + todo-haproxy-incusos.md (the Caddy-fronting HAProxy fleet; agni/rudra) · acme-off-fleet-clients §6 (F5-return fwmark machinery — ruleset operator-side) · cilium-gateway-l7 (Cilium 0x200 TPROXY marks — fwmark collision) · plan-popquiz-rrroster-k0s.md (LE DNS-01 vs rrchnm.org via RFC2136 — the wildcard precedent) · plan-k0s-lamp-hybrid.md (the direct-Gateway path this reverses) · internal-dns + internal-ca (DNS authority split; wildcard/DNS-01 is Phase-5/deferred) · k0s-gateway-placement (why LB IPs must be AQ114-side).