HAProxy on IncusOS

Status: LIVE on 3 of 5 nodes (updated 2026-08). atlas (hyperion), prometheus

epimetheus (theia) are built and serving production VIPs (VRRP masters; MCP edge + internal VIP added 2026-07-15). Remaining: build the last two nodes (agni/rudra, still #-commented at .144/.145 in fleet.yaml) and Phase 5 — decommission the retiring XCP-ng LBs. Original in-repo automation landed as commit 5cc9b852. Addressing was renumbered from reclaim-in-place to a contiguous 10.112.12.128/25 block (host IPs .144–.148, VIPs .134–.138) — see the 2026-06-13 “renumber + Phase 1 build” decision-log entry; the Proposed target state table below predates it. Live execution tracker: todo-haproxy-incusos.md (operator note).

Depends on:

5 new F5 NAT entries 10.112.12.134–.138 → 129.174.134.65–.69:80,443 (network-team ticket). The slots are already reserved in inventory/aq114.yaml (“.134 haproxy VIP1 … .138 haproxy VIP5”).
The public-ingress.md (operator note, superseded) Phase A reachability gate, extended to the VIP block: campus L3 must forward to an L2/VRRP-announced IP in 10.112.12.134–.138 on AQ114.
Caddy fleet up — it is the backend tier (CHANGELOG 2026-07-01).

Blocks: retiring the XCP-ng agni/rudra HAProxy pair and the atlas/prometheus nginx pair; reclaiming .70/.71/.75 (VIPs) + .73/.74/.76/.77 (hosts); closing public-ingress.md Phase E + Phase F.

Background: haproxy (how the fleet is configured today), public-ingress.md (operator note; Phase E is the canonical migration stub), the caddy rebuild (CHANGELOG 2026-07-01) / plan-caddy-public-tls.md (the backend tier + TLS posture), the out-of-tree vibe/setup-haproxy-servers.yaml + keepalived/ role being ported, and the changelog HAProxy hairpin-routing entry.

Motivation

HAProxy — and the legacy nginx reverse-proxy pair this plan folds into it (see Proposed target state) — is the last reverse-proxy tier still on XCP-ng. Every other fleet — caddies, LAMP DBs, DNS, CA, object storage, k0s — runs on IncusOS and is built, snapshotted, and lint-gated by make up. HAProxy is not: it’s automated by an out-of-tree playbook (vibe/setup-haproxy-servers.yaml) wired to roles (utility, common, logrotate) and a per-site schema (reverse_proxy.haproxy.*) that do not exist in this repo. So it can’t be rebuilt from infra/ansible, can’t be rolled back via the per-host Incus snapshot convention, and drifts from inventory without make lint noticing.

Folding the automation into infra/ansible and moving the VMs to IncusOS closes the XCP-ng book and unifies the bring-up under one make up. Doing it now also lets us go from 2 nodes → 5 (and 2 VIPs → 5), which adds capacity headroom and removes the current 2-node failure correlation (lose one of two and you’re at 50% with no spare).

This is a port + re-platform, not a lift-and-shift: the HAProxy/keepalived logic is sound and well-documented in haproxy, but its inputs (per-site catalog, error pages, group membership, firewall) all have to be rewired onto infra/ansible’s reality.

Current state

XCP-ng agni (10.112.12.73) + rudra (10.112.12.74), active-active, Keepalived VRRP holding two VIPs .70/.71, F5-NAT’d to public 129.174.134.62/.63.
Config source of truth, both outside infra/ansible:
- vibe/setup-haproxy-servers.yaml — HAProxy 3.0 (bookworm-backports-3.0), frontend http (:80, mode http: HTTP→HTTPS, per-site IP-whitelist 403, request-rate measurement + bwlim-in throttle), frontend https (:443, mode tcp: TLS passthrough routed by req.ssl_sni, conn/bytes rate limiting), per-site backend http-/https-<fqdn>, listen stats :8181, custom log-format with rl_* per-source rate fields, errorfiles. Package haproxy.cfg owns global/defaults; the playbook only writes conf.d/*.cfg via EXTRAOPTS="-f /etc/haproxy/conf.d".
- keepalived/ role — tasks/{main,install,configure}.yaml + templates/reverse_proxy--haproxy.conf.j2 (+ nginx variant) + check/notify scripts. Renders one vrrp_instance per entry of keepalived_vrrp_instances (name/state/priority/router_id/virtual_ip), binds the VIP on network_primary_net_interface, fails the VIP over on haproxy process death, notifies via rrchnm-notify.sh.
Per-site routing was fed by roles/utility/get-websites-with-proxy-list.yaml (filter_proxy: haproxy) over a reverse_proxy.haproxy.{ip_whitelist, proxy_protocol, acme_pin_first, ports, websockets} schema — gone in this repo. infra/ansible instead has vars/websites-static.yaml + vars/websites-lamp.sops.yaml with a nested caddy: block and a top-level site: selector.
A separate XCP-ng nginx reverse-proxy pair — atlas (.76) + prometheus (.77), “active-passive-passive” (the keepalived role’s nginx variant) — also fronts traffic on XCP-ng. This plan absorbs its role into the unified HAProxy fleet and reclaims its names + IPs.
infra/ansible has no haproxy or keepalived role. The IncusOS k0s LB block is live at .88–.90; .73/.74, .76/.77, and every VIP slot are XCP-ng-occupied.
Half-anticipated scaffolding already in place: inventory/aq114.yaml flags .73 agni/.74 rudra “planned IncusOS HAProxy” and reserves .134–.138 as “haproxy VIP1–5 → 129.174.134.65–.69”; group_vars/caddy_proxies.yaml already trusts the .73/.74 /32s in caddy_haproxy_proxy_protocol_sources; its http_status_messages carries a reason field put there explicitly “so the future HAProxy errorfile consumer can emit the correct HTTP/1.1 status line.”

Proposed target state

Fleet — 5 nodes, one VIP each, full active-active

Superseded by the 2026-06-13 renumber (see decision log). As built, host IPs are .144/.145/.146/.147/.148 (a contiguous 10.112.12.128/25 block), not the reclaimed .73/.74/.76/.77+.96 below. VIPs (.134–.138) + F5 NAT mapping are unchanged. The names are still reused; the host IPs are not.

Node	Host IP	VRRP MASTER for	VIP	F5 NAT → public
`agni`	`.73` (reclaimed)	VIP1	`.134`	`129.174.134.65`
`rudra`	`.74` (reclaimed)	VIP2	`.135`	`129.174.134.66`
`atlas`	`.76` (reclaimed)	VIP3	`.136`	`129.174.134.67`
`prometheus`	`.77` (reclaimed)	VIP4	`.137`	`129.174.134.68`
`epimetheus`	`.96` (new)	VIP5	`.138`	`129.174.134.69`

Names: atlas, prometheus, and Prometheus’s brother epimetheus are Greek Titans (matching the bare-metal Titan hosts hyperion/theia) joining the Vedic agni/rudra. atlas (.76) and prometheus (.77) are existing XCP-ng hostnames — the nginx reverse-proxy pair (“active-passive-passive” in inventory/aq114.yaml). Reusing them means this migration also retires that nginx tier and folds its role into HAProxy, with the two nodes reclaiming their own names + IPs (just as agni/rudra reclaim .73/.74). Only epimetheus is net-new, on the (free) non-F5 slot .96. So four of five nodes reclaim an XCP-ng name+IP; the VIPs carry the F5-mapped public traffic, so host IPs need no NAT entry. The .70–.77 allocation map now lives in fleet.yaml (the plan-aq114-renumber.md note that used to track it was superseded + retired 2026-07-01 — see CHANGELOG).

Two snags to confirm. (1) The in-cluster observability Prometheus TSDB (prometheus.observability.svc.cluster.local) shares the bare name — no hard DNS collision (cluster.local vs the prometheus.rrchnm.internal host record), but it’s a readability tax; the XCP-ng inventory already tolerated it. (2) Reclaiming .76/.77 requires the XCP-ng nginx proxies to be fully drained first — confirm nothing still routes through the nginx tier before the IncusOS atlas/prometheus take those IPs.

Alternative placement: put all three siblings on fresh (free) slots (.96–.98) instead of reclaiming .76/.77 — avoids the nginx-drain ordering constraint, but then the IncusOS atlas/prometheus can’t share a name with the still-live XCP-ng atlas/prometheus during parallel-run (inventory + internal-DNS clash), so the nginx pair would need renaming or earlier decommission anyway. Recommendation: reclaim .76/.77.

Every node is MASTER for exactly one VIP and BACKUP for the other four, so all five carry live traffic and any single node’s failure migrates just its one VIP to a peer. This replaces the 2-VIP XCP-ng setup; legacy VIPs .70/.71 (public .62/.63) are retired or repointed (see Sequencing Phase 4).

New `infra/ansible` shape (mirrors the caddy fleet)

Piece	Path	Notes
HAProxy role	`roles/haproxy/`	install (3.0 backport, `conf.d` + `EXTRAOPTS`), render `frontends.cfg` / `stats.cfg` / per-site `<id>.cfg` + errorfiles + logrotate, `haproxy -c` validate-before-reload handler. Per-task tags `haproxy` / `config`, like caddy.
Keepalived role	`roles/keepalived/`	ported from `/workspace/keepalived/`, kept separate + reusable (the split the old repo had). Template nearly unchanged; what changes is how instances are computed.
Playbook	`playbooks/setup-haproxy-servers.yaml`	thin orchestrator, parallel to `setup-caddy-servers.yaml`: `pre-flight.yaml` (post-cloud-init snapshot) → `init` + `nftables` baseline → `haproxy` role → `keepalived` role.
Inventory group	`inventory/groups.yaml`	new `haproxy_servers` group (child of `incus_instances`); host defs in `aq114.yaml`.
Group vars	`group_vars/haproxy_servers.yaml` (+ `.sops.yaml`)	VIP/VRRP model + rate-limit thresholds + stats bind; the VRRP `auth_pass` lives SOPS-encrypted.
VM provisioning	`opentofu/haproxy.tf`	5 Incus VMs (macvlan, static cloud-init IPs), modeled on `caddy.tf`. Stateless → no `prevent_destroy`.
Make target	`make haproxy`	sequenced in `make up` after `caddy` — HAProxy’s backends are the caddies.

Per-site routing — derive from the existing catalog, no `reverse_proxy.haproxy`

HAProxy routes by SNI/Host to a Caddy pool, not to origin servers. So each site’s backend is the set of caddy_proxies whose IP-derived rrchnm_site matches that site’s site: field in the caddy-served catalogs vars/websites-{static,hugo}.yaml (one shared schema). (Corrected from the original “{static,lamp}”: LAMP — vars/websites-lamp.sops.yaml — moved to the k0s public Cilium Gateway on 2026-05-29, so it is NOT caddy-served and this L4 tier does not front it; websites-hugo.yaml is the catalog the original text missed.)

frontend https (mode tcp): use_backend https-<id> keyed on req.ssl_sni (fqdn + aliases); backend server list = the caddy-pool IPs for the site. send-proxy-v2 when the target caddy expects PROXY-protocol — i.e. site == rh461 (kyogre/groudon already allow it); AQ114 caddies stay plain. This is consistent with caddy_proxy_protocol_sources being rh461-gated in group_vars/caddy_proxies.yaml.
frontend http (mode http): HTTP→HTTPS redirect + per-site ip_whitelist 403 + ACME HTTP-01 pinning, same as the vibe template.
Add a haproxy: block to the catalog schema (mirrors caddy:). As built (2026-06-13 decision), the block’s presence IS the fronting opt-in — a site is HAProxy-fronted iff it carries a haproxy: block (no implicit enable_proxy_protocol fallback). Fields: enable_proxy_protocol (send PROXY-v2 to the pool; defaults to caddy.enable_proxy_protocol), enforce_ip_allowlist (LB-tier source-IP 403/SNI gate, default false), acme_pin_first, websockets, aliases (incl. *.-suffix wildcard fan-out). Backend pool + SNI still derive from site:. The 19 public dev sites (static + hugo) each carry haproxy: {enable_proxy_protocol: true}.
Shared error pages. Reuse http_status_messages (already in group_vars/caddy_proxies.yaml, with the reason field added for exactly this) + one shared error-page template. Promote both to a shared location (group_vars/all.yaml or a small common role) so caddy and haproxy read a single source.
Port vibe/analyze-haproxy-log.py + the threshold-tuning workflow alongside the role; re-point the haproxy subsystem page at the new locations.

VRRP model for 5 VIPs × 5 nodes

Define haproxy_vips in group_vars/haproxy_servers.yaml — a 5-entry list of {name, vip, router_id, master_host}. The role computes each host’s keepalived_vrrp_instances by looping all five VIPs: where inventory_hostname == master_host, emit state: MASTER + top priority; otherwise state: BACKUP + a staggered priority ladder so a failed node’s VIP redistributes deterministically (not all onto one peer). The existing reverse_proxy--haproxy.conf.j2 already renders one vrrp_instance per list entry, so this is a vars-modeling change with a minimal template edit.

virtual_router_ids must be campus-unique and distinct from the XCP-ng fleet’s existing VRIDs for the whole parallel-run window — a VRID collision on the shared L2 segment is a split-brain.

Firewall (`nftables` role, per-group ruleset)

Accept :80/:443 from anywhere (public arrives via F5), :8181 stats from internal only, and VRRP (IP proto 112) + its multicast among the five fleet /32s. Critically, port the XCP-ng “hairpin-routing trap” mitigation — the mangle-table fwmark on sport-80/443 packets + the policy-routing rule that fixes the F5 asymmetric-return path (see the changelog HAProxy hairpin entry + the group_vars/dns_resolvers.yaml note). It is easy to miss and external F5’d traffic black-holes without it while internal direct-LAN traffic looks fine.

Caddy-side change

Extend caddy_haproxy_proxy_protocol_sources (group_vars/caddy_proxies.yaml) from the two .73/.74 /32s to all five node /32s (.73, .74, .76, .77, .96), so the rh461 caddies trust PROXY-protocol from the whole fleet. Safe to do pre-cutover — the Caddy listener wrapper is mixed-mode (a source in the list that sends no PROXY header is still served as plain TCP).

Why this approach (and not the alternatives)

Choice	Rejected alternative	Why
Backends = caddy pool derived from `site:`	Port the old `reverse_proxy.haproxy` per-site schema	One site catalog already feeds caddy + deploy + dashboards; a parallel haproxy catalog would drift.
Separate `keepalived` role	Merge VRRP into the haproxy role	Keeps VRRP reusable (e.g. a future nginx pair) and the haproxy role single-purpose — the split the old repo deliberately had.
Preserve TLS passthrough (`mode tcp`, SNI)	F5 terminates, HAProxy plaintext (the “simplest” option floated in Phase E)	Passthrough keeps certs on origins, matches `haproxy`, and keeps the Caddy PROXY-protocol contract intact. Terminating shifts the trust model + cert-distribution story and has a much bigger blast radius.
5 VIPs across `.134–.138`	Keep 2 VIPs + warm spares	Uses the block inventory already earmarked; every node carries load instead of idling as backup. (Cost: 5 F5 NAT entries — accepted.)

Sequencing

Phase 0 — prerequisites

Network-team ticket: 5 new F5 NAT entries 10.112.12.134–.138 → 129.174.134.65–.69:80,443.
Resolve the .65 double-booking first. inventory/aq114.yaml maps both legacy .75 (the XCP-ng nginx-pair VIP) and planned .134 to public 129.174.134.65. These are the same public IP being handed from the retiring nginx tier to the new fleet — so draining + decommissioning the atlas/prometheus nginx pair (Phase 5) frees .75 and its .65 mapping, which .134 then claims. Sequence the F5 work so .75 vacates before .134 is mapped; don’t file the ticket with the overlap live.
Pass the public-ingress.md Phase A reachability gate for the VIP block: from an off-AQ114 host, curl --resolve a test L2/VRRP IP in .134–.138 and tcpdump the SYN arriving at the announcing node. AQ114’s /26 is static (no DHCP snooping table), so this is expected to pass where RH461 didn’t — but do not proceed if it fails (fallbacks: F5/NodePort direct-to-host, or a campus snooping-binding request for AQ114).
Allocate VRIDs distinct from the XCP-ng fleet; pick final sibling names + host IPs.

Phase 1 — build the automation (no traffic) ✅ DONE (commit `5cc9b852`)

Author roles/haproxy, roles/keepalived, playbooks/setup-haproxy-servers.yaml, group_vars/haproxy_servers.{yaml,sops.yaml}, the aq114.yaml host defs + groups.yaml group, opentofu/haproxy.tf, the catalog haproxy: block, the caddy_haproxy_proxy_protocol_sources extension, and make haproxy.

Verify: make lint clean (OpenTofu↔inventory MAC drift, yamllint, ansible-lint); a --check run of the playbook renders configs that pass haproxy -c -f /etc/haproxy/haproxy.cfg -f /etc/haproxy/conf.d.

Phase 2 — stand up `epimetheus` as the pathfinder (low risk)

epimetheus is the only net-new node — host IP .96 + VIP .138 are free, no XCP-ng contention. tofu apply the VM, then ansible-playbook playbooks/setup-haproxy-servers.yaml --limit epimetheus. It proves the whole stack (role render, VRRP, F5 NAT .69, the hairpin rules, PROXY-protocol to rh461 caddies) in production on one VIP before any reclamation touches a live XCP-ng node.

Verify: /var/run/keepalived.<vip>.state shows MASTER for VIP .138; :8181 stats reachable; curl --resolve <site>:443:10.112.12.138 end-to-end (SNI → caddy → origin; PROXY-protocol intact to rh461 caddies); termination-state log breakdown sane (low cD, no SC).

Phase 3 — go live on `epimetheus`’s public IP (canary)

Network team points F5 NAT .69 at VIP .138. Canary a few low-risk hostnames’ public DNS at 129.174.134.69, watch logs + stats. Pure parallel-run — no hostname is dual-homed yet, so DNS alone picks XCP-ng vs IncusOS. This is the production go/no-go for the whole design before any live XCP-ng node is touched.

Phase 4 — reclaim the four XCP-ng nodes (blue-green)

With epimetheus proven, migrate the four XCP-ng-held nodes whose names + IPs this fleet reclaims: agni (.73) + rudra (.74) from the HAProxy pair, and atlas (.76) + prometheus (.77) from the now-drained nginx reverse-proxy pair. For each: build the IncusOS VM and — because the IP is XCP-ng-held — either stage on a temporary free IP then renumber after the XCP-ng node stops, or stop the XCP-ng node first and reclaim the IP directly (short window; epimetheus is already carrying traffic). Attach VIPs .134–.137; F5 NAT .65–.68. Do them one at a time, validating between.

Public-DNS softener (recommended): repoint the existing public IPs 129.174.134.62/.63 (today behind XCP-ng VIPs .70/.71) at two of the new VIPs, so the live *.rrchnm.org hostnames need no DNS flag-day — only the genuinely new public IPs introduce new DNS records.

Verify: each migrated hostname serves through its new VIP; XCP-ng VIPs + nginx proxies idle in show table / tcpdump.

Phase 5 — drain + decommission XCP-ng

Stop the XCP-ng agni/rudra HAProxy pair + the atlas/prometheus nginx pair + their keepalived; confirm zero traffic; power off the VMs. Reclaim .70/.71/.75 (VIPs) and .73/.74/.76/.77 (hosts); update inventory/aq114.yaml annotations.

Phase N — cleanup

Delete /workspace/vibe/ + /workspace/keepalived/ once the in-tree role is load-bearing.
Rewrite the haproxy subsystem page to reference playbooks/setup-haproxy-servers.yaml (drop the vibe/ paths).
Close public-ingress.md Phase E + Phase F.
Spawn todo-haproxy-incusos.md from this plan’s phases when execution starts.

Risks

VRID collision / VRRP split-brain during the parallel run. Two fleets on one L2 with overlapping virtual_router_ids fight over a VIP. Mitigation: unique VRIDs distinct from XCP-ng; tcpdump proto 112 before each VIP goes live.
F5 asymmetric-return black-hole if the mangle/fwmark hairpin rules aren’t ported. External F5’d replies get dropped while internal traffic looks healthy — a silent, partial outage. Mitigation: port the rules in Phase 1; test true external reach (not just LAN curl) in Phase 3.
VRRP multicast filtered on the campus segment. Mitigation: the Phase 0 gate surfaces it; fall back to unicast VRRP peers (unicast_peer) if needed.
.65 public-IP double-booking between legacy .75 and planned .134. Mitigation: resolved in Phase 0 before any F5 work.
PROXY-protocol mismatch — HAProxy send-proxy-v2 to a caddy that doesn’t yet trust the new node /32 ⇒ Caddy reads the header as TLS bytes ⇒ 400 for every connection. Mitigation: extend caddy_haproxy_proxy_protocol_sources in Phase 1, before any backend flips to send-proxy.

Rollback

Every phase is a parallel-run, so rollback = point F5/DNS back at the XCP-ng VIPs (.62/.63), which stay live and untouched until Phase 5. Per-node, the pre-flight Incus snapshot allows incus snapshot restore <host> post-cloud-init. Inventory/TF changes are plain git revert + re-apply (state is committed; review the diff).

Open questions

XCP-ng nginx-pair retirement — reclaiming atlas (.76) + prometheus (.77) assumes the XCP-ng nginx reverse-proxy tier is being fully retired and its role absorbed by HAProxy. Confirm nothing still routes through it (and what its .75 VIP fronted) before reclaiming the names/IPs.
Host-IP placement — reclaim .73/.74/.76/.77 in place (recommended), or renumber all five into a contiguous .128/25 block for a clean block (frees the scattered .64/26 slots but burns new IPs)?
Public-DNS strategy for migrated hostnames — repoint the existing .62/.63 NAT backends (no DNS change) vs. move public DNS onto the new .65–.69 IPs. The softener in Phase 4 assumes the former.
F5 health-checking — does the campus F5 health-check each VIP (TCP :443) and drive its own failover, or do we rely on VRRP failover alone behind a single F5 backend per NAT entry?

Decision log

2026-06-13 — initial design

5 nodes / one-VIP-per-node (.134–.138) / TLS-passthrough chosen with the operator. Backends derived from the existing site catalog’s site: field rather than porting the old reverse_proxy.haproxy schema. Keepalived kept as a separate reusable role.

2026-06-13 — sibling names + nginx-tier consolidation

Siblings named atlas, prometheus, epimetheus (Greek Titans, joining Vedic agni/rudra). atlas (.76) + prometheus (.77) reclaim the names + IPs of the existing XCP-ng nginx reverse-proxy pair — so this migration now also retires the XCP-ng nginx tier and folds its role into HAProxy. Only epimetheus (.96) is net-new, making it the contention-free pathfinder (Phase 2); the other four are blue-green IP reclamations (Phase 4). The nginx pair’s .75 → .65 VIP is the same public IP .134 will reuse, which resolves the Phase 0 .65 double-booking.

2026-06-13 — renumber to .128/25 + Phase 1 build landed

Renumber. Reversed the host-IP placement open question: instead of reclaiming the XCP-ng host IPs in place (.73/.74/.76/.77 + new .96), the whole fleet renumbers into a contiguous 10.112.12.128/25 block — host IPs .144–.148, VIPs .134–.138 (F5 NAT unchanged: .134–.138 → 129.174.134.65–.69). Consequences:

All five host IPs are net-new, so Phase 4 builds every node in parallel with zero XCP-ng IP contention (the stage-on-temp-IP / stop-XCP-ng-first dance is gone). Hosts + VIPs now share one subnet (no cross-subnet VIP binding).
The scattered .73/.74/.76/.77 slots are no longer reclaimed by this fleet — they free at XCP-ng decommission (Phase 5).
F5-return next-hop candidate moves to .130 (the .128/25 floating F5), from the XCP-ng .64/26 .66. CONFIRM with the network team.
atlas/prometheus names are reused on fresh IPs, so they clash with the still-live XCP-ng nginx pair in internal DNS during parallel-run — rename or decommission that pair before the IncusOS pair gets DNS records (Phase 3+).

Phase 1 landed (commit 5cc9b852, no traffic): roles/haproxy + roles/keepalived (nginx variant dropped, kept reusable), opentofu/haproxy.tf (5 VMs), playbooks/setup-haproxy-servers.yaml, group_vars/ haproxy_servers.{yaml,sops.yaml}, inventory group + host defs, the nftables input branch + table ip mangle hairpin, the optional catalog haproxy: block, http_status_messages promoted to group_vars/all.yaml, caddy_haproxy_proxy_protocol_sources → the 5 node /32s, make haproxy, and the ported log analyzer. Key design calls: backends derive from site: → caddy pool (send-proxy-v2 to the rh461 pool only); sites opt into HAProxy fronting via a haproxy: block OR caddy.enable_proxy_protocol, drawn from BOTH caddy-served catalogs vars/websites-{static,hugo}.yaml (19 dev sites fronted; internal sites

LAMP — k0s-Gateway-served — excluded). VRRP priority ladder gives each VIP a distinct top backup. Verified offline: make lint (MAC drift), ansible-lint, --syntax-check, yamllint, tofu fmt, full template render. Deferred to a key-mounted env / Phase 2: tofu validate (needs provider cache) + --check (SOPS private key not mounted here). VRIDs 134–138 + F5-return gw .130 flagged CONFIRM before go-live.

2026-06-13 — explicit `haproxy:` block as the fronting opt-in + catalog sources

Two corrections to the Phase-1 build, landed after review:

Catalog sources. HAProxy fronts the caddy-served catalogs **websites-static
- websites-hugo** (the original “{static,lamp}” was wrong both ways: it missed hugo, and LAMP moved to the k0s public Gateway on 2026-05-29 so it isn’t caddy-served). The first build read only websites-static, silently dropping the 5 hugo dev sites — fixed.
Fronting signal. Switched from “front anything with caddy.enable_proxy_protocol” (implicit, footgun: a future public site without PROXY-protocol would be silently un-fronted) to “front iff the entry has a haproxy: block” (explicit). Renamed the block’s fields to match caddy’s enable_* naming: send_proxy → enable_proxy_protocol (default caddy.enable_proxy_protocol), ip_whitelist → enforce_ip_allowlist (LB-tier source-IP 403/SNI gate, default false). Added explicit haproxy: {enable_proxy_protocol: true} to all 19 public dev sites. Net: 19 sites fronted (14 static + 5 hugo), all rh461 → kyogre/groudon with send-proxy-v2; internal *.rrchnm.internal + LAMP excluded by having no block. Verified by rendering every fronted site offline.

Last updated: 2026-06-13