k0s LAMP hybrid

Historical design note — substrate executed, pod tier not built (last updated 2026-05-30; migrated 2026-09-05)

The DB substrate landed: Garage on brahma, the izanami/izanagi MariaDB pair, and the websites_lamp catalog (Phases 0 / 0.5 / 0.6). The k0s side — per-site pods, HTTPRoutes, container images (Phase 1+) — was never built. Placement and schema have drifted since: izanami sits at .181 (not .81), izanagi moved kyojin → atomsk, opentofu/mariadb.tf was never created (the DB VMs live in fleet.tf’s class model under lamp_databases), and the catalog now uses nested mariadb:/garage: blocks (2026-06-04) rather than the flat leaves shown here. fleet.yaml and the catalog file are the authority. Kept for the alternatives analysis and the decision log.

Plan for migrating the PHP-FPM half of the LAMP fleet into k0s as per-site Pods fronted by the AQ114 public / internal Cilium Gateways, while keeping per-site MariaDB on its existing Incus VMs. Inherits the AQ114 ingress-resilience properties for LAMP sites without forcing a stateful-MariaDB-in-k0s decision that the cluster’s storage layer isn’t ready for.

Status: Phase 0 complete (2026-05-28) + Phase 0.5 landing (2026-05-29) — db consolidation pulled forward of the Phase 1 pilot. Final shape (after the 2026-05-29 (late evening) per-site revision): two per-site MariaDB instances, Shinto creator-deity pair, married couple: izanami on hyperion (AQ114 .81) for AQ114-resident LAMP sites + izanagi on kyojin (RH461 .136, reusing the legacy omeka-s MAC reservation) for RH461-resident LAMP sites. No replication between them. Today: all 4 recroom CMS DBs → izanagi (their historical placement); izanami idle (reserved for future AQ114 LAMP additions). Earlier same-day draft framed this as single-instance consolidation on hyperion; revised after recognizing that per-site DB placement lets k0s pods prefer same-site DB-connection paths + preserves the legacy RH461 MAC reservation slots. Decision log at bottom for full reasoning.

Depends on:

• public-ingress.md (operator note) Phase B — the public (.150, behind the janus/portunus/terminus HAProxy tier) + internal (.154) Gateways must exist before LAMP HTTPRoutes can attach (both exist — plan-k0s-public-edge.md).
• Garage S3-compatible object storage, single-node mode sufficient — see Phase 0 + plan-object-storage.md (operator note). brahma’s _object pool already exists on kyojin; the garage_topology: single flag in group_vars/all.yaml brings RF=1 storage up today, then flips to cluster (RF=3) when vishnu+shiva’s pools land. The cluster has local-path-provisioner for ephemeral RWO; per-site mutable state (uploads, media) goes to Garage via S3 modules, not into any cluster PVC.

Blocks:

• Eventual full-stateful LAMP-in-k0s (once a production-grade block CSI lands, MariaDB can follow the PHP-FPM path).
• AQ114 LB-IP pressure relief — every future LAMP site that lands via this plan costs 0 AQ114 IPs instead of 1.

Background:

• public-ingress.md — where the Gateways live, why AQ114 is the resilient site, the LB-IP budget.
• changelog.md:713 (2026-05-10) — the original LAMP stack landing; the deliberate macvlan-to-LAN + nftables-isolation pattern this plan partially supersedes.
• changelog.md:639 (the MariaDB-on-virtiofs incident) — confirms MariaDB is the sharp edge of stateful storage in this fleet; preserving the working lamp_site_db block-volume contract is a deliberate design constraint here.
• todo-recroom-migration.md — recroom is the most-recent LAMP buildout; its design is the baseline this plan diverges from.
• incus-bridge-mesh.md:138 — bridge meshing is explicitly NOT for LAMP today; the hybrid here keeps LAMP off the mesh question entirely.

---

Motivation

Three drivers, in order:

1. AQ114 ingress resilience for LAMP sites. Production-serving LAMP backends today land on kyojin (RH461) because the caddy-↔-same-host constraint pinned them there when hugin was the only same-host caddy for the publicly-routed Omeka site. RH461 LB IPs aren’t reachable off-subnet (k0s-gateway-placement.md Issue 3), and even the LAN side of RH461 is a less-resilient surface than AQ114. The public Cilium Gateway on AQ114 .150 (behind the janus/portunus/terminus HAProxy tier — plan-k0s-public-edge.md) is the resilient public path — LAMP should be on it.

2. LAN IP savings on AQ114. Per the prior public-ingress.md analysis, each macvlan’d Incus VM consumes one LAN IP. A LAMP site fronted via HTTPRoute on the existing public Gateway consumes zero new LAN IPs — per-Host routing happens inside envoy. With MariaDB staying on Incus (1 IP per site for the db VM), the per-site cost drops from “1 IP backend VM + caddy-host coupling” to “1 IP db VM (off-AQ114) + HTTPRoute (free)”. Future sites that consolidate onto a multi-tenant MariaDB drop to zero net new LAN IPs entirely.

3. Storage layer isn’t ready for MariaDB-in-k0s. The cluster has local-path-provisioner only (plan-object-storage.md:49, plan-backup-strategy.md:171). Local-path is ReadWriteOnce + node-pinned + has no snapshot/backup story integrated with the existing borg pipeline. MariaDB’s storage requirements (block-typed volume per the 2026-05-10 virtiofs incident, prevent_destroy, borg-client coverage) don’t map onto local-path without rewriting the backup story from scratch. The hybrid sidesteps this: PHP-FPM becomes stateless (code-in-image, uploads to Garage S3, sessions in MariaDB), so it can move now; MariaDB waits for a real block CSI.

Cost framing: a full LAMP-in-k0s migration is a multi-month project (storage layer, backup integration, per-site StatefulSet rebuilds, operator runbook). The hybrid captures ~80% of the IP + resilience benefit on a multi-week timescale and leaves the MariaDB decision for when the storage layer is positioned to make it.

---

Current state

Per-site shape today (recroom in flight, others shipped 2026-05-10):

F5 (public sites only)
   │
   ▼                                   ┌── Incus VM on kyojin (RH461) ──┐
HAProxy → caddy_proxies caddy ──FCGI──►│ PHP-FPM :9000                  │
(L7, TLS at HAProxy)                   │ MariaDB                        │
                                       │ lamp_site_www (RW, FS-typed)   │
                                       │ lamp_site_db  (RW, block-typed)│
                                       │ macvlan, .136–.139             │
                                       └────────────────────────────────┘

Inventory + config locations:

• VMs declared in opentofu/lamp.tf via local.lamp_backends. Block + filesystem volumes, prevent_destroy on both. Site-pinned to a caddy host (recroom → kyojin, others → theia).
• Provisioning: ansible/playbooks/setup-lamp-servers.yaml, with playbooks/lamp/templates/ Caddyfile + per-site fragments.
• Public reach: F5 NAT → agni/rudra HAProxy → caddy_proxies (hugin on kyojin for recroom; mitra/varuna on theia for the others) → FastCGI to the backend VM.
• Isolation: per-host nftables; :9000 on each backend accepts only from the caddy_proxies’ LAN IPs.
• Storage volumes:
  • lamp_site_www: filesystem-typed, multi-attach (RW backend + RO caddies). Static assets + PHP source.
  • lamp_site_db: block-typed, exclusive RW on backend at /var/lib/mysql. Path resolved via /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_incus_db. (Volume serial fixed by Incus convention incus_<device-name> — renaming the device silently breaks discovery; the lamp.tf header documents this contract.)

Key constraints baked into the current shape:

• Cross-host LAMP impractical without OVN — site pins to a host pair.
• Caddy ↔ backend co-location enforced by lamp_site_www multi-attach (RO mount on caddy host).
• Each backend VM consumes 1 LAN IP on its host’s subnet.

---

Proposed target state

Per-site shape after this plan ships:

F5 (public) → HAProxy tier (.141-.143)
   │
   ▼
Cilium `public` Gateway @ AQ114 .150             ┌── Incus VM on RH461 ────┐
   │                                             │ MariaDB (db-only)       │
   │ HTTPRoute per site                          │ lamp_site_db (block, RW)│
   ▼                                             │ macvlan, .136–.139…     │
ClusterIP Service ─── per-site Pod ──── MySQL ───┘
   │                  ┌───────────────────────┐
   └─ FastCGI ──────► │ in-pod caddy          │
                      │  (HTTP→FastCGI shim,  │
                      │   not the public      │
                      │   caddy_proxies fleet)│       ┌── Garage S3 ───────┐
                      ├───────────────────────┤  S3   │ brahma (RF=1 today,│
                      │ PHP-FPM :9000         │ ◄───► │ RF=3 once AQ114    │
                      │  • code: read-only    │       │ disks land — flip  │
                      │    from container     │       │ garage_topology    │
                      │    image              │       │ flag, no app       │
                      │  • uploads: S3 module │       │ change)            │
                      │    → Garage bucket    │       │ Bucket per site    │
                      │  • sessions: MariaDB  │       │ for uploads/media  │
                      │  • cache: emptyDir    │       └────────────────────┘
                      └───────────────────────┘

What lives where:

Concern	Today	After
PHP-FPM execution	Incus VM on kyojin (RH461)	k0s Pod, schedulable on any worker
MariaDB	same VM	same VM, role reduced to “db-only”
PHP source code + themes/plugins	Incus FS volume lamp_site_www, multi-attached	per-site container image (built in CI, pushed to thoth) — read-only at runtime
User uploads / media library	same lamp_site_www volume	per-site Garage bucket, accessed via the app’s S3 module (Omeka S3 module, WP Offload Media, Drupal flysystem-s3)
Sessions	files in lamp_site_www/sessions	MariaDB (apps’ DB session handler) — no shared filesystem needed
lamp_site_db	Incus block volume on backend	unchanged — stays on the db VM at /var/lib/mysql
Public ingress	F5 → HAProxy → caddy_proxies → FCGI	F5 → public Cilium Gateway → HTTPRoute → in-pod caddy → in-pod PHP-FPM
Internal ingress (*.rrchnm.internal)	(planned via internal caddy fleet)	internal Cilium Gateway on .154 → HTTPRoute → in-pod caddy → PHP-FPM
TLS	HAProxy-terminated for public	LE via cert-manager for public, heimdall for internal (inherits the existing Gateway listener certs — no per-site cert work)
Per-site backup	borg-client on VM (covers DB + www)	borg-client on db VM (covers DB only); Garage handles upload-bucket durability via RF=3 + its own backup story per plan-object-storage.md
Site-host pinning	Required (caddy ↔ www RO mount)	Dissolved — k0s pods schedule freely; Garage reachable from any worker
Per-site LAN IP cost	1 (backend VM) + caddy host	1 (db VM); Phase 3 multi-tenancy can take this to 0 amortized

What gets deleted from the old state:

• Per-site caddy_proxies fragments (ansible/playbooks/lamp/templates/lamp-site.caddy.j2 and friends) for the sites that migrate. The caddy_proxies fleet stays — it keeps owning static-sites.yaml. LAMP just exits that fleet.
• HAProxy LAMP backends pointing at the LAMP VMs’ :9000. HAProxy keeps its other duties.
• nftables :9000 allowlist rules for the migrated backends (the port closes entirely — PHP-FPM only listens on the pod-local socket now).
• The per-site site pinning constraint in local.lamp_backends — the new resource shape is “db-only backend” which doesn’t carry the caddy-host pin.
• The lamp_site_www Incus filesystem volume per site — replaced by the per-site Garage bucket. opentofu/lamp.tf drops the incus_storage_volume.lamp_site_www resource entirely; only lamp_site_db survives. (Greenfield: no data to migrate; for sites with live uploads under a future rollout, an s3 sync step would seed the bucket before cutover.)

Why this approach (and not the alternatives)

Option	IP saving	Resilience	Storage risk	Effort	Verdict
0. Status quo (Incus LAMP VMs on RH461)	none	low (RH461 site)	none	none	baseline
1. Move LAMP VMs to AQ114 (theia/hyperion)	none (still 1 IP/site, just different subnet)	medium (AQ114 site, no Gateway HA)	none	low	rejected — doesn’t relieve IP pressure; AQ114 LB-IP /26 competes for the same slots
2. Full LAMP-in-k0s (PHP-FPM + MariaDB both)	high (0 IP/site)	high (Gateway HA inherited)	high — no production block CSI; backup story has to be rewritten	high (multi-month)	deferred — right long-term direction, wrong time
3. Hybrid (this plan) — code-in-image PHP-FPM in k0s, uploads to Garage S3, MariaDB stays on Incus	medium (db VM still costs 1 IP/site; Phase 3 amortizes)	high (Gateway HA inherited)	low — MariaDB stays on the proven block-volume path; uploads land on Garage’s proven object-storage path	medium (weeks)	chosen
4. Earlier-draft hybrid — PHP-FPM in k0s, NFS-CSI mounts lamp_site_www from the db VM	medium	high	medium — NFS-CSI is new to this cluster; cross-subnet AQ114-pod-to-RH461-NFS latency hits every PHP file read	medium	rejected — greenfield fleet has no lamp_site_www bytes to preserve; Garage is the strategically-correct storage layer per plan-object-storage.md (using NFS as a bridge tech would be two migrations instead of one); admin-UI plugin/theme installs (the one feature NFS preserves and S3 doesn’t) are an ops responsibility for this fleet anyway
5. Bridge-mesh / OVN private network for LAMP	high (LAMP off LAN entirely)	none from Gateway side	medium	high (OVN is its own project per incus-bridge-mesh.md)	deferred — overlaps with the eventual OVN work; not a near-term move

Option 3 wins on the timescale-vs-benefit curve: most of the resilience benefit, no exposure to the unsolved block-storage problem, no new storage layer introduced (Garage was already on the roadmap), and the per-site decisions (Pod shape, HTTPRoute, S3 module config) are mechanical once Phase 0 lands.

---

Sequencing

Phase 0 — prerequisites — DONE 2026-05-28

Storage and Gateway plumbing that everything else depended on. Status legend: ✅ landed + verified; ⏳ partially done (notes inline).

• ✅ Garage up in single-node mode. garage_topology flag added to group_vars/all.yaml; single is the default. make object-storage brings brahma online with RF=1 against the existing kyojin_object pool. Endpoint obj.rrchnm.internal:3900 resolves via the existing DNS round-robin in group_vars/dns_resolvers.yaml. Single→cluster transition is in-place: when vishnu+shiva’s pools land, flip the flag to cluster and re-run; Garage rebalances replicas in the background with no S3 endpoint change for consumers. The playbook now layout-assigns by computing the next version dynamically from garage layout show (was hardcoded --version 1 — broke on second run). nftables garage_cluster block opens 3900/3901/3902/3903 from CLI + INT nets. Survives make down (DOWN_EXCLUDES preserves garage_meta + garage_data volumes).
• ✅ Per-site Garage buckets + scoped access keys. Four buckets on brahma: recroom-omeka-s-uploads, recroom-omeka-classic-uploads, recroom-drupal-uploads, recroom-wordpress-uploads. One scoped RW key per bucket (no owner perms). Credentials stored in SOPS-encrypted ansible/group_vars/lamp_backends.sops.yaml as a lamp_garage_keys map — bucket + access_key_id plaintext (diff-readable), garage_secret_key SOPS-encrypted via a new pattern added to the group_vars regex. Auto-decrypted by community.sops vars plugin when targeting any lamp_backend; Phase 1 manifests will consume the map to render per-site k8s Secrets. The map’s header comments how to add a new site (provision bucket + key on brahma, append entry, sops-encrypt).
• ✅ Pod-to-Garage probe. Throwaway pod in default namespace ran aws s3api list-buckets + s3 cp PUT + s3 cp - GET + s3 ls LIST against connectivity-probe bucket via brahma.rrchnm.internal:3900. All five operations green; in-cluster DNS resolves brahma.rrchnm.internal correctly via Cilium CoreDNS → mesprit/azelf/uxie. Probe artifacts (bucket + key + 36 B object) left in place as future re-probe targets — they survived a full destroy/rebuild cycle, demonstrating the volume-preservation contract.
• ✅ Pod-to-MariaDB probe. setup-lamp-servers.yaml against omeka-s-recroom-rrchnm-org: MariaDB now binds 0.0.0.0:3306 with skip-name-resolve (mandatory — the internal DNS pair doesn’t serve PTRs for fleet subnets, and reverse-DNS lookups were dropping handshakes with “host ‘X.X.X.X’ could not be resolved”). nftables lamp_backends block opens :3306 from INT_NET_SET. Per-site mysql user moved to host: '%'; new playbook task drops the legacy <user>@localhost (mysql_user keys by (name, host), so the host: change created a duplicate rather than renaming). Probe pod did full CREATE TABLE → INSERT → SELECT → DROP round-trip via both hostname (omeka-s-recroom-rrchnm-org.rrchnm.internal) and IP — auth as omeka_s@%.
• ✅ Gateways in place. public on .150 (behind the janus/portunus/terminus HAProxy tier — F5 129.174.134.71/72/73 → VIPs .141-.143 → .150) + internal on .154, both Programmed=True. TLS on the public Gateway is LE-prod via cert-manager’s DNS-01/RFC2136 solver (plan-k0s-public-edge.md), so a LAMP HTTPRoute inherits the listener cert — no per-site cert work.
• ✅ Pilot site picked. recroom omeka-s. Setup-lamp-servers.yaml has been run end-to-end against this VM — MariaDB is up + LAN-bound, the omeka_s database + user exist, the VM is in the state Phase 1 pods will consume.

Verify (executed): Garage S3 LIST + PUT/GET reachable from a k0s pod via DNS ✅; same-pattern pod did MariaDB CREATE/INSERT/SELECT/DROP against the LAMP VM via DNS ✅; public Gateway HTTPRoute end-to-end (external HTTP → Gateway → pod) not yet exercised — the Gateway is Programmed=True but no HTTPRoute against a dummy backend has been tested in this fleet. Worth doing as the first step of Phase 1 before the per-site image becomes the dependency.

Surprises caught during Phase 0 (now documented in commits, not just lore):

• garage layout assign had peer_id positional argument AFTER variadic -t <tag>...; clap consumed it as a tag value, leaving <node-ids> empty. Fixed: put peer_id first.
• garage layout apply --version N doesn’t error on “no staged changes” in v2.3.0 — it produces a trivial new version with 0 movement. Gate apply on garage_layout_assign is changed + compute next version from garage layout show’s “Current cluster layout version: N” line.
• nftables garage_cluster block was missing entirely — Garage daemon was reachable only on 127.0.0.1.
• setup-oci-mirrors.yaml didn’t force a cert renewal on re-runs; the renewer.timer fires every 8h, which lost the race after a teardown where multiple slots had failed. New db-tagged tasks check openssl x509 -checkend 3600 + force step-cert-renewer.service if near-expiry.
• MariaDB’s default reverse-DNS lookup on inbound connections breaks against subnets without PTR records. skip-name-resolve required, not optional.

The pattern across all five: only surface when the relevant path runs end-to-end. The verify-before-claim-complete discipline this session paid for itself.

Phase 0.5 — per-site MariaDB hosts (izanami AQ114 + izanagi RH461) — pre-Phase-1, added 2026-05-29

Promoted from the original Phase 3. With the 2026-05-29 IaC pivot (the four recroom-*-rrchnm-org VMs became MariaDB-only shells with no data on them), the migration cost that justified deferring consolidation to after Phase 2 evaporated. Building 4 throwaway VMs only to migrate-and-decommission later is wasted work — fold the consolidation into the substrate before the pilot lands.

Target shape (final, after the 2026-05-29 late-evening per-site revision):

AQ114 (Aquia)  ── hyperion ── Incus VM ── izanami (MariaDB) ──┐
                                          block: izanami_db (RW, /var/lib/mysql)
                                          DBs: (none today — reserved for future
                                                AQ114-resident LAMP sites)
                                          borg-client: one (idle until DBs land)

RH461          ── kyojin   ── Incus VM ── izanagi (MariaDB) ──┐
                                          ├── db: omeka_s,           user: omeka_s@%
                                          ├── db: omeka_classic,     user: omeka_classic@%
                                          ├── db: drupal,            user: drupal@%
                                          └── db: wordpress,         user: wordpress@%
                                          block: izanagi_db (RW, /var/lib/mysql)
                                          borg-client: one, covers all 4 tenant DBs

Per-site independent — no replication between izanami and izanagi. Each VM hosts only the DBs whose db_host field in websites_lamp (host in the pre-consolidation lamp_sites — see Phase 0.6) matches its inventory_hostname; cross-VM noise is filtered at the playbook loop level.

Decisions (final, after the 2026-05-29 late-evening revision):

Decision	Choice	Rationale
Instance count + topology	Two per-site MariaDB hosts (izanami AQ114 + izanagi RH461); no replication	Each LAMP site lives on the DB local to its physical site, so k0s pods can prefer same-site workers for shorter DB-connection paths. Earlier same-day draft was single-instance consolidation on hyperion; revised after recognizing that per-site placement preserves the existing RH461 MAC reservation slots AND aligns DB locality with future per-site LAMP additions.
Placement	izanami → hyperion (AQ114 .81) · izanagi → kyojin (RH461 .136, reusing legacy omeka-s MAC reservation)	AQ114 placement inherits the reliability bias (k0s-cluster-rebalance.md); RH461 placement reuses an existing MAC reservation (zero network-admin coordination needed). The remaining .137-.139 reservations are freed (now placeholder host entries in inventory).
Naming	izanami (AQ114) + izanagi (RH461)	Shinto creator deities, married pair. The pair captures the two-host topology naturally — neither is “primary” vs “replica,” they’re peer per-site DBs. Adds traditional Shinto alongside the existing anime-themed Japanese fleet names (Attack on Titan on kyojin, FLCL on atomsk-anticipated).
Site assignment (today)	All 4 recroom CMS DBs → izanagi; izanami idle	Matches historical placement of the recroom fleet on kyojin (RH461). Future AQ114-resident LAMP sites: assign host: izanami.

Steps:

• TF. New opentofu/mariadb.tf (clean separation from the to-be-deleted opentofu/lamp.tf). Two explicit incus_instance resources (izanami on hyperion, izanagi on kyojin) + their two incus_storage_volume.{izanami,izanagi}_db block volumes (both prevent_destroy = true). izanami uses AQ114 static cloud-init network-config; izanagi uses RH461 DHCP cloud-init (MAC-bound reservation). Filename rationale: mariadb.tf (specific) over database.tf (generic) — leaves room for other DB classes (etcd, OpenBao raft, future Postgres) in their own files.
• Inventory. Add izanami (hyperion .81) + izanagi (kyojin .136) to ansible/inventory.yaml. New lamp_databases group contains both. Retire the old lamp_backends group entry + the 4 orphan recroom-*-rrchnm-org hostvars entries (their host_vars files are gone too).
• Ansible. setup-lamp-databases.yaml re-targeted from lamp_backends → lamp_databases. Per-site DB/user creation tasks iterate over lamp_sites (in group_vars/lamp_databases.yaml) AND filter with selectattr('value.host', 'eq', inventory_hostname) so each VM only creates the DBs it owns. Password lookup uses flat lamp_<site>_db_password vars from group_vars/lamp_databases.sops.yaml (each individually SOPS-encrypted via the .*_password wildcard).
• db_password migration (one-time human step, before playbook run). Per sop-sops-handling.md recipe 4: decrypt each of host_vars/{drupal,omeka-classic,omeka-s,wordpress}-recroom-rrchnm-org.sops.yaml to tmpfs, extract the db_password value, compose group_vars/lamp_databases.sops.yaml with the 4 flat lamp_<site>_db_password vars, encrypt in place. Then delete the legacy host_vars files.
• Backup. One borg-client per MariaDB host. izanagi covers the 4 recroom DBs today. izanami’s borg-client is idle until DBs land there.
• DNS. Pin izanami.rrchnm.internal + izanagi.rrchnm.internal via the internal DNS pattern. No db.rrchnm.internal service-alias today — explicit per-host name is clearer in a per-site topology (the site → DB-host binding is in lamp_sites, not in DNS).
• Decommission the 4 legacy db VMs. Already done via make down earlier 2026-05-29; the lamp_site_db volumes will be destroyed on the next tofu apply (resource declarations removed in opentofu/lamp.tf deletion).
• Inventory cleanup. Already done: 4 recroom-*-rrchnm-org host entries removed; borg_clients in backup2.sops.yaml loses those 4 entries on the next setup-backup-servers.yaml run. Net change: -4 hosts, +2 hosts, -2 net.

Verify:

• izanami.rrchnm.internal + izanagi.rrchnm.internal reachable from k0s pods over their LAN IPs.
• All 4 recroom DBs exist on izanagi (SHOW DATABASES; shows omeka_s, omeka_classic, drupal, wordpress). izanami has no app DBs.
• borg backup against izanagi captures the 4 tenant DBs; borg-client on izanami is configured but covers no DBs today.
• The k0s pod for any recroom site (Phase 1) reaches izanagi.rrchnm.internal:3306 with the right per-site password and gets only its own DB visible.

Group naming — decided 2026-05-29: lamp_backends does NOT persist post-decoupling. The “backend” framing only made sense when one VM ran PHP-FPM + MariaDB + the site’s filesystem volume as a coupled stack. After decoupling, the PHP-FPM half lives in k0s pods (not in any ansible group) and the MariaDB half lives on izanagi → new group lamp_databases. Naming parallels the historical lamp_backends (lamp_X where X is the role) and stays specific to LAMP (vs a generic database_servers group that could mean anything). The lamp_backends group entry in ansible/inventory.yaml is deleted entirely; preserved historical context lives in this plan + the changelog, not in the live inventory.

Where lamp_garage_keys lives after the move: currently in group_vars/lamp_backends.sops.yaml (because the legacy lamp_backend VM ran both DB and PHP-FPM). After Phase 0.5, the consumer is k0s pod-manifest rendering (Phase 1), not any ansible group’s hosts. Two options:

• Move to ansible/vars/lamp-garage.sops.yaml (new explicit-load SOPS file). Cleanest — the map is logically per-site config, not per-host, and Phase 1 playbooks will load it via vars_files: rather than the group_vars auto-load. Consumed by pod-manifest templating, not by izanagi itself.
• Co-locate in lamp_databases.sops.yaml. Tempting (single file for “LAMP-fleet secrets”), but conflates DB-host secrets with k8s-pod secrets. izanagi doesn’t need the Garage keys.

Recommendation: explicit-load ansible/vars/lamp-garage.sops.yaml — keeps the map’s purpose clear + sets the pattern for any future per-site k8s-rendering secrets. Eventually ESO-materialized from OpenBao per the k0s-eso pattern; this file is the bridge.

Superseded 2026-05-29 (night) — lamp_garage_keys, the DB-side lamp_sites map, and the lamp_<site>_db_password vars are all folding into a single websites_lamp catalog at ansible/vars/websites-lamp.sops.yaml. The standalone lamp-garage.sops.yaml recommended here was the intermediate shape. See the decision-log entry “consolidate per-site LAMP vars into a single websites_lamp catalog” for the final design, the leaf-naming-vs-regex detail, and the controller-side-over-decryption tradeoff.

Phase 0.6 — fold per-site vars into the websites_lamp catalog — pre-Phase-1, added 2026-05-29 (night)

Implemented 2026-05-30. Collapses the three scattered per-site var sources into one diff-readable catalog before Phase 1 piles on more per-site config (hostname, Gateway choice, PHP version, image ref). Full rationale + the SOPS leaf-naming-vs-regex detail + the controller-side-over-decryption tradeoff live in the decision-log entry “consolidate per-site LAMP vars into a single websites_lamp catalog”; the catalog landed in the list shape per the same-day “switch websites_lamp from map to list” decision-log entry. This section is the as-built record.

websites_lamp is the single source of truth per site — ansible/vars/websites-lamp.sops.yaml, a list of per-site dicts (no map keys — see the 2026-05-30 decision-log entry “switch websites_lamp from map to list”; fqdn is the natural identity), flat-per-site so the existing vars/ regex catches the secret leaves untouched:

websites_lamp:
  - fqdn: omeka-s.recroom.rrchnm.org
    gateway: public                 # public | internal
    php_version: "8.2"
    db_name: omeka_s
    db_user: omeka_s
    db_host: izanagi
    db_password: ENC[...]           # .*_password → encrypted
    garage_bucket: omeka-s.recroom.rrchnm.org  # == fqdn; path-style only
    garage_access_key_id: GK58793…  # _id suffix → stays plaintext
    garage_secret_key: ENC[...]     # .*_key → encrypted

Steps:

• Compose vars/websites-lamp.sops.yaml by merging the three current sources (lamp_sites, lamp_<site>_db_password, lamp_garage_keys) per sop-sops-handling.md: decrypt to tmpfs, merge, re-encrypt. The existing vars/.*\.sops\.yaml$ creation_rule + regex already cover the new file — no .sops.yaml regex change, just comment housekeeping.
• (The original key-convention-unification step from when this was a map became moot in the list shape — no keys to unify. Site identity is fqdn alone. Bucket names track fqdn at cutover; until then they keep their legacy recroom-<app>-uploads form as a plain value field.)
• Rewire setup-lamp-databases.yaml: load the catalog at play-vars time via catalog: "{{ lookup('community.sops.sops', 'vars/websites-lamp.sops.yaml') | from_yaml }}" (NOT vars_files: — that loads as plain YAML and the encrypted leaves come through as literal ENC[...] strings; see the 2026-05-30 decision-log entry “switch websites_lamp from map to list” → “test-driven discovery” subsection. Established pattern: playbooks/k0s/ops/{configure,rekey,unseal}-openbao.yaml.). Then iterate catalog.websites_lamp | selectattr('db_host','eq',inventory_hostname); read password from item.db_password (drop the lookup('vars','lamp_'+key+'_db_password')).
• Delete group_vars/lamp_databases.sops.yaml + vars/lamp-garage.sops.yaml; shrink group_vars/lamp_databases.yaml to just the db_version: 11.8 group default.

Verify: setup-lamp-databases.yaml --check still creates the 4 recroom DBs on izanagi, sourced from the catalog; make lint clean; git diff on websites-lamp.sops.yaml shows plaintext structure with only db_password/garage_secret_key as ENC[...].

Phase 1 — pilot site (recroom omeka-s)

End-to-end migration of one site to validate the k0s-side pattern. The db-side substrate (izanagi) already exists from Phase 0.5.

• (No legacy-VM repurposing step — Phase 0.5 already consolidated.) The 4 recroom-*-rrchnm-org VMs are gone; only izanagi holds MariaDB.
• Per-site container image. New image (built in CI, pushed to thoth) layering Omeka S + the S3 storage module on top of a php-fpm-base:<version> (e.g. 8.2). PHP code lives at /var/www/html read-only in the image. Site repo + image build instructions live alongside the other OCI artifacts pulled through thoth (see CHANGELOG 2026-07-01).
• Garage bucket omeka-s.recroom.rrchnm.org with a scoped access key — all three (garage_bucket, garage_access_key_id, garage_secret_key) come from the site’s websites_lamp entry (selected by fqdn == 'omeka-s.recroom.rrchnm.org'). Configure Omeka S’s S3 module to use this bucket: endpoint http://obj.rrchnm.internal:3900 (plaintext over LAN, Phase A; HTTPS once plan-object-storage.md Phase B+ lands), access key + secret from the per-site Secret.
• Site Pod manifest in k0s/lamp/recroom-omeka-s/ — Deployment with two containers: caddy (HTTP→FastCGI shim, minimal Caddyfile) and php-fpm (per-site image). Shared emptyDir for the FCGI socket. No PVCs — /var/www/html is in-image read-only; sessions go to MariaDB via Omeka’s DB session handler; ephemeral caches in emptyDir. ClusterIP Service fronting :80.
• HTTPRoute on the public Gateway for omeka-s.recroom.rrchnm.org → the new Service. ReferenceGrant in platform if the route lives in a separate lamp namespace.
• DB + S3 credentials via Secret. One Secret rendered from the catalog entry matching fqdn == 'omeka-s.recroom.rrchnm.org' — MariaDB db_user/db_password + Garage garage_access_key_id/garage_secret_key. DB host = the entry’s db_host (izanagi) → izanagi.rrchnm.internal (or its LAN IP) as env var, or an ExternalName Service if you want to keep the pod manifest portable across DB-host renames. Eventually ESO-materialized from OpenBao kv per the k0s-eso pattern; Phase 1 can use a hand-applied Secret.
• Cutover — public DNS for omeka-s.recroom.rrchnm.org resolves to the public edge’s F5 IPs (129.174.134.71/.72/.73 → the HAProxy tier → the public Gateway .150). The old caddy_proxies fragment for this site (if any) disabled; HAProxy rule for this site deleted. (Per todo-recroom-migration.md, recroom subdomains haven’t yet been pinned to a public IP — DNS gets set once for the new path, not switched.)

Verify:

• curl https://omeka-s.recroom.rrchnm.org/ returns Omeka.
• File upload through Omeka admin lands in the recroom-omeka-s-uploads Garage bucket (verify with aws --endpoint http://brahma.rrchnm.internal:3900 s3 ls s3://recroom-omeka-s-uploads/); same file served back through Omeka’s media URL.
• MariaDB connection works from the pod (check via mysql exec into the pod or via app behavior).
• borg backup on the db VM covers lamp_site_db. Garage’s per-bucket durability covers the uploads bucket (RF=1 single-node today; RF=3 once cluster mode flips).
• Killing the pod and letting it reschedule onto a different worker doesn’t break the site (proves the host-pin dissolved).

Phase 2 — generalize to remaining LAMP sites

Roll the pattern to the other backends (wordpress-rrchnm-internal, drupal-rrchnm-internal, omeka-rrchnm-internal, plus any added between now and execution).

• Per-site: repeat Phase 1’s Pod + HTTPRoute + Garage-bucket + S3-module-config pattern. Internal-only sites attach to the internal Gateway instead of public. Per-app S3 module: Omeka S (official), WordPress (WP Offload Media), Drupal (flysystem + flysystem_s3), Omeka Classic (verify community-plugin availability before that site moves — see §Risks).
• Tooling: turn the Phase-1 manifests into a Helm chart or kustomize base. Per-site values come straight from the matching websites_lamp list entry (php_version, fqdn, gateway, db_*, garage_*) — the chart/base reads one entry per site, selected by fqdn. The only field not in the catalog is the site container image (lives with the OCI build per CHANGELOG 2026-07-01).
• Caddy_proxies pruning: for each migrated site, remove its fragment from the caddy_proxies templates. The fleet shrinks to its static-site responsibility.
• HAProxy pruning: for each public site migrated, drop its HAProxy backend.

Verify: every migrated site responds correctly via its Gateway path; no LAMP-related fragments remain in caddy_proxies; HAProxy config carries only the surviving non-LAMP backends (which after migration is “all of them on the LAMP side” — HAProxy LAMP role disappears entirely).

Phase 3 — HA expansion (optional, evaluate per-host when single-instance outage tolerance breaks down)

The original Phase 3 was “consolidate the per-site db VMs onto a shared instance” — that work moved to Phase 0.5 (2026-05-29). What stays as Phase 3: when a single MariaDB host’s outage stops being acceptable (production traffic, SLA commitments, restore-time intolerance), add an HA replica for THAT host. Each per-site DB can scale independently — izanagi (RH461) might justify HA before izanami (AQ114) does, or vice versa.

Trigger criteria (any one is enough, scoped per-DB-host):

• The DB serves production traffic with an SLA.
• A single-instance outage takes too long to recover from (operator measures: borg restore time + DB recovery + cache warm-up).
• Read load justifies a read-replica even without HA pressure (CMS read scaling).

HA shapes (decided per-host when triggered):

Option	Pros	Cons
Async primary/replica (existing host → new replica)	Read scaling; warm restore target; simple to add	No automatic failover without a proxy layer (haproxy / ProxySQL / MaxScale); replication lag matters for CMS write-then-read patterns
Galera 2-node (existing host + new peer)	Sync multi-master; tolerates 1 node loss	Galera quirks (no DDL in transactions, deadlock-on-writes-to-same-row); 2-node Galera has split-brain risk without a 3rd arbiter
Galera 3-node	Full HA, no split-brain	3× resource cost; the 3rd node usually has to come from somewhere awkward (a different physical site, defeating the per-site locality); full Galera complexity

Defer the choice — single-instance per DB is fine until the trigger fires.

Naming notes for replicas. The izanami + izanagi names are already in use for the two per-site primaries. Replica naming convention TBD when the first replica lands — options: site-suffixed (izanagi-replica-rh461), child-of-deity (Kagutsuchi, the fire-god child of izanami in Shinto myth — fitting since the replica would be born “of” izanami), or numbered (izanagi-2). Decide at Phase 3 trigger time.

Phase N — cleanup / decommission

Most of the legacy LAMP-on-Incus cleanup actually happens in Phase 0.5 (the 4 recroom-*-rrchnm-org VMs disappear there). Phase N is the final documentation + naming sweep after Phase 2 is fully landed.

• Inventory: lamp_backends group entry in ansible/inventory.yaml deleted entirely (no surviving hosts; the historical name lives in this plan + the changelog, not in the live inventory). lamp_databases is the only post-decoupling LAMP-fleet group.
• Playbook naming: setup-lamp-databases.yaml (renamed from setup-lamp-servers.yaml in the 2026-05-29 IaC pivot) graduates to target the lamp_databases group instead of the now-defunct lamp_backends. Either rename to setup-database-servers.yaml (if a future non-LAMP DB host might also use it) OR keep as setup-lamp-databases.yaml (LAMP-specific by design). Recommend the latter for symmetry with the group name.
• group_vars migration: rename group_vars/lamp_backends.yaml + group_vars/lamp_backends.sops.yaml → group_vars/lamp_databases.{yaml,sops.yaml}. Then (Phase 0.6) lamp_sites, lamp_<site>_db_password, and lamp_garage_keys all fold into the single websites_lamp catalog at ansible/vars/websites-lamp.sops.yaml; group_vars/lamp_databases.sops.yaml + vars/lamp-garage.sops.yaml are deleted and group_vars/lamp_databases.yaml keeps only the db_version default. (The standalone lamp-garage.sops.yaml from the original Phase 0.5 decision was an intermediate shape — superseded.)
• TF cleanup: Phase 0.5 deleted opentofu/lamp.tf entirely (vestigial) and put izanagi in a new opentofu/mariadb.tf (clean separation; filename describes the scope precisely).
• Docs sweep: todo-recroom-migration.md reflects the migration path actually taken (Phase 0.5 + 1 + 2); public-ingress.md Phase D inventory adds the LAMP HTTPRoutes; AGENTS.md topology + inventory tables reflect izanagi + the retired group name.
• Changelog entry covering the architectural shift: decoupling → consolidation → group rename → playbook rename.

---

Risks

• Garage single-node SPOF. During the single-node bring-up period (until AQ114 disks land and garage_topology flips to cluster), brahma is the only Garage replica — its outage takes all migrated sites’ uploads offline. Same blast radius as today’s “single backend VM down = single site down,” just shifted to a different single host. Mitigation: the single→cluster transition is in-place via the flag — flip and re-run, no app-side change, Garage rebalances in the background. Until then, this is the same SPOF the fleet already has, not a worse one.
• MariaDB connection over LAN. Pod-to-VM crosses Cilium’s CNI → macvlan → LAN; latency higher than today’s same-host loopback. Mitigation: unlikely to matter for CMS workloads (queries are not the latency floor); measure during Phase 1, revisit if pages get slow.
• S3 storage module maturity per app. Omeka S has an official S3 storage module. WordPress has WP Offload Media (free tier covers Garage). Drupal has flysystem + flysystem_s3. Omeka Classic may need a community plugin or a Zend_Service_Amazon_S3-based shim — confirm availability + maintenance status before that site migrates. Mitigation: Phase 1 pilots with Omeka S (best S3 support of the four); Phase 2 verifies per app before rolling. If a site’s S3 path is genuinely unviable, that one site stays on the legacy Incus-VM pattern indefinitely — the hybrid is per-site, not all-or-nothing.
• Admin-UI plugin/theme installs stop working. WordPress lets admins install plugins through the UI; Drupal lets admins install modules; Omeka has similar surfaces. All write .php files into /var/www at runtime, which is now read-only-in-image. Mitigation: this fleet is operator-managed (Ansible + image rebuild via CI); plugin/theme additions become an ops responsibility, not a site-owner self-serve. Acknowledged design constraint, not a regression for this fleet — but worth communicating to site owners up front.
• Caddy in-pod vs. caddy_proxies divergence. Two caddies serving different traffic with different configs is a maintenance fork. Mitigation: the in-pod caddy is tiny (HTTP-to-FastCGI shim, no WAF/ratelimit — those can land in the Gateway envoy or stay at the caddy_proxies layer for non-LAMP traffic). Treat it as a thin shim, not a parallel fleet. If it grows, that’s a signal to reconsider.
• Cilium / k0s upgrade blast radius. LAMP availability now follows k0s lifecycle. Mitigation: Gateway envoy + workload pods are decoupled in upgrade cadence; pod evictions are smooth with two replicas per site. For single-replica sites that aren’t worth doubling, schedule upgrades in maintenance windows.
• Per-site image build for PHP version pin. Today’s per-site PHP version is a host_vars tag; in k0s it becomes a per-site container image. Mitigation: a php-fpm-base:<version> family of images covers the discrete PHP versions in use (8.1, 8.2, 8.3 — pick what’s actually deployed); per-site image is a thin layer with site-specific extensions + the S3 module config. Build in CI; push to thoth via the existing OCI pattern.

Rollback

For the greenfield recroom pilot (no production traffic + no live uploads yet), per-site rollback is trivial:

1. Delete the k0s HTTPRoute + Service + Deployment + Secret.
2. Re-create lamp_site_www for the site in opentofu/lamp.tf. Re-enable PHP-FPM + on-VM Caddy on the VM via the original setup-lamp-servers.yaml path.
3. Re-enable the site’s caddy_proxies fragment + HAProxy backend (revert the relevant git commit).
4. Re-point DNS at HAProxy.

lamp_site_db never moves; that side of the rollback is just “stop, switch back, start.” For sites that have accumulated uploads in their Garage bucket post-launch, an s3 sync s3://<bucket>/ <vm-path>/ step seeds the recreated lamp_site_www before cutover. For the greenfield pilot, that step is a no-op.

---

Open questions

• Should the new lamp namespace live in k0s/lamp/ or as a subdir of k0s/platform/? Argues either way: dedicated namespace matches the per-tenant model (Phase 3 consolidation argues for keeping it dedicated); folding into platform matches the “platform Gateway lives here” pattern. Decide before Phase 1 manifests land — cheap to flip then, annoying after.
• Does the Garage single→cluster flip need to happen before Phase 2 (multi-site rollout), or are sites OK landing against single-node RF=1? Strictly: each site has its own bucket, so RF=1 brahma-only storage works for any number of buckets. The only concern is outage during the single-node window. Probably: ship Phase 2 against single-node, flip to cluster mid-Phase-2 when the AQ114 disks arrive. Document this so future-us doesn’t conflate “Phase 2 ready” with “cluster mode required.”
• Multi-tenant MariaDB in Phase 3: shared instance with per-site users, or shared host with per-site MariaDB processes? The former is the standard pattern; the latter preserves blast-radius isolation at the cost of process overhead. Probably the former for non-prod sites and the latter for production; codify before Phase 3.
• WAF + ratelimit placement. Today these live in the caddy_proxies tier (per the recently-landed caddy rebuild). After LAMP moves to k0s, the public Gateway envoy is the natural place — but envoy’s WAF story is less mature than Coraza-in-Caddy. Open question whether to (a) accept “no WAF on LAMP sites” as a temporary regression, (b) run Coraza in the in-pod caddy, or (c) wait for a Gateway-level WAF posture. Probably (b) — copy the existing Coraza config into the in-pod caddy fragment.
• Does the in-pod caddy speak PROXY protocol from the Gateway envoy? Affects how client IPs propagate. Cilium Gateway can pass X-Forwarded-For; PROXY protocol is another option. Depends on what the apps need (e.g., Omeka access logs, security log enrichment).
• Per-site image build pipeline. A php-fpm-base:<version> image family + per-site thin layer needs an actual build path — Forgejo Actions runner (per plan-github-actions-runner.md)? Hand-built then pushed to thoth? CI doesn’t exist yet; needs to be settled before Phase 2 generalizes. For Phase 1’s single image, a hand-built artifact pushed to thoth is acceptable.
• Public DNS for the migrated sites. Today recroom subdomains haven’t been pinned to a public IP (per todo-recroom-migration.md). Post-migration they resolve to the public edge’s F5 IPs (129.174.134.71/.72/.73 → the HAProxy tier → the public Gateway .150). Confirm Route53 / campus DNS authority + the change process before Phase 1 cutover.

---

Decision log

2026-05-28 — initial draft

Hybrid framing settled after walking through three options in public-ingress.md follow-up conversation:

1. Pure k0s LAMP (PHP-FPM + MariaDB both in pods) — blocked by storage layer immaturity (local-path-provisioner only).
2. Status-quo LAMP-on-Incus, just relocated to AQ114 — doesn’t solve IP pressure, only buys site-resilience marginally.
3. Hybrid: PHP-FPM in k0s, MariaDB stays on Incus — captures the resilience + most of the IP savings + sidesteps the storage decision. Chosen.

Two open framings defaulted in this draft and noted as flippable: (a) 1:1 db VMs in Phase 1 (rather than starting with multi-tenant MariaDB), and (b) bypassing the caddy_proxies fleet for LAMP entirely (rather than keeping caddy_proxies in front and having it speak FCGI over the LAN to a k0s Service).

2026-05-28 (evening) — Garage replaces NFS for the www path

The earlier draft’s NFS-CSI prerequisite gets replaced with Garage S3-compatible object storage. Three driving observations:

1. Greenfield removes the migration constraint. No live recroom data on lamp_site_www means there’s nothing to preserve in-place. NFS’s main rationale — “keep the bytes where they are, just expose them differently” — evaporates. Migration cost was the strongest argument for NFS over S3; without it, S3 wins on every other axis.

2. Garage is the strategically-correct storage layer for this fleet. plan-object-storage.md already commits to it; brahma already exists; the garage_topology: single | cluster flag (added to group_vars/all.yaml 2026-05-28) means single-node operation works today without waiting for the AQ114 IronWolfs. Using NFS as a bridge tech we’d later replace with Garage would be two migrations instead of one.

3. Admin-UI plugin/theme installs are not required for this fleet. That’s the one feature NFS preserves and S3 doesn’t (PHP can’t execute .php files served from S3). For an operator-managed fleet, plugins and themes go through Ansible + image build; admins don’t self-serve. The cost of giving up admin-UI installs is zero here.

Consequences captured in the body edits:

• §Phase 0 pivots from “NFS-CSI driver” to “Garage single-node + per-site buckets + connectivity probe.”
• §Phase 1 drops NFS export setup + PV/PVC manifests; adds per-site image build + Garage bucket + S3 module config in the pod.
• §Risks drops three NFS-specific bullets (SPOF on the www path, NFS-CSI maturity, NFSv4 upload semantics) and adds two Garage-relevant ones (single-node SPOF mitigated by the topology flip; S3 module maturity per app). The admin-UI-plugin-install constraint becomes its own explicit risk bullet.
• §Rollback gets simpler (greenfield, no data to sync back) with a documented s3 sync step for any future site that rolls back post-launch.
• §Alternatives row 4 reframes the previous “Hybrid (this plan)” NFS variant as a rejected alternative with the rationale captured.
• §Open questions drops “NFS server placement in Phase 3” (moot) and adds “Garage single→cluster timing relative to Phase 2.”

2026-05-28 (late) — Phase 0 executed end-to-end

What landed in this session, in commit order on cc-brave-birch:

Commit	What
2dca96a	garage_topology single
5a487bf	nftables garage_cluster block — opens 3900/3901/3902/3903 from CLI + INT nets
20955e8	cert-renewal race fix in setup-oci-mirrors.yaml (caught while verifying make up survives teardown)
75863b2	Four per-site Garage buckets + scoped keys on brahma; credentials SOPS-encrypted in ansible/group_vars/lamp_backends.sops.yaml
fe4166f	LAMP MariaDB switched to LAN-bound: bind-address 0.0.0.0 + skip-name-resolve + host: '%' on the per-site user + nftables :3306 from INT_NET_SET
02338e3	Drop legacy <user>@localhost cleanup task (mysql_user keys by (name, host); the host: change created a duplicate rather than renaming)

Probe pods left no permanent state in k0s; bucket + key artifacts on Garage are intentional re-probe targets (they survived a make down/make up cycle in the middle of the session — confirming the volume-preservation contract).

Phase 0 verify steps that were declared in this plan and actually executed:

• ✅ Garage obj.rrchnm.internal reachable from a k0s pod (LIST + PUT + GET + LIST again, all via signed HMAC)
• ✅ Same pod can MariaDB into the db VM (CREATE TABLE / INSERT / SELECT / DROP via both hostname and IP)
• ⏳ public Gateway responds to an HTTPRoute attached to a dummy backend Service — not exercised. The Gateway is Programmed=True, but the external-HTTP-through-Gateway-to-pod path hasn’t been demonstrated this session. First task of Phase 1 should be to land a 5-line httpbin-style HTTPRoute on the public Gateway and curl it through F5 before the per-site image work begins.

Phase 1 is unblocked. The substrate (Garage, MariaDB, Gateways, in-cluster DNS) is all green; the next surface is application code + container images + HTTPRoute templating.

2026-05-29 — IaC pivot: lamp_backends become MariaDB-only, hard

What was “Phase 1 cutover for each site” in the original sequencing collapsed into a single maximal pivot of the LAMP IaC. Rationale: nothing was actually serving traffic on the four recroom backends yet, so the gradual-migration scaffolding (per-site mode flag, gated PHP tasks, lamp_db_only child group) added complexity without protecting any real state. Cleaner to commit to the terminal shape across the entire stack now.

What landed:

• Playbook renamed. setup-lamp-servers.yaml → setup-lamp-databases.yaml. PHP-FPM install + Sury repo + domain fact lookup + web-root-ownership all gone; only the MariaDB + per-site DB/user tasks remain. Group default in group_vars/lamp_backends.yaml collapses to just db_version: 11.8. Four host_vars/*-recroom-rrchnm-org.yaml strip php_version examples, php_extra_modules, cms flag, and the PHP commentary.
• Caddy decoupled from LAMP. setup-caddy-servers.yaml loses its Fetch user.domain for same-host lamp_backends task block + the Render per-site LAMP Caddyfile fragments task block; the orphan-fragment sweep’s when: no longer references lamp_domains. lamp/templates/lamp-site.caddy.j2 deleted. Caddyfile.j2 header rewritten to reflect static-only fragments.
• TF. opentofu/lamp.tf drops incus_storage_volume.lamp_site_www (resource + www device on the VM); only lamp_site_db survives. Makefile DOWN_EXCLUDES drops the lamp_site_www exclude. Each LAMP VM gets rebuilt by the next tofu apply (the www device removal changes the instance shape) — fine because nothing was serving from those volumes.
• nftables. lamp_backends block drops the per-caddy :9000 accept loop (keeps :3306 from INT_NET_SET). caddy_proxies egress block drops the :9000 allows (caddy no longer reaches PHP-FPM over the LAN).
• Docs. ansible/README.md group description + section header + per-site config table + tag table + usage examples all updated to reflect the MariaDB-only role.

Net result: the four recroom VMs are now pure MariaDB hosts. Phase 1 cutover for omeka-s is purely k0s-side — image build + Pod manifest + HTTPRoute + Garage S3 module wiring. No remaining lamp-backend-side work needed at the time of cutover. Phase 2 generalization is the same per remaining site. Phase 3 db consolidation still on the table, evaluated after Phase 2.

The “rebuild the VMs” operator step: tofu apply after this IaC lands triggers the VM rebuild because the www device removal changes the instance shape. PHP-FPM doesn’t survive the rebuild; the new VMs come up bare, then setup-lamp-databases.yaml brings them to MariaDB-only state. lamp_site_db is preserved (DOWN_EXCLUDES still covers it). Greenfield, so no DB content to preserve either.

2026-05-29 (late) — Pull db consolidation forward of Phase 1; izanagi on hyperion as the single consolidated MariaDB

What was “Phase 3 (optional, evaluate after Phase 2)” promoted to Phase 0.5 (pre-Phase-1-pilot). Same reasoning that drove the prior same-day IaC pivot from gradual-migration to hard-pivot: greenfield removes the migration cost that justified deferring. The 4 recroom-*-rrchnm-org VMs were throwaway MariaDB shells (no data, no traffic), so building them only to migrate-and-decommission later was wasted work.

Drivers:

1. Greenfield = no migration cost. No mysqldump+import step needed; just create per-site DBs + users on izanagi against the existing SOPS-encrypted db_password values. Same logic that drove NFS→Garage and gradual→hard-pivot earlier today.
2. LAN IP reclamation in RH461. The 4 legacy VMs occupy .136-.139 on RH461. Consolidating frees those for other RH461 uses; izanagi consumes 1 IP on AQ114 (the more-reliable site). Net: -3 IPs RH461, +1 IP AQ114.
3. Site placement bias. Per k0s-cluster-rebalance.md, AQ114 is the more-reliable site. Today’s 4 db VMs are all on kyojin (RH461) — moving the DB layer to AQ114 inherits that reliability.
4. Backup overhead. 4 borg-clients → 1 borg-client covering all 4 tenant DBs. Less retention math, less restore drill complexity.
5. Doing consolidation later means doing it twice. Building 4 db-only VMs in Phase 0 + Phase 1 just to dismantle them in Phase 3 is throwaway work.

Decisions made:

Decision	Choice	Note
Instance count	Single (izanagi)	Mirrors garage_topology: single → cluster flag pattern. HA via izanami deferred to new Phase 3 with explicit trigger criteria.
Placement	hyperion (AQ114)	Reliability bias + same-host operational pattern as existing services (Aesir, heimdall, ho-oh, arceus). QVO disk concern acknowledged; MariaDB’s write pattern less fsync-aggressive than etcd’s.
Naming	izanagi + izanami (Shinto creator deities, married pair)	Cross-pantheon fit with hyperion’s mixed naming (Norse Aesir + Pokemon birds + heimdall + ho-oh + arceus). Adds traditional Shinto alongside the existing anime-themed Japanese fleet names.
Group	New lamp_databases; lamp_backends retired entirely	Decoupling means “backend” framing no longer fits — PHP-FPM lives in k0s pods (not in any ansible group), MariaDB lives in lamp_databases. Keeps LAMP context (vs a generic database_servers group that could mean anything) + parallels the historical lamp_backends naming shape. lamp_backends group entry deleted from ansible/inventory.yaml; historical name preserved in this plan + changelog.
lamp_garage_keys placement	Move to ansible/vars/lamp-garage.sops.yaml (new explicit-load SOPS file)	The map is per-site k8s-pod config, not per-host DB config; izanagi doesn’t consume it. Co-locating in lamp_databases.sops.yaml would conflate DB-host secrets with k8s-pod secrets. Eventually ESO-materialized from OpenBao; this file is the bridge.

What changed in this plan:

• Status preamble: notes Phase 0.5 is the next-to-land scope, with rationale.
• New “Phase 0.5” section inserted between Phase 0 and Phase 1, with the full target shape, decision table, step-by-step (TF / inventory / Ansible / DB+user creation / backup / DNS / decommission), verify list, and one open question (new lamp_databases group vs reusing lamp_backends).
• Phase 1 simplified: the “repurpose omeka-s VM as db-only” step is gone (Phase 0.5 already did it). Pod’s DB host points at izanagi.rrchnm.internal.
• Old Phase 3 (“optional consolidation”) rewritten as new Phase 3 (“HA expansion to izanami when needed”) with explicit trigger criteria + the three HA-shape options (async, Galera-2 + arbitrator, Galera-3) documented for the future decision.

What this does NOT change:

• Per-site SOPS db_password values (already in the right shape — just consumed from a new central location instead of per-host).
• Per-site Garage buckets (already provisioned in Phase 0).
• Gateway / HTTPRoute / Pod manifest design (Phase 1 spec unchanged on the k0s side).
• Phase 2 generalization pattern (all sites point at izanagi, same per-site pod work).
• Rollback story (greenfield, same as before — Phase 0.5 itself is reversible by recreating the 4 db VMs from TF + restoring per-site DBs from mysqldump izanagi if needed).

2026-05-29 (late evening) — Revise Phase 0.5 from single-instance to per-site (izanami AQ114 + izanagi RH461)

Recognized that the late-afternoon “single-instance consolidation on hyperion” design left value on the table. Two adjustments:

1. DB locality. k0s pods that serve a LAMP site can schedule on any worker, but if the site’s data lives on a same-site DB, the pod can prefer that site’s workers for shorter DB-connection paths. Two per-site MariaDBs (one per physical site) make this affinity natural without any cross-site replication or HA complexity.
2. RH461 MAC reservation reuse. The legacy lamp_backends used RH461 .136-.139 MAC reservations (network-admin-coordinated). Putting one of the new MariaDB hosts on RH461 at .136 reuses an existing reservation slot — zero coordination needed. The .137-.139 slots get freed back to the placeholder pool.

Changed shape:

	Before (late afternoon draft)	After (this revision)
# MariaDB instances	1 (izanagi on hyperion)	2 (izanami on hyperion + izanagi on kyojin)
Replication	n/a	None — per-site independent
AQ114 placement	izanagi at .120	izanami at .81 (inherits the XCP-ng izanami LAMP-server slot — see follow-up entry below)
RH461 placement	none	izanagi at .136 (reuses omeka-s MAC)
Site assignment	All 4 recroom DBs → izanagi (only DB)	All 4 recroom DBs → izanagi (RH461-resident); izanami idle
lamp_sites schema	{site: {db_name, db_user}}	{site: {db_name, db_user, host}} — host: field added
Playbook loop	loop: lamp_sites | dict2items	Same + selectattr('value.host', 'eq', inventory_hostname)
DOWN_EXCLUDES	izanagi_db only	izanami_db + izanagi_db

Why now vs Phase 3. This is still Phase 0.5 (pre-Phase-1-pilot) — same greenfield window, same “do the right shape once” rationale. The earlier draft’s “single instance, HA later if needed” framing was a reasonable conservative choice; the per-site revision is a more honest reflection of the actual topology preference. Each per-site host CAN still get HA later (Phase 3) — that’s now scoped per-host independently rather than “expand izanagi into a pair.”

Naming clarification. izanami + izanagi are peer per-site DBs, not primary/replica — neither is “first” anymore. The Shinto-myth pair (married couple) still fits naturally; the geographic split (izanami AQ114, izanagi RH461) is arbitrary but stable.

2026-05-29 (night) — consolidate per-site LAMP vars into a single websites_lamp catalog

Map-shape decision superseded 2026-05-30 — the catalog still exists, still in one file, still under the same SOPS regex. The shape changed from map keyed by site to list of per-site dicts. See “switch websites_lamp from map to list” below for the rationale + cost (consumers hadn’t been written yet, so the cost was ~zero). Rest of this entry (file location, SOPS regex reasoning, controller-side-over-decryption tradeoff, leaf-name-vs-regex detail) still applies as-written.

Phase 0.5 left a single LAMP site declared across three files with two key conventions: lamp_sites (DB schema/user/host) in group_vars/lamp_databases.yaml, keyed omeka_s; lamp_<site>_db_password (flat) in group_vars/lamp_databases.sops.yaml; and lamp_garage_keys (bucket + access_key_id + secret) in vars/lamp-garage.sops.yaml, keyed omeka-s (hyphen — doesn’t even agree with the underscore key in lamp_sites). Adding a site meant editing three files and remembering each one’s key style. Phase 1 was about to pile on more per-site config (hostname, public/internal Gateway choice, PHP version, image ref, S3 endpoint) with no consolidated home — so the scatter was about to get worse, not better. Fix it before Phase 1 lands.

Decision: one websites_lamp map, one file — ansible/vars/websites-lamp.sops.yaml. Both consumers (setup-lamp-databases.yaml on the DB host + the Phase 1 k0s-render playbooks) load it via vars_files: and select by site key. Catalog + secrets live together; SOPS encrypts only the secret leaves via the existing vars/ encrypted_regex, so the structure stays diff-readable.

Structure — flat-per-site, leaf names chosen to match the EXISTING regex (zero .sops.yaml regex change):

websites_lamp:
  omeka_s:
    hostname: omeka-s.recroom.rrchnm.org
    gateway: public                 # public | internal
    php_version: "8.2"
    db_name: omeka_s
    db_user: omeka_s
    db_host: izanagi
    db_password: ENC[...]           # matches .*_password
    garage_bucket: recroom-omeka-s-uploads
    garage_access_key_id: GK58793…  # ends _id → fully-anchored .*_key does NOT match → stays plaintext ✓
    garage_secret_key: ENC[...]     # matches .*_key

The regex payoff: the vars/.*\.sops\.yaml$ creation_rule + its ^(.*_key|.*_htpasswd|.*_password|github_token_.*|garage_rpc_secret|garage_admin_token)$ regex already cover this file unchanged. db_password → .*_password; garage_secret_key → .*_key; garage_access_key_id ends in _id, and because the terms are fully anchored (^(...)$) the .*_key term skips it (stays plaintext — it’s public anyway). The only .sops.yaml edit is comment housekeeping: point the explanatory block at websites-lamp.sops.yaml and drop the lamp-garage.sops.yaml / lamp_databases.sops.yaml references. Flat-per-site (not nested db:/garage: submaps) is deliberate — a nested db: { password: … } leaf is named password, which .*_password does NOT match (no leading underscore); flat descriptive leaf names keep the regex trivial.

Key convention fixed: unify on underscore (omeka_s, omeka_classic), killing the omeka_s-vs-omeka-s split. Bucket names keep their hyphenated Garage-side form (recroom-omeka-s-uploads) but as a plain value field, no longer the map key.

What gets retired:

• lamp_sites map → websites_lamp.<site>.db_* + the new ingress/PHP fields.
• lamp_<site>_db_password flat vars → websites_lamp.<site>.db_password.
• lamp_garage_keys map → websites_lamp.<site>.garage_*.
• group_vars/lamp_databases.yaml shrinks to just the db_version: 11.8 group default; group_vars/lamp_databases.sops.yaml + vars/lamp-garage.sops.yaml are deleted.
• setup-lamp-databases.yaml swaps its lamp_sites | dict2items | selectattr('value.host','eq',inventory_hostname) for the same against websites_lamp + value.db_host, and its password lookup from lookup('vars','lamp_'+key+'_db_password') to item.value.db_password.

Tradeoff (conscious): single file → controller-side over-decryption. Because both playbooks vars_files: the one file, the DB-setup play decrypts the garage secret keys it never uses, and the k0s-render play decrypts the DB passwords it never uses. This is controller-memory over-decryption during the play, NOT host exposure — and it does not re-create the Phase 0.5 problem. Phase 0.5 moved garage keys out of group_vars precisely because group_vars auto-attaches to the host’s fact namespace (izanagi would carry the garage secrets as host facts). vars/*.sops.yaml is explicit-load: it never auto-attaches to izanagi; the secrets sit in the controller’s play vars and reach a host only if a task templates them there (and the DB tasks don’t reference garage_*). So the property Phase 0.5 cared about — the DB host doesn’t hold k8s-pod secrets — is preserved; what’s given up is the stricter “the DB play doesn’t even decrypt them.” If that stricter posture is ever wanted, split the secret leaves back into a second explicit-load file keyed by the same site key — the catalog shape is unchanged. Recorded so a future reader doesn’t mistake this for an accidental Phase-0.5 regression.

Supersedes the Phase 0.5 “Where lamp_garage_keys lives after the move” recommendation (standalone lamp-garage.sops.yaml). That was right when lamp_sites and the garage map were separate concerns; the catalog merges them, so the standalone garage file folds in. Eventual target is still OpenBao kv → ESO-materialized per-site Secrets (k0s-eso); websites-lamp.sops.yaml is the bridge — now one bridge file instead of three.

2026-05-29 (night, post-make up) — izanami IP correction: .120 → .81 (IP conflict with bare-metal XCP-ng baird postgres)

make up succeeded structurally but the setup-lamp-databases.yaml play against izanami exhibited bizarre SSH symptoms — TCP :22 reachable, host-key fingerprint inconsistent across attempts, “Permission denied (publickey)” despite byte-identical authorized_keys, sshd journal showing zero connection attempts. Burned a lot of debugging cycles on PAM/account-lock/key-perms before the operator caught the actual root cause: inventory.yaml:146 had a commented reservation note # .120 baird — bare-metal postgresql database server — .120 is already held by a legacy XCP-ng PostgreSQL server. The earlier IP survey for izanami missed the commented reservation; izanami at .120 was an ARP / routing collision with baird, which explains every symptom (packets sometimes reached izanami’s sshd, sometimes reached baird’s sshd which has a different host key + no ansi user).

Correction: izanami moves to .81 — the slot held by the XCP-ng izanami LAMP server this Incus VM is replacing per the k0s-LAMP-hybrid plan itself. Same name, same IP, continuity through the migration; the XCP-ng host gets decommissioned when the migration completes. The brief overlap during the transition window is operator-coordinated.

Lesson learned: future IP-survey passes for AQ114 allocations should grep both active hostvars entries AND the inventory’s commented reservation block. Adding a lint check for “TF resource declares static_ip not matching any documented reservation” would catch this class of bug at make lint time — recorded as a future-improvement item (no commit yet).

Touches in this correction: opentofu/mariadb.tf (IP + comment), ansible/inventory.yaml (izanami hostvars block moved to its IP-order slot right after ho-oh at .80, .81 reservation comment now points at the Incus VM, .120 reservation comment stays as baird), group_vars/lamp_databases.yaml, ansible/README.md, AGENTS.md topology + host description. Decision-log table above edited inline (AQ114 placement row now reflects .81).

2026-05-30 — Garage buckets named by FQDN; data classes under prefixes

Bucket-naming convention changed: per-site bucket = the site’s canonical FQDN (e.g. omeka-s.recroom.rrchnm.org), replacing recroom-<app>-uploads. Data classes live under top-level prefixes inside the one bucket — set by each app’s S3 module (uploads/ WordPress, original|large|medium|square/ Omeka S, files/ Drupal) — not separate -uploads buckets. The bucket stays the per-site isolation boundary (one scoped RW key); prefixes only organize within it.

Dots caveat → path-style only. A dotted bucket name is incompatible with Garage’s vhost routing (<bucket>.<root_domain>) + single-level wildcard TLS (*.obj.rrchnm.internal can’t match a multi-label fqdn.obj.…). So FQDN buckets must be accessed path-style (obj.rrchnm.internal:3900/<bucket>/<key>, use_path_style=true) and are not reachable via Garage’s bare web-endpoint vhost. Acceptable here: internal LAN, all S3 modules support path-style, and media serving is via app/Caddy/Gateway, never Garage’s bare web-vhost (dormant/Phase B). The dot-free alternative (omeka-s-recroom-rrchnm-org) was rejected — only needed if we wanted web-vhost serving, which we don’t.

Applied (repo): websites_lamp.omeka_s.garage_bucket → omeka-s.recroom.rrchnm.org; convention documented in the websites-lamp.sops.yaml header. The other three sites keep recroom-<app>-uploads until their cutover assigns an fqdn (then bucket = that fqdn).

Operator step (pending — Garage nodes were SSH-unreachable from the workspace at edit time): buckets are empty/greenfield, so rename via alias (no data move; the scoped key stays bound to the bucket ID):

garage bucket alias   <bucket-id> omeka-s.recroom.rrchnm.org
garage bucket unalias <bucket-id> recroom-omeka-s-uploads
garage bucket info    omeka-s.recroom.rrchnm.org   # verify

Until that runs, the catalog (omeka-s.recroom.rrchnm.org) is ahead of brahma (still recroom-omeka-s-uploads) — reconcile before Phase 1 wires the S3 module.

2026-05-30 — switch websites_lamp from map to list

Catalog stayed in one file, with the same SOPS regex and the same set of leaf names; only the top-level shape flipped from a map keyed by site (omeka_s / omeka_classic / drupal / wordpress) to a list of per-site dicts.

Why the original map shape stopped earning its keep. When the catalog was first sketched on 2026-05-29 night, the site key was the natural identity carrier — it WAS db_name, it WAS db_user, it was the bucket-name fragment in recroom-<key>-uploads. Phase 0.6’s own consolidation work + the 2026-05-30 garage-bucket-naming-by-FQDN entry above broke every one of those: db_name got its own field, db_user got its own field, garage_bucket migrated to == fqdn. By the time the other three sites were given an fqdn on 2026-05-30 (alongside omeka_s), the site key was a synonym for db_name (mostly) or a stripped form of fqdn — purely vestigial. fqdn is now the natural identity: canonical, unique-by-definition, never duplicated across sites, already the identity in DNS / TLS cert SANs / HTTPRoute.spec.hostnames downstream.

Cost-to-switch was effectively zero at the moment of the decision. grep across the tree for websites_lamp / websites-lamp consumers turned up: the file itself, plus prose references in this plan doc; no playbook had yet rewired to the catalog (Phase 0.6 is still implementation-pending — the file existed in the working tree but hadn’t been committed, and the setup-lamp-databases.yaml rewire hadn’t landed). Every actual consumer would be written against whichever shape this file held — which made today’s window the cleanest one Phase 0.6 was ever going to have. Deferring would have meant rewriting setup-lamp-databases.yaml + the Phase 1 pod-manifest render plays + the Phase 2 chart/base for the same payoff.

Tradeoffs accepted vs. the map shape:

Property	Map shape	List shape	Verdict
Uniqueness enforcement	YAML parser errors on duplicate keys	Needs an explicit lint (one-line check: len(fqdns) == len(set(fqdns)))	Lint pass is trivial; can fold into scripts/lint-inventory.py or a sibling. Net cost: small.
Direct field reference	websites_lamp.omeka_s.db_password	`websites_lamp	selectattr(‘fqdn’,‘eq’,‘omeka-s.recroom.rrchnm.org’)
Identity stable across hostname renames	Yes — key persists through fqdn changes	No — fqdn IS the identity, so rename = identity change	Hostname renames are rare-ops at this scale (the recroom subdomains were chosen years ago); the gain on the common path beats the cost on the rare path.
Loop syntax	`websites_lamp	dict2items	selectattr(‘value.db_host’,‘eq’,inventory_hostname)`
“Adding a site” UX	Invent a key matching the cross-source-underscore-convention, append a block	Append a block	Marginal.

What got touched in this commit:

• ansible/vars/websites-lamp.sops.yaml — shape change, header comment updates (replaced “Site key = underscore convention…” paragraph with list-shape rationale; replaced “Adding a site: append one block keyed by…” with list-entry append + lint-pass note; dropped dict2items from the consumer-pattern paragraph).
• plan-k0s-lamp-hybrid.md Phase 0.6 YAML example + step descriptions; Phase 1 prose references to websites_lamp.omeka_s → fqdn-selector form; Phase 2 reference to websites_lamp.<site> → “matching websites_lamp list entry”; superseded-note on the 2026-05-29 (night) catalog-consolidation entry; this decision-log entry.

What stayed the same: the file location, the SOPS regex (.*_password + .*_key + the rest of the vars/ group still catch the right leaves with no rule change), the controller-side-over-decryption tradeoff from the 2026-05-29 (night) entry, the per-site leaf names, the secret-vs-plaintext split. The 2026-05-30 garage-bucket-by-FQDN convention also carries through unchanged — the bucket field is now a peer of fqdn in the same list entry, rather than nested under a map key with the same fqdn field.

Open follow-up: add the fqdn-uniqueness lint to make lint. Not blocking — the consumer playbooks haven’t been written yet, so a duplicate fqdn today produces no user-visible failure. When the Phase 0.6 implementation lands, add the check alongside.

Test-driven discovery (caught during the verification pass for this switch). Wrote a localhost-only consumer simulation to prove the selectattr('db_host','eq', <host>) pattern works against the new list-shape catalog before any real playbook gets rewired. First version used vars_files: [vars/websites-lamp.sops.yaml] — copied straight from the 2026-05-29 (night) decision-log entry’s “load it via vars_files:” prescription and from this section’s own step #3 (pre-update). The decrypted db_password came out 151 characters long; the original cleartext is 32. The 151-char string was the literal ENC[AES256_GCM,data:...,type:str] raw form — vars_files: had loaded the file as plain YAML, no decrypt. vars_files: does NOT decrypt sops files, even though the .sops.yaml-rule regex catches them at edit time. The community.sops vars plugin only auto-decrypts files under host_vars/ and group_vars/; explicit-load files need the explicit lookup('community.sops.sops', file) | from_yaml pattern (established in the openbao playbooks). Rewriting the sim to use that pattern yielded db_password_len=30-32 (the real cleartext lengths) and the first 6 bytes matched the un-encrypted source — list-shape catalog works end-to-end.

The 2026-05-29 (night) entry’s vars_files: mention + this section’s pre-update step #3 were both wrong. Both have been corrected — Phase 0.6 step #3 now describes the correct lookup pattern; the 2026-05-29 (night) entry’s mention is left as-is with the superseded note pointing here, since this entry now carries the corrected guidance. Value of running the verification pass at all: caught a planning bug that would have shipped broken otherwise.

---

Last updated: 2026-05-30