Architecture

Authority

This page describes the model. The data authority is fleet.yaml in the systems repo (read directly by both OpenTofu and Ansible), and the class registry is fleet_resource_class_spec in opentofu/fleet.tf. Repo mechanics (file indexes, playbook tags, edit checklists) live in the per-directory READMEs: ansible/README.md, opentofu/README.md, k0s/README.md.

Five independent IncusOS hosts — deliberately not an Incus cluster (see the de-clustering postmortem; cross-host moves are warm migrations) — across three GMU campus sites. OpenTofu provisions the VMs and containers, Ansible configures them, and everything deploys through a single make up.

The fleet

Host	Site / subnet	Runs
kyojin	RH461 · 10.112.113.128/25 (DHCP-reserved)	k0s controller (erwin) + 2 workers · docker fleet · caddy (groudon) · CoreDNS (mesprit) · apt-cacher · Garage (brahma) · backup2 (USB JBOD)
theia	AQ114 · 10.112.12.64/26 + .128/25 (static)	k0s controller (zagreus) + 2 workers · caddy (moltres) · CoreDNS (azelf) · HAProxy LBs (prometheus/epimetheus) · CI/CD runner (nidavellir) · Garage (vishnu) · Kanidm standby (muninn)
hyperion	AQ114 · 10.112.12.64/26 + .128/25 (static)	k0s controller (odin) + 2 workers · caddies (articuno/zapdos) · step-ca (heimdall) · apt-cacher · Knot DNS (arceus) · MariaDB (izanami) · HAProxy LB (atlas) · Garage (shiva) · Kanidm IdM (huginn)
ibuypwr	RH463 · 10.112.113.0/25 (client subnet)	GPU VM (logi + Ollama) only — no k0s (single-disk host)
atomsk	RH461 · 10.112.113.128/25 (DHCP-reserved)	k0s 2 workers (naota/canti; haruko reserved) · docker trio (jean/connie/sasha) · caddy (kyogre) · Zot mirror (thoth) · CoreDNS (uxie) · Valkey (rayquaza) · MariaDB (izanagi) · backup1

Hardware

Host	Chassis	CPU	RAM	Disks	GPU
theia	Dell R540	Xeon Gold 5218 — 16c/32t	64 GiB	2× 250 GB SSD · 2× 870 EVO 1 TB · 1× IronWolf 4 TB	BMC only
hyperion	Dell R540	Xeon Gold 5218 — 16c/32t	64 GiB	2× 250 GB SSD · 2× 870 EVO 1 TB · 1× IronWolf 4 TB	BMC only
kyojin	Dell R540	Xeon Silver 4214R — 12c/24t	64 GiB	2× 250 GB SSD · 2× 870 EVO 2 TB · 1× IronWolf 4 TB	BMC only
atomsk	Dell R540	Xeon Silver 4208 — 8c/16t	64 GiB	2× 250 GB SSD · 2× 870 EVO 2 TB	BMC only
ibuypwr	whitebox	Core i7-9700K — 8c/8t	32 GiB	1× 870 QVO 1 TB · 1× WD Blue 1 TB	RTX 2080 SUPER (passed to logi)

AQ114 is the more reliable site, so the k0s control plane is deliberately AQ114-biased — 2 of 3 etcd controllers and 4 of 8 workers live there, so an RH461 outage degrades capacity without losing quorum. Rationale: k0s cluster rebalance.

fleet.yaml — the single source of truth

Every instance is one entry in fleet.yaml: {host, ip, mac, fleet_resource_class, [additional_groups], [status], [volumes], [devices]}. The same file is read directly by both tools — OpenTofu via yamldecode in fleet.tf, Ansible via the custom fleet inventory plugin — so there is no second copy to reconcile and nothing to drift-lint. The file is IP-ordered and carries the full address-allocation map (gateways, F5 slots, free ranges, legacy XCP-ng occupants, reserved #hostNNN pool) as comments.

Two deliberately separate axes:

• fleet_resource_class — the single provisioning identity. The fleet_resource_class_spec map in fleet.tf turns it into kind (VM/container), image, NIC name, root pool, CPU/memory/disk sizing, cloud-init flavor, DNS source, secure-boot, and lifecycle. One class per host means no double-ownership is possible. The class name is also its 1:1 Ansible group (when declared in group_tree).
• additional_groups — extra Ansible config membership (sub-roles like docker_rootful, cross-cutting groups). Not read by OpenTofu. The common case is no additional_groups at all — a non-empty list always signals something non-obvious.

The inventory plugin synthesizes the whole group tree from fleet_resource_class + additional_groups + group_tree; site/cidr/gateway derive from each instance’s ip against the networks table. Adding or changing a host is a one-entry edit — tofu plan and Ansible both pick it up.

Per-site addressing conventions

• AQ114 static (10.112.12.64/26): VMs configure their IP at first boot via cloud-init.network-config — no DHCP on this subnet. MACs are fully randomized (locally-administered + unicast) and decorative; the IP is set inside the guest, not by the network.
• RH461 DHCP (10.112.113.128/25): VM MACs are bound to campus DHCP reservations — the network assigns the IP keyed off the MAC. Changing a MAC requires a reservation update in lockstep with network admin.

Class sizing (indicative)

The authoritative registry is fleet_resource_class_spec in fleet.tf; representative allocations (8 GiB is the container root-disk floor):

Class	Kind	CPU	Mem	Root disk
docker	VM	4	8 GiB	64 GiB
k0s_workers	VM	4	8 GiB	32 GiB
cicd_servers, gpu	VM	2–4	4–8 GiB	32 GiB
k0s_controller	VM	2	4 GiB	20 GiB
lamp_databases	VM	2	4 GiB	12 GiB
haproxy_servers	VM	2	2 GiB	10 GiB
backup_servers	VM	2	8 GiB	8 GiB (data on JBOD volumes)
apt_proxies	container	1	1 GiB	20 GiB
oci_mirrors	container	2	4 GiB	10 GiB
other containers (caddy, dns, garage, step-ca, valkey)	container	1–2	1–2 GiB	8 GiB

Volumes and lifecycle

Instances are cattle; volumes hold the irreplaceable state. Instances carry no prevent_destroy — they rebuild from cloud-init + Ansible. Volumes are declared per-instance in fleet.yaml and split into two OpenTofu resources by protection: incus_storage_volume.protected (prevent_destroy = true) for state that must survive anything (Garage meta+data, caddy static-sites, the Zot cache, Valkey data, Ollama models, the Kanidm directory), and .unprotected for state that is reproducible from catalogs (the MariaDB db volumes). The authoritative list is whatever carries protected: true in fleet.yaml.

Operationally, make down preserves the protected set via the Makefile’s DOWN_EXCLUDES; a protected volume missing from that list makes make down fail loudly rather than destroy it. terraform.tfstate is committed to git — no remote backend, no locking, operator discipline only.

One sharp edge worth knowing before placing state: a filesystem-typed volume reaches a VM over virtiofs, which cannot back shared file mappings — anything that mmaps a shared file (databases, SQLite WAL) needs type: block. Full symptom table and the 10-second test: virtiofs and MAP_SHARED.

Kubernetes and ingress

k0s v1.32.2 — 3 controllers + 8 workers, all Incus VMs, two workers per R540. Cilium CNI with kube-proxy replacement and Gateway API ingress; L2 load-balancer announcement across three pools. Ingress follows the three-tier shared-Gateway model — public (reached through the janus/portunus/terminus HAProxy tier), internal (LAN-only), and devl-vdi (RH461) — one Gateway per trust tier, HTTPRoutes attaching cross-namespace; Gateway addresses live in ansible/vars/k0s-gateways.yaml. Design history: k0s public edge; networking internals: k0s networking.

Deploy pipeline

make up is a serial chain whose order encodes the cross-subsystem dependencies:

1. incus-hosts — per-host storage pools, networks, profiles
2. dns — the internal DNS containers first (fleet VMs resolve only against them)
3. infra — the full tofu apply (the fleet now resolves cleanly)
4. step-ca + haproxy (after package-proxies as shared prereq)
5. object-storage + oci-mirrors
6. caddy + lamp + backup + docker + cicd-servers
7. k0s — cluster, namespaces, network policies, per-namespace deploys, GitOps (Argo CD syncs the vibes stacks from main)

Every stage is idempotent and individually runnable. The vibes workloads deploy via Argo, not Ansible — committing to main is the deploy action for those.