k0s cluster rebalance

The rebalance executed. As-built topology today: 3 controllers + 8 workers across four hosts — erwin (ctrl) + levi/hange on kyojin, zagreus + megaera/thanatos on theia, odin + thor/loki on hyperion, and atomsk racked workers-only (naota/canti, with haruko a reserved 5th controller) — i.e. the pending “atomsk controller decision” below resolved as option 2: preserve the AQ114 bias. The kyojin trio names in the tables (jean/connie/sasha) were later reassigned to docker VMs on atomsk, and the anticipated mabase/miyu-miyu names never shipped. fleet.yaml is the roster authority. The rationale sections (AQ114 bias, MAC conventions, phased sequencing) remain the operative thinking; the tables are a dated record and are not maintained.

Status: kyojin’s k0s 1+2 (jean/connie/sasha) is TF-live; theia’s 1+2 (zagreus/megaera/thanatos) is TF-live; hyperion’s 1+2 (odin/thor/loki) is TF-live; atomsk’s 1+2 reserved (commented in the locals map until racked). This doc captures the design rationale; the codebase now matches the target topology.

Renames applied to the body below (2026-05-29 single-pass pass). The Phases 1-4 narrative + per-VM detail tables originally used the pre-rename Norse names. To stop forcing readers to mentally double-translate, those names have been replaced with current names in-place. Full chain captured here so the historical record stays auditable:

Final (current) Original Intermediate Final-rename date Host context
jean (ctrl) jean — 2026-05-11 kyojin trio
connie (worker) connie — 2026-05-11 kyojin trio
sasha (worker) sasha — 2026-05-11 kyojin trio
zagreus (ctrl) zagreus — 2026-05-15 theia trio
megaera (worker) megaera — 2026-05-15 theia trio
thanatos (worker) thanatos — 2026-05-15 theia trio
osiris (ctrl, reserved) osiris jean (relocated) 2026-05-15 AQ114 4th-host slot (.117)
isis (worker, reserved) isis connie (relocated) 2026-05-15 AQ114 4th-host slot (.118)
horus (worker, reserved) horus sasha (relocated) 2026-05-15 AQ114 4th-host slot (.119)

Hyperion’s trio (odin/thor/loki) was always Norse — unchanged. MACs + IPs unchanged on all renames. Whether to align hyperion’s Aesir trio under the same Greek/Egyptian umbrella is an open question. The migration narrative in Phases 1-4 below describes intermediate states that didn’t all materialize literally — see the top-of-doc Status for current reality.

Final (current)	Original	Intermediate	Final-rename date	Host context
`jean` (ctrl)	`jean`	—	2026-05-11	kyojin trio
`connie` (worker)	`connie`	—	2026-05-11	kyojin trio
`sasha` (worker)	`sasha`	—	2026-05-11	kyojin trio
`zagreus` (ctrl)	`zagreus`	—	2026-05-15	theia trio
`megaera` (worker)	`megaera`	—	2026-05-15	theia trio
`thanatos` (worker)	`thanatos`	—	2026-05-15	theia trio
`osiris` (ctrl, reserved)	`osiris`	`jean` (relocated)	2026-05-15	AQ114 4th-host slot (`.117`)
`isis` (worker, reserved)	`isis`	`connie` (relocated)	2026-05-15	AQ114 4th-host slot (`.118`)
`horus` (worker, reserved)	`horus`	`sasha` (relocated)	2026-05-15	AQ114 4th-host slot (`.119`)

Note on the migration plan below (Phases 1-4): these phases were written against the original explicit-incus_instance.<name>-block topology. As of 2026-05-08, opentofu/kube.tf was refactored to a for_each over local.k0s_nodes — the resource address is now incus_instance.k0s_node["<name>"] and adding/moving a node is one map-entry edit, not a per-resource block edit. Read the phases for intent (drain, etcd member transitions, k0s rejoin sequencing) but translate references like “edit incus_instance.zagreus” to “edit the zagreus row in local.k0s_nodes.”

Why rebalance

The cluster grew organically and now skews heavily toward kyojin:

Host	Controllers	Workers	Total VMs
kyojin	odin, jean, zagreus (3)	freya, thor, loki, connie, sasha, tyr, surtr (7)	10
theia	(heimdall — drafted, not in current TF)	(megaera, thanatos — drafted, not in current TF)	0 in TF state
hyperion	(drafted, host not up)	—	—
ibuypwr	—	—	0 (hosts only the `logi` GPU VM)

Issues with the current shape:

Kyojin overloaded — 10 VMs on one host means a single failure takes out 100 % of controllers + 100 % of in-state workers. The “skadi temporarily removed to fit 8 GiB worker memory” footnote in AGENTS.md is a symptom of that pressure.
Even controller count once theia is back in state — 4 active controllers (odin/jean/zagreus on kyojin + heimdall on theia) means etcd has even quorum (still works, but odd is preferred).
No 1-host-loss tolerance for workers — if kyojin goes, the cluster keeps API quorum (heimdall on theia, plus eventually controllers on hyperion) but workload capacity drops to ~22 %.
Outdated drafted names on hyperion (yggdrasil/nidhogg/dvalin) — superseded by the current design.

Target shape

1 controller + 2 workers per IncusOS host. Four IncusOS hosts in the cluster (kyojin + atomsk on RH461, theia + hyperion on AQ114); ibuypwr stays out (client-subnet position, hosts only logi).

Host	Controller	Workers
kyojin (RH461 DHCP)	`jean`	`connie`, `sasha`
atomsk (RH461 DHCP, anticipated — `10.112.113.201`, MAC `4c:d9:8f:a5:c7:bd`, not yet racked)	`mabase`	`canti`, `miyu-miyu`
theia (AQ114 static)	`zagreus`	`megaera`, `thanatos`
hyperion (AQ114 static, drafted)	`odin`	`thor`, `loki`

Totals once all hosts up: 4 controllers + 8 workers = 12 active k0s nodes.

Currently active (kyojin + theia operational): 2 controllers + 4 workers = 6 active k0s nodes.

Plus reserved (future-host slot, AQ114 10.112.12.117-.119):

osiris (controller, .117), isis (worker, .118), horus (worker, .119) — Egyptian Osirian family. Names + IPs reserved for the eventual 4th IncusOS host on AQ114. Not active in the cluster until that host is racked.

Plus reserved (world-tree anchor):

yggdrasil at AQ114 10.112.12.110 — name + MAC + IP held in inventory; not part of the k0s cluster (neither controller nor worker). The mythology was originally that the Norns at .117-.119 tended Yggdrasil’s roots — .110 was the anchor for that AQ114-spanning theme. The Norse-Norns thematic tie was broken twice (2026-05-11 + 2026-05-15 — see rename table at top); yggdrasil’s slot remains as an inventory placeholder until assigned an operational purpose.

Plus standalone (not in k0s):

logi (ibuypwr, GPU VM)
backup2 (kyojin, USB JBOD VM)
test2 (kyojin, alpine container — diagnostic)
test1, test3 (theia, diagnostic)

RH461 DHCP allocation

.201   atomsk (host)   — anticipated; MAC 4c:d9:8f:a5:c7:bd
.202   kyojin (host)
.211   eren       — docker testbed VM on kyojin (setup TBD)
.212   mikasa     — docker testbed VM on kyojin (setup TBD)
.213   armin      — docker testbed VM on kyojin (setup TBD)
.214   jean       — kyojin controller
.215   connie     — kyojin worker
.216   sasha      — kyojin worker
.217   mabase     — atomsk controller (MAC `82:75:1a:ae:7a:09`; DHCP reservation pre-allocated, VM not yet provisioned)
.218   canti      — atomsk worker     (MAC `e6:a4:17:a1:0c:f9`; DHCP reservation pre-allocated)
.219   miyu-miyu  — atomsk worker     (MAC `2e:59:e0:0c:9b:f7`; DHCP reservation pre-allocated)
.252   backup2    — borg backup VM (USB JBOD passthrough)

RH461 MAC convention: MAC values ARE bound to specific campus DHCP reservations (the IP is assigned by the network keyed off the MAC). MACs are fully randomized locally-administered + unicast (no 52:54:00: QEMU prefix, no shared bytes between VMs) — same convention as AQ114. The difference between RH461 and AQ114 is only operational: RH461 MACs are committed (network admin has them in DHCP reservations); AQ114 MACs are cosmetic (cloud-init.network-config sets the IP, MAC is ignored). Don’t change RH461 MACs without coordinating with the network admin to update reservations in lockstep.

AQ114 static IP allocation

.65    gateway
.88    (free)  — was articuno, moved to .93
.89    (free)  — was zapdos, moved to .94
.93    articuno   — theia caddy (was .88; consolidated into .93–.95 block)
.94    zapdos     — theia caddy (was .89)
.95    moltres    — theia caddy (3rd; previously reserved/uncommitted)
.103   hyperion (host)
.104   theia (host)
.110   yggdrasil  — reserved (placeholder; not active in cluster)
.111   zagreus    — theia controller
.112   megaera    — theia worker (moved from .94)
.113   thanatos   — theia worker (moved from .95)
.114   odin       — hyperion controller
.115   thor       — hyperion worker
.116   loki       — hyperion worker
.117   osiris     — future host controller (Egyptian Osirian family)
.118   isis       — future host worker
.119   horus      — future host worker

Convention: .111 / .114 / .117 are controller positions, workers between them. .110 is a one-off reservation for yggdrasil outside the spaced-controllers pattern. .117-.119 reserved for a future 4th IncusOS host on AQ114 — names already assigned (osiris / isis / horus, Egyptian Osirian family) so the IPs aren’t anonymous; see Re-evaluation triggers.

AQ114 MAC convention: unlike RH461 (where MAC values are bound to specific DHCP reservations on the campus network), AQ114 uses cloud-init.network-config to set static IPs at first boot. The MAC plays no role in IP assignment on AQ114 — it’s purely the VM’s link-layer address. So AQ114 MACs are fully randomized with no shared prefix at all: each MAC has a random first byte (locally-administered + unicast: low 2 bits = 10, otherwise random) and 5 random remaining bytes. No 52:54:00: QEMU/KVM prefix — that prefix would announce “Linux KVM hypervisor” via MAC OUI, which is unnecessary fingerprint information for an internal VM. Matches the existing convention used for the RH461 MAC-reservation pool (host192-199).

Per-host detail

kyojin — Hel’s underground forge

Node	Role	IP	MAC	Notes
`jean`	controller	`10.112.113.214`	`ba:8e:24:00:71:c4`	Moved from `.233` as part of the RH461 cleanup (consolidating k0s IPs into a contiguous `.214-.219` block per kyojin/atomsk hosts; `.211-.213` reserved for docker testbed). MAC regenerated to fully-random LA-unicast (was `52:54:00:9c:4f:02`); DHCP reservation live. Operator-convention “primary controller” — bootstrap entry-point.
`connie`	worker	`10.112.113.215`	`aa:fb:9f:4e:2a:30`	Moved from `.218`. MAC regenerated (was `52:54:00:0d:00:02`); DHCP reservation live.
`sasha`	worker	`10.112.113.216`	`3a:37:1f:da:9c:54`	Moved from `.219`. MAC regenerated (was `52:54:00:0d:00:01`); DHCP reservation live.

atomsk — anticipated, not yet racked

Node	Role	IP	MAC	Notes
(TBD)	controller	`10.112.113.217`	`82:75:1a:ae:7a:09`	RH461 DHCP reservation pre-allocated; pick a name when atomsk is racked.
(TBD)	worker	`10.112.113.218`	`e6:a4:17:a1:0c:f9`	DHCP reservation pre-allocated.
(TBD)	worker	`10.112.113.219`	`2e:59:e0:0c:9b:f7`	DHCP reservation pre-allocated.

The RH461 .211-.219 block is allocated:

.211-.213: docker testbed (eren, mikasa, armin) — standalone Docker VMs on kyojin, not in k0s
.214-.216: kyojin’s k0s 1+2
.217-.219: atomsk’s k0s 1+2 (reserved)

theia — Njord watching the sun and moon

Node	Role	IP	MAC	Notes
`zagreus`	controller	`10.112.12.111` (static)	`32:0e:31:1f:7b:9c`	Relocated from kyojin (was `10.112.113.232` DHCP, MAC `52:54:00:3b:12:ba`) → theia static. MAC regenerated to a fully-random AQ114 value (no `52:54:00:` prefix); old MAC was an RH461 DHCP reservation that frees on retirement.
`megaera`	worker	`10.112.12.112` (static)	`3e:7a:60:65:03:5f`	Moved from `.94`. TF replace = destroy + recreate; drain workloads first. MAC regenerated to fully-random (was `52:54:00:bd:16:34`).
`thanatos`	worker	`10.112.12.113` (static)	`92:d9:2a:4b:ed:5e`	Moved from `.95`. Same drain-first caveat. MAC regenerated to fully-random (was `52:54:00:aa:df:c2`).

hyperion — the Aesir trinity (drafted, disabled until host is up)

Node	Role	IP	MAC	Notes
`odin`	controller	`10.112.12.114` (static)	`26:74:1c:50:24:4e`	Relocated from kyojin (was `10.112.113.231` DHCP, MAC `52:54:00:5d:9c:21`) → hyperion static. MAC regenerated to a fully-random AQ114 value (no shared prefix).
`thor`	worker	`10.112.12.115` (static)	`f6:3b:12:70:cc:75`	Relocated from kyojin (was `10.112.113.214` DHCP, MAC `52:54:00:0d:00:06`) → hyperion static. MAC regenerated — old value was a sequential `52:54:00:0d:00:0X` RH461 reservation.
`loki`	worker	`10.112.12.116` (static)	`ce:39:96:f4:47:d7`	Relocated from kyojin (was `10.112.113.215` DHCP, MAC `52:54:00:0d:00:05`) → hyperion static. MAC regenerated — same reasoning as `thor`.

Reservations (AQ114, not on a specific host)

Node	Role	IP	MAC	Notes
`yggdrasil`	reserved	`10.112.12.110` (static)	`16:13:6d:19:80:a3`	Inventory entry held as the world-tree anchor for the AQ114-spanning Norse theme. Operational purpose TBD; document when assigned. MAC fully randomized (no shared prefix).
`osiris`	future controller	`10.112.12.117` (static)	`ca:ef:72:e2:a4:cd`	Norn of the past. Reserved for the eventual 4th IncusOS host on AQ114.
`isis`	future worker	`10.112.12.118` (static)	`5e:07:df:95:9f:34`	Norn of the present. Reserved alongside `osiris`.
`horus`	future worker	`10.112.12.119` (static)	`da:20:55:fd:f9:c5`	Norn of the future. Reserved alongside `osiris`.

Mythological reading

Each host has an internally cohesive theme to make the topology memorable:

kyojin = Hel’s underground forge. Hel rules Helheim (the realm of the unheroic dead). She’s surrounded by two dwarven smiths: Brokkr and Sindri (the brothers who forged Mjölnir, Skíðblaðnir, and Gullinbursti for the gods). Dwarves traditionally inhabit underground realms — placing them with Hel keeps the “subterranean” theme tight. Tradeoff: had to drop Alvíss (the third dwarf) to fit 1+2 density.
atomsk (anticipated): theme TBD when racked. atomsk takes its name from FLCL’s “Pirate King” — the existing Norse-only theming on other hosts may or may not extend here; the operator picks at racking time.
theia = Njord watching sun-and-moon siblings. Njord (Vanir sea-god) plus Sól (sun) and Máni (moon). Theia in Greek myth was the titaness mother of the sun and moon — pleasing cross-pantheon coherence with the host name. Tradeoff: had to drop Skadi (Njord’s myth-wife) to fit 1+2 density.
hyperion = the Aesir trinity. Odin (allfather) + Thor (thunder) + Loki (trickster). The three Norse names everyone recognizes — most iconic cluster on the newest hardware. Tradeoff: had to drop Týr (war/justice) to fit 1+2 density.
yggdrasil + the Norns (reserved): the world-tree at .110 and the three Norns who tend its roots at .117-.119. Yggdrasil anchors the AQ114 range; Urd (past), Verdandi (present), and Skuld (future) bookend it. The Norns being literal embodiments of past/present/future is a perfect fit for “reserved for the future.” Strong cohesion: when a 4th host appears, the names are already there, already thematic.

What gets dropped

Node	Was	Reason
`freya`	live kyojin worker	1+2 density on kyojin only fits connie + sasha
`surtr`	live kyojin worker	same — recent transplant from the bricked-ibuypwr era, not load-bearing
`tyr`	live kyojin worker (was hyperion in earlier rebalance round)	1+2 density on hyperion only fits thor + loki
`alvis`	drafted in earlier rebalance round (never built)	1+2 density on kyojin
`skadi`	drafted in earlier rebalance round (never built)	1+2 density on theia
`heimdall`	was theia controller in earlier rounds (not in current TF state)	zagreus takes the theia controller slot under 1-ctrl-per-host
`nidhogg`, `dvalin`	drafted in earlier rebalance round (never built)	hyperion’s workers are now thor + loki (Aesir theme) instead of world-tree-creatures

Migration plan

The change touches opentofu/kube.tf, ansible/inventory.yaml, opentofu/system_images.tf (when hyperion comes up), and the narrative docs (AGENTS.md, the relevant subsystem READMEs). The migration is multi-phase to keep cluster availability bounded.

Phase 0 — Pre-flight

Drain workloads off the VMs that will be destroyed: freya, surtr, tyr (kubectl drain). They’re being deleted entirely.
Snapshot the cluster state in case rollback is needed: k0s backup on the controller saves etcd + cluster certs to a tarball. See k0s docs on backup — that’s the right tool for restoring cluster-wide state, not incus export.
Coordinate with network admin:
- 5 RH461 DHCP reservations become unused (see MAC reservation hygiene). Decide whether to retire or hold.
- kyojin VMs need DHCP re-reservation — jean moves from .233 → .214; connie from .218 → .215; sasha from .219 → .216. MACs were also regenerated (see per-host detail above) so the reservations are net-new (new MAC, new IP), not lease renewals.
- Confirm AQ114 10.112.12.110-119 block is free for our use (yggdrasil is already drafted at .110; the rest are new allocations).
- atomsk’s IP (when racked) needs assigning — coordinate as for any new RH461 host.

Phase 1 — Drop unused kyojin workers

Goal: shrink kyojin from 10 VMs to 7 (3 k0s + backup2 + test2 + zagreus/odin/thor/loki temporarily kept until they relocate).

Edit opentofu/kube.tf: delete incus_instance.freya, incus_instance.surtr, incus_instance.tyr. Edit inventory.yaml: drop those host entries and group memberships.
make lint (verifies TF↔inventory consistency).
tofu apply — destroys 3 VMs.
kubectl delete node freya thor surtr (or whichever names are in k0s state; cleanup if not handled by drain).
Verify: 7 active workers + 3 controllers (odin, jean, zagreus) + 1 future controller pending hyperion.

State after Phase 1: still all-on-kyojin, 4 controllers (etcd over-provisioned but stable). No HA gain yet — but no HA loss either.

Phase 2 — Relocate `zagreus` to theia, recreate megaera/thanatos at new IPs

Goal: theia hosts its 1+2 block (zagreus + megaera + thanatos).

Drain zagreus (it’s a controller — drain doesn’t apply directly, but stop scheduling to it: kubectl cordon zagreus and let etcd quorum hold via odin + jean + heimdall… wait, heimdall isn’t in current TF state).
Edit opentofu/kube.tf:
- Change incus_instance.zagreus from remote = "kyojin" to remote = "theia"; add cloud-init.network-config block matching theia’s existing pattern (see existing incus_instance.megaera/thanatos/heimdall blocks for template), with addresses: [10.112.12.111/26], parent = local.theia_nic, pool = "theia_instances", image = incus_image.vm_debian["theia"].fingerprint.
- Add (or recreate) incus_instance.megaera and incus_instance.thanatos on theia with new IPs .112 / .113.
Edit inventory.yaml: move zagreus, megaera, thanatos host entries into the AQ114 section with the new IPs; ensure k0s group memberships.
make lint; tofu apply.
Run k0s join procedure on the new theia VMs — for zagreus controller: k0s controller token create on jean, run k0s install controller --token-file <token> on zagreus. For megaera/thanatos workers: similar with k0s worker token. (Or setup-cluster.yaml if it handles incremental adds — verify the playbook’s idempotency on partial cluster.)
Remove the old etcd member (zagreus’s old address) from the etcd cluster: k0s etcd member-list then k0s etcd leave --peer-address <old>.
Verify: 4 controllers (jean + zagreus + odin + … wait, only 3 since heimdall isn’t in state); cluster spans kyojin + theia.

State after Phase 2: 3 controllers (jean + zagreus + odin), 5 workers (connie + sasha + megaera + thanatos + the kyojin-resident loki/thor not yet relocated). Wait — after Phase 1 we dropped freya/surtr/tyr; after Phase 2 we move zagreus (ctrl) to theia and recreate megaera/thanatos on theia. odin/loki/thor stay on kyojin until Phase 3.

Phase 3 — Hyperion bring-up + odin/thor/loki relocation

Goal: hyperion provisioned, odin moved from kyojin → hyperion, thor/loki follow.

Wait until hyperion is racked, flashed, IncusOS-installed, reachable as an Incus remote. Run setup-hyperion.yaml (re-enabled in Makefile).
Add "hyperion" to local.vm_image_hosts in opentofu/system_images.tf.
Edit opentofu/kube.tf:
- Change incus_instance.odin from remote = "kyojin" to remote = "hyperion"; add cloud-init.network-config for .114, parent = local.hyperion_nic, pool = "hyperion_instances", image ["hyperion"].
- Same for thor (.115) and loki (.116).
- Add yggdrasil as a reserved entry — but since yggdrasil isn’t a k0s VM, it’s actually just an inventory entry, not a TF resource. Decide: do we provision a yggdrasil VM at all, or leave its purpose TBD until needed?
Edit inventory.yaml: move odin, thor, loki host entries into the AQ114 .110-.119 block with new IPs.
Drain odin (it’s a controller, same caveat as zagreus). Drain thor/loki workers.
make lint; tofu apply — destroys old odin/thor/loki on kyojin, creates new on hyperion.
Run k0s join procedure on the new hyperion VMs. Remove old etcd member entries.
Verify: 3 controllers (jean + zagreus + odin), 6 workers (connie + sasha + megaera + thanatos + thor + loki). Cluster spans 3 hosts.

State after Phase 3: 9-node k0s cluster (3 hosts × 1+2). atomsk still pending.

Phase 4 — atomsk bring-up (when hardware arrives)

Goal: 4th IncusOS host on RH461 brings 1 ctrl + 2 workers to fill the .217-.219 slots.

Pick names for atomsk’s 3 k0s VMs (controller + 2 workers). Document the theme in docs/k0s-cluster-rebalance.md’s mythological-reading section.
Network admin coordination: assign atomsk’s host IP and 3 fresh DHCP reservations (continuing the 52:54:00:0d:00:0X sequential RH461 pattern is fine — pick MACs that don’t collide with retired ones from the bricked-ibuypwr era).
IncusOS install: the network seed is rendered at runtime from ansible/playbooks/incus/templates/network.yaml.j2 against the host’s inventory entry (mac_addr, ansible_host, network_mode/network_prefix/network_gateway); install-media seed lives at /configs/incus/seeds/install_<host>.yaml (operator-side, outside repo). Boot drive serial captured at install time goes in the install-side seed.
Storage setup: Ansible playbook setup-atomsk.yaml (mirror setup-kyojin.yaml) — creates the atomsk_instances ZFS pool.
TF additions:
- Add "atomsk" to local.vm_image_hosts in opentofu/system_images.tf.
- Add atomsk_nic and atomsk_gateway locals in opentofu/system_network.tf; add atomsk to local.tfbr0_subnets and local.host_nics.
- Add 3 incus_instance resources in opentofu/kube.tf (kyojin-style — RH461 DHCP, no cloud-init.network-config).
Inventory additions: uncomment the atomsk: host entry; add the 3 k0s VM entries; uncomment in incus_servers and the appropriate k0s_* groups.
make lint; tofu apply; setup-cluster.yaml --limit <new-VMs> to join them to k0s.

State after Phase 4: 12-node k0s cluster (4 hosts × 1+2). Final target topology. Etcd quorum: 4 controllers — even, so consider whether the Norn-reservation slot at .117 (osiris) should activate via a 5th host to bring it back to odd.

Migration order alternative — hyperion-first

The phased order above runs Phase 2 and Phase 3 sequentially; between them, the cluster has fewer controllers than the steady state (the transient HA dip discussed below). An alternative order avoids that:

Phase 1’ — drop unused kyojin workers (same as Phase 1 above).
Phase 2’ — bring up hyperion + relocate odin/thor/loki first (Phase 3’s TF and k0s steps).
Phase 3’ — relocate zagreus + megaera/thanatos to theia (Phase 2’s steps).

This keeps the controller count at ≥3 throughout. Tradeoff: requires hyperion to be up before the rebalance starts, so the migration is gated on hardware availability.

Recommend: if hyperion is available when starting, use the alternative order. Otherwise the sequential order is fine — etcd handles the intermediate 2-controller window if no controller fails during it.

Rollback

If a phase wedges:

TF state corruption: cp opentofu/terraform.tfstate.backup opentofu/terraform.tfstate (backup is auto-written on every apply).
Cluster state corruption: k0s restore from the Phase 0 etcd snapshot.
Stuck controller: k0s etcd leave from a healthy member to forcibly remove a misbehaving peer; re-join with fresh credentials.
Worst case (all controllers unrecoverable): rebuild from k0s setup-cluster.yaml against the new VM set. Workloads get re-deployed via deploy-workloads.yaml. Loses anything etcd was tracking that wasn’t in declarative manifests — usually negligible for this stack.

Open considerations

AQ114 reliability bias

Design intent: AQ114 is the more reliable site (datacenter-grade power/cooling/network vs. RH461’s research-hall environment). The cluster is deliberately AQ114-biased so that an RH461 outage degrades capacity but doesn’t take the cluster down.

Current bias (kyojin + theia + hyperion live; atomsk not yet racked):

Resource	RH461	AQ114	Bias
Controllers	jean	zagreus, odin	2-of-3 on AQ114
Workers	connie, sasha	megaera, thanatos, thor, loki	4-of-6 on AQ114
DNS resolvers	mesprit + uxie	azelf	2 RH461-side / 1 AQ114-side (all on-prem since 2026-05)

Survival behavior with current bias:

RH461 host (kyojin) dies → cluster keeps etcd quorum (2/3) and stays read-write; capacity drops to 4/6 workers.
AQ114 hosts (theia + hyperion) both die → quorum lost, cluster goes read-only; capacity drops to 2/6 workers.
Single AQ114 host dies → quorum survives (2 of 3 controllers, since the dead host had only one), capacity drops modestly.

Where the bias DOESN’T currently help: all 8 Cilium Gateway IPs are on the RH461 L2 pool, so RH461 outage still takes ingress down even though the cluster keeps running. Critical-service gateways need to migrate to an AQ114 L2 pool — see k0s-gateway-placement.

Atomsk decision pending: when atomsk is racked and mabase becomes a controller candidate, the bias choice becomes explicit. Three options:

Option	Resulting controller layout	Pros	Cons
Promote `mabase` to controller (default per current 1+2 design)	jean + mabase (RH461) + zagreus + odin (AQ114) = 2-2	Symmetric, more redundancy	Loses AQ114 etcd bias — RH461 outage now risks losing 2 of 4 controllers (still 1-fault-tolerant but no longer biased)
Keep atomsk workers-only (don’t promote mabase)	jean (RH461) + zagreus + odin (AQ114) = 1-2, unchanged	Preserves AQ114 bias intentionally	Atomsk’s 1-host-failure tolerance becomes asymmetric — losing atomsk is fine, but losing kyojin loses both jean and a controller-eligible-but-unused mabase
Promote `mabase` AND a 5th AQ114 controller (e.g. promote megaera or thanatos)	2 RH461 + 3 AQ114 = 5 controllers	Restores AQ114 bias with 5-controller quorum (2-fault-tolerant)	Breaks the “1 controller per host” invariant; etcd at 5 members is the practical max before Raft churn becomes a concern

Recommendation: decide before atomsk goes live. Default expectation in this repo’s design is option 1 (promote mabase) because the 1+2-per-host rule is structurally clean and 2-2 is acceptable. If we want to keep AQ114-biased after atomsk, option 2 (workers-only) is simpler than juggling 5 controllers.

This decision intersects with k0s-gateway-placement: if AQ114 is the planned home for critical-service gateways AND the etcd bias has been intentionally surrendered to even (option 1), then “AQ114 ingress survives RH461 outage” still holds (only ingress depends on AQ114 nodes being up, and ingress survival is independent of etcd quorum side). But if AQ114 is also where critical pod data lives, lost AQ114 quorum AND lost AQ114 nodes is a bigger blast radius — worth keeping in mind.

Transient HA dip during sequential migration

In the sequential phase order (Phase 1 → 2 → 3), between Phase 2 and Phase 3 the cluster has 3 controllers across 2 hosts (jean + zagreus on kyojin&theia + odin temporarily on kyojin). A whole-host failure during this window could lose 2 controllers (if kyojin goes — odin + jean both gone) and leave only zagreus, breaking quorum.

Mitigations:

Use the hyperion-first alternative order above.
Or keep the sequential order short — don’t let Phase 2 sit indefinitely without Phase 3.

Primary-controller convention

k0s itself doesn’t have a primary — controllers are HA peers electing leadership via etcd. The “primary” designation is purely operator convention: it picks the bootstrap entry-point (kubectl context default, setup-cluster.yaml’s k0s_primary_controllers group, kubeconfig fetch source).

This designation has migrated over time as the cluster’s shape and reliability story evolved:

Era	Primary	Why
Pre-rebalance	`odin`	Was on kyojin, the only IncusOS host with active VMs at the time.
Post-rebalance (Phase 4 → 2026-05-15)	`jean` (later renamed `jean`)	Sole controller on kyojin, then framed as the “always-up host.”
Current (2026-05-15+)	`odin` (on hyperion)	Aligns with the AQ114 reliability bias — the bootstrap entry-point lives on the more-reliable side. RH461 outage no longer affects the operator’s default kubectl context.

The Phases 1-4 narrative + per-host detail tables below predate the most recent flip — jean/jean is described as primary there, but that was the post-rebalance state, not the current state. Read with the era table above in mind.

MAC reservation hygiene

After migration, these RH461 DHCP MAC reservations become fully unused — the VMs that used them have either been dropped OR moved to AQ114 where they get freshly-randomized MACs (per the AQ114 MAC convention). The relocating VMs do not carry their RH461 MACs to AQ114.

RH461 MAC (no longer used)	RH461 IP	Was	Disposition
`52:54:00:0d:00:04`	`10.112.113.216`	tyr (dropped entirely)	retire or hold
`52:54:00:0d:00:05`	`10.112.113.215`	loki (relocated to hyperion `.116`, new MAC `ce:39:96:f4:47:d7`)	retire or hold
`52:54:00:0d:00:06`	`10.112.113.214`	thor (relocated to hyperion `.115`, new MAC `f6:3b:12:70:cc:75`)	retire or hold
`52:54:00:0d:00:07`	`10.112.113.213`	freya (dropped entirely)	retire or hold
`52:54:00:5d:9c:21`	`10.112.113.231`	odin (relocated to hyperion `.114`, new MAC `26:74:1c:50:24:4e`)	retire or hold
`52:54:00:3b:12:ba`	`10.112.113.232`	zagreus (relocated to theia `.111`, new MAC `32:0e:31:1f:7b:9c`)	retire or hold
`a2:7a:87:ca:43:99`	`10.112.113.191`	surtr (dropped entirely)	retire or hold

Coordinate with network admin to retire (frees DHCP table space) or hold (keeps the option to add more kyojin-hosted RH461 VMs later if hardware allows).

Why regenerate MACs on relocation rather than carry the existing one?

The RH461 MACs were chosen to fit campus DHCP reservation patterns (52:54:00:0d:00:0X is a sequential block) — those patterns have no meaning on AQ114 where IPs are bound by cloud-init.
Carrying the same MAC across hosts implies (incorrectly) that the MAC is significant. Randomizing makes it explicit that AQ114 MACs are decorative.
TF replace already destroys and recreates the VM on the new host; regenerating the MAC at the same time is free.
Avoids a possible (but rare) edge case where two unrelated systems reuse the same MAC across networks and confuse switches/monitoring.

`yggdrasil` and the Norns — reservations without operational purpose

This design preserves yggdrasil at .110 and the three Norns (osiris/isis/horus at .117-.119) as inventory entries with no immediate operational role. They’re documented thematic placeholders: Yggdrasil is the world-tree, and the Norns at its roots embody past/present/future — fitting for the “future host” slot.

Risk: placeholders without context decay into noise. Inventory entries with no mac_addr, no group memberships, and no clear purpose look like leftovers after a few months and either get cleaned up by mistake or accumulate into permanent confusion.

Two ways to mitigate:

Carry context in the inventory comment. Each placeholder gets a one-line comment explaining the reservation: “yggdrasil — world-tree anchor for AQ114 k0s theme; not active in cluster” and “osiris/isis/horus — reserved for future 4th IncusOS host on AQ114 (the Norns).” Future operators see why the IP is held.
Defer the reservation until commitment. Don’t add the entries until either (a) the future host is being racked, or (b) someone has a concrete plan for the yggdrasil slot. Keep inventory.yaml matching live state until then.

Recommended: option 1 (carry context). The IPs are cheap; the documentation overhead is small; the AQ114 k0s block reads more coherently with named slots than with anonymous gaps. Reconsider if no concrete purpose for yggdrasil materializes within ~6 months.

Re-evaluation triggers

Revisit this design when:

A 4th IncusOS host is added — the AQ114 .117-.119 slots are reserved for it (named osiris / isis / horus in inventory — Egyptian Osirian family). The 1+2-per-host pattern would extend naturally; controller count jumps to 4 (even — back to the same odd-quorum question). Consider whether the new host should be a worker-only host (no controller, drop osiris from the design) to keep odd quorum, OR add a 5th controller somewhere. Also intersects with the AQ114 reliability bias decision — a 4th AQ114 controller pushes the bias to 1-3 (4 controllers, 3 on AQ114), which strengthens AQ114 survival but means RH461 outage tolerance shrinks proportionally.
atomsk is racked — triggers the controller decision documented in AQ114 reliability bias above. Don’t promote mabase to controller without first deciding whether to preserve, surrender, or strengthen the bias.
kyojin’s RAM or storage becomes a sustained bottleneck — the 1+2 shape on kyojin (3 k0s VMs + backup2 + test2 = 5 VMs) leaves headroom but isn’t infinite.
ibuypwr gets MAC-reservation status on its client subnet (or moves to a reservation-friendly subnet) — at which point ibuypwr could host k0s VMs, taking the cluster to 4 IncusOS hosts. Same controller-count consideration as above.
Workload demand exceeds 6 workers’ capacity — bumping to 1+3 per host would give 9 workers but breaks the AQ114 3-IPs-per-host block layout. Either change the AQ114 range or accept non-contiguous worker IPs.