QVO fsync cliff

The Tier-3 mitigation happened: theia’s and hyperion’s 860 QVO pools were replaced with 2× 870 EVO zfs-raid1 mirrors, and both the catastrophic hyperion_instances QVO pool and the interim hyperion_databases sub-pool no longer exist (fleet.tf pool selection is uniform again). Names and paths reflect May 2026: kube.tf → now fleet.yaml/fleet.tf; controllers are now erwin/zagreus/odin (jean/connie/sasha were reassigned to docker duty on atomsk); wait_for_cluster_quiet.yaml lives under ansible/playbooks/k0s/tasks/. The Prometheus-TSDB exposure flagged below was later addressed with a dedicated block volume. Still useful regardless of hardware era: the fsync-sensitivity workload taxonomy, the fio benchmark methodology, and the decision tree. Dated record; not maintained.

Originally an etcd-specific troubleshooting plan written 2026-05-16 after make k0s failed mid-pipeline with cascading API timeouts traced to slow etcd raft commits on QVO-backed ZFS pools. Broadened on 2026-05-23 once the same fsync-pSLC-cliff pattern showed up in other workloads:

OpenBao file backend on theia_instances (QVO raid10) under load
Grafana SQLite migrations on theia_instances — fresh DB on QVO took 13 min of migrations before the HTTP server bound (vs probably <2 min on the EVO host)
Prometheus TSDB WAL — same fsync-per-block-commit pattern, untested but exposure noted

The root cause for all of them is the same: Samsung 860 QVO pSLC cache exhausts under sustained random writes / high fsync rate, write latency goes from a few ms to hundreds of ms with multi-second tails. etcd was the loudest victim because k0s catches its own failures and bails; other workloads silently slow down without aborting.

This doc covers: which workloads are affected, which disks are at fault, how to diagnose, and the mitigation tiers (from “free cheapest” through “buy hardware”). etcd remains the canonical worst-case example for the diagnostic procedures.

Disk classes in the fleet

Disk class	Hosts	k0s nodes hosted	Behavior
Samsung 870 EVO 2 TB (raid1)	kyojin (RH461)	connie, sasha	Healthy. fsync p99 ~6 ms. Good general-purpose VM storage.
Samsung 860 QVO 2 TB (raid10/raidz1)	theia, hyperion (AQ114)	megaera, thanatos, thor, loki	fsync cliff under sustained random write. pSLC cache (~20 GB) exhausts, latency goes 50–500 ms+ with multi-second tails. Fine for sequential / low-rate / read-heavy.
WDC 250 GB SATA SSD (raid1, host `local` pool)	kyojin/theia/hyperion `local`	system disk, not VMs by default	Healthy. fsync p99 ~6 ms. Smaller but fast — useful as a `<host>_databases` sub-pool for fsync-sensitive controllers on slow hosts (deployed on hyperion).
WDC 1 TB SATA SSD (single, raid0)	ibuypwr (RH461)	no k0s	Not in the k0s data path.

The k0s scheduler doesn’t know about disk class — local-path PVCs land on whatever node the pod schedules on. So pod placement decides disk class for any PVC-backed workload.

Workload taxonomy: which are fsync-sensitive

Workload	fsync pattern	OK on QVO?	Why
etcd (k0s control plane)	Sync per raft commit	❌ Catastrophic	Sub-100ms target; raft elections flap above 1s
OpenBao file backend	Sync per kv write + token op	❌ Bad	Every secret read/write fsyncs the data file
Prometheus TSDB WAL	Sync per block commit	⚠️ Risky	Bursty during scrape intervals; tolerates some latency
Grafana SQLite	Sync per migration / dashboard save	⚠️ Painful on first boot	Slow but functional after migrations finish
Forgejo SQLite	Sync per write transaction (commits, issues, PRs)	⚠️ Slow under load	Git push is sync-heavy; light usage OK
Loki segments	Append-only, batched fsync	✅ Fine	Buffers writes, infrequent fsyncs
Zot blob storage	Large sequential writes	✅ Fine	pSLC absorbs the burst, no sync amplification
App pod ephemeral storage	Mostly reads + occasional small writes	✅ Fine	Not the QVO failure mode

The pattern that breaks QVO is “small synchronous write at high rate” — not “large total volume”, not “high read rate”. A workload that writes 200 MB total but fsync()s 1000 times is much worse for QVO than one that writes 20 GB in one big stream.

Symptoms observed

make k0s failures (across multiple re-runs):

etcdserver: request timed out (HTTP 500) on namespace + CNP creates
Internal error occurred: resource quota evaluation timed out
context deadline exceeded on kubectl-apply status-subresource patches
kubectl get --raw=/healthz/etcd returns context deadline exceeded instead of ok

Persistent etcd warnings on odin (k0s primary controller):

apply request took too long ... took: 3.077s, expected: 100ms
agreement among raft nodes before linearized reading (duration: 2.881s)
waiting for ReadIndex response took too long, retrying
local node read indexes queueing up and waiting to be in sync with leader

Even idle workloads (Cilium agents updating CiliumNode CRDs, kube-scheduler lease renewal, kube-controller-manager lease) generate enough write traffic to keep etcd backlogged.

Root cause (working hypothesis — twice-revised on 2026-05-16)

Two false starts before the right answer:

First hypothesis: “QVO fsync latency” — assumed all QVO drives were slow once pSLC cache was exhausted. Refuted by benchmark showing theia’s QVO pool in raid10 fsyncs in 3 ms (healthy).
Second hypothesis: “raidz1 small-write parity amplification on hyperion” — assumed the topology was the proximate cause given theia (raid10, healthy) vs hyperion (raidz1, catastrophic) on the same drive model. Refuted by rebuilding hyperion_instances as zfs-raid10 and re-measuring: still 4-9 IOPS, p50 ~150-200 ms, max latency >3 seconds. Topology change made no measurable difference.

Current ranking, with the corrected understanding:

hyperion_instances drives are individually unfit for sync-heavy workload — despite SMART reporting “passed” on all four QVOs (40,245 / 40,245 / 37,367 / 37,367 power-on hours). Same model + similar age as theia’s drives, which perform fine — so it’s not “QVO drives in general” but “these specific drives” or “this host’s storage subsystem”. Candidate explanations (none yet narrowed down): drive-specific wear that basic SMART doesn’t expose, write-endurance exhaustion (TBW), worn/exhausted pSLC zones, host-level hardware issue (SATA controller / cabling / backplane), or firmware-level cache state. The 3.4-second max-latency stalls observed during the 30-s sustained run suggest one or more drives intermittently freezes — consistent with garbage collection storms on a worn QLC drive.
No SLOG on the ZFS pools. etcd’s WAL fsync hits the main pool directly. A small mirrored SLOG (Optane M.2 or enterprise NVMe with PLP) would absorb sync writes and let the data pool handle only async background flushes — beneficial on every pool, and on a broken-drive host like hyperion it would mask the problem entirely (SLOG sees the writes; broken QVOs only see async background flushes which they handle fine).
Cross-site raft hop — 1 etcd member on RH461 (jean on kyojin) + 2 on AQ114 (zagreus on theia, odin on hyperion). Every write needs majority ACK = at least one ~1–2 ms cross-site round-trip. Bounded and small relative to the disk problem.
Shared ZFS pool contention. etcd’s data dir sits on the same pool as every other VM’s filesystem. Bursty write workloads from co-tenants (apt updates on workers, Cilium agent writes, container image pulls) compete for the same fsync queue — irrelevant if the pool is already the bottleneck.

Note on the raidz1 → raid10 rebuild: not wasted work. raid10 is the right topology for general VM-host pools (avoids raidz parity amplification on small writes, retains 1-disk fault tolerance per mirror pair, easy to grow by adding mirror pairs). The rebuild was correct on its merits — it just doesn’t fix the actual root cause for hyperion (the drives), so the etcd problem persists until that’s addressed.

fio fsync benchmark — 2026-05-16

Methodology: launch a short-lived Debian VM on each candidate storage pool, install fio, run an etcd-style sync workload (fio --rw=write --ioengine=sync --fdatasync=1 --bs=2300 --size=22m --runtime=10). Teardown after.

Host	Pool	Topology	Drives	IOPS	fsync p50	fsync p99	Verdict
hyperion	`local`	zfs-raid1	2× WDC SATA	—	—	—	(not measured, but kyojin+theia equivalents are healthy)
hyperion	`hyperion_instances` (raidz1)	zfs-raidz1	4× Samsung 860 QVO 2 TB	5	201 ms	464 ms	❌ Catastrophic
hyperion	`hyperion_instances` (raid10) — post-rebuild	zfs-raid10	same 4 QVOs	4-9	~150 ms	287-2903 ms	❌ Still catastrophic — topology not the cause
kyojin	`local`	zfs-raid1	2× WDC SATA	561	1.9 ms	5.9 ms	✅ Healthy
kyojin	`kyojin_instances`	zfs-raid1	2× Samsung 870 EVO 2 TB	257	4.8 ms	5.9 ms	✅ Healthy
theia	`local`	zfs-raid1	2× WDC SATA	521	2.0 ms	5.9 ms	✅ Healthy
theia	`theia_instances`	zfs-raid10	4× Samsung 860 QVO 2 TB	382	3.2 ms	6.4 ms	✅ Healthy

Target (etcd hardware reqs): fsync p99 < 10 ms. Only hyperion_instances fails — both before and after the raidz1 → raid10 rebuild.

Key finding (corrected): this is per-host drive condition, not topology. theia’s QVOs in raid10 are healthy; hyperion’s QVOs in either raidz1 OR raid10 are catastrophic. Same drive model, comparable power-on hours. Something about hyperion’s specific drives (or its SATA subsystem) makes sync writes 50-100× slower than equivalent drives on theia. SMART reports “passed” but doesn’t expose enough wear/endurance detail to confirm which specific drive(s) are at fault.

SMART power-on hours:

hyperion QVOs: 40245 / 40245 / 37367 / 37367
theia QVOs: 29134 / 39380 / 39380 / 39379
kyojin EVOs: (not captured yet, EVOs benchmarked healthy regardless)

Comparable ages — not the differentiator. Run a full smartctl -a per drive on hyperion to look at Wear_Leveling_Count, Total_LBAs_Written, Reallocated_Sector_Ct etc. to narrow down which drive(s) are degraded; that requires shell access to the drive (need a VM on the pool + smartctl --device=... or similar).

Diagnostic checklist

Run from the workspace container.

1. Confirm etcd health state

# Should return "ok"; "context deadline exceeded" = etcd unhealthy
KUBECONFIG=/configs/k0s/config kubectl get --raw=/healthz/etcd

# Detailed health (each subsystem)
KUBECONFIG=/configs/k0s/config kubectl get --raw='/healthz?verbose'

# Member list — verify all 3 are reachable
incus exec hyperion:odin -- k0s etcd member-list

2. Look at etcd’s own slow-warning logs

# Last 5 min of etcd warnings on each controller
for ctrl in hyperion:odin theia:zagreus kyojin:jean; do
  echo "=== $ctrl ==="
  incus exec $ctrl -- journalctl -u k0scontroller --since "5 min ago" \
    --no-pager 2>&1 | grep -iE "took too long|slow|election|leader" | tail -10
done

Look for apply request took too long > 1s, waiting for ReadIndex retries, leader election events.

3. Measure disk fsync latency (the smoking gun)

On each controller, with etcd running:

# Quick fsync-rate benchmark — etcd is most sensitive to this metric
incus exec hyperion:odin -- bash -c '
  cd /var/lib/k0s/etcd
  fio --rw=write --ioengine=sync --fdatasync=1 --directory=. \
      --size=22m --bs=2300 --name=etcd-fsync-test --runtime=10
'

Healthy etcd target: p99 fsync latency < 10ms. QVO with exhausted cache often shows 50–500ms. Optane / PLP NVMe shows <1ms.

Equivalent metric exposed by etcd itself (if metrics scraping is set up):

etcd_disk_wal_fsync_duration_seconds_bucket
etcd_disk_backend_commit_duration_seconds_bucket

4. Check pSLC cache state on QVOs (smartctl)

# Per-host on each IncusOS host with QVO drives
incus exec theia: -- smartctl -a /dev/sdX | grep -iE "wear|pe_cycle|host_writes|life"

Worn / used drives will show high Percentage Used (or low Available Spare). pSLC cache health is not a directly-readable SMART attribute on consumer QVOs — but high Total_Host_Writes correlates with cache wear.

5. ZFS-side latency

incus exec theia: -- zpool iostat -ly theia_instances 5 3

Look at the wait columns. Sync writes > 50ms = problematic for etcd.

Target storage architecture (per-host tier plan)

The fleet’s storage shape should be three pools per IncusOS host, with workloads assigned by I/O characteristic rather than ad-hoc co-tenancy. The tier names match the existing <host>_<purpose> convention.

Pool	Backing	Workloads	Why
`local`	2× WDC boot drives, zfs-raid1 (unchanged from today)	IncusOS state only	OS install mirror; IncusOS owns this. No user workloads belong here in the steady state.
`<host>_databases` (new tier)	1× or 2× TLC SSD / PLP NVMe — mirrored if 2+ drives	k0s controllers (etcd WAL); MariaDB on LAMP backends; Forgejo’s DB; OpenBao raft store; anything else explicitly fsync-bound	Capacity-modest, latency-critical. Sized for state stores (~30 GiB controllers + ~50 GiB DBs with growth → 256–500 GB NVMe is plenty). Per-VM `size` or pool `refquota` caps individual workload sprawl.
`<host>_instances`	4× QVO 2 TB, zfs-raid10 (post-2026-05-16)	k0s workers; caddies; LAMP doc-roots; DNS containers; static-site content; container image cache; everything not opting into `_databases`	Capacity-first; latency-tolerant. The “default” pool.

Opt-in is at the VM level. Each instance declares which pool serves its root device in opentofu/kube.tf (or the corresponding TF file). Default pool is <host>_instances. Workloads needing fast fsync explicitly point at <host>_databases. No automatic placement — keeps the decision visible in code review.

Status today (2026-05-16)

	kyojin	theia	hyperion	ibuypwr
`local`	✅ exists	✅ exists	✅ exists	✅ exists (single drive, no mirror)
`<host>_databases`	❌ pending hardware	❌ pending hardware	✅ interim sub-pool of `local` (committed in `0a7b5ce`)	n/a (no k0s on ibuypwr)
`<host>_instances`	✅ EVO mirror (healthy)	✅ QVO raid10 (healthy)	✅ QVO raid10 (unhealthy — see root-cause §1)	✅ single WDC raid0 (GPU VM only)

The <host>_databases tier is partially deployed via an interim sub-pool pattern — a ZFS dataset within local, registered as a distinct Incus storage pool. Same physical media as local (WDC SATA mirror), so it inherits the WDC mirror’s healthy fsync performance without IncusOS-level repartitioning or new hardware. Trial deployed on hyperion only as of 2026-05-16:

hyperion — odin runs on hyperion_databases (sub-pool). Etcd healthy: wait_for_cluster_quiet retries dropped from 111 (all-QVO baseline) to 0; each Wait for /healthz/etcd task completes in <0.5 s instead of stalling for minutes. Sub-pool fsync benchmark: 531 IOPS, p50 1.97 ms, p99 5.99 ms — equivalent to local directly.
kyojin — jean stays on kyojin_instances (healthy EVO mirror, 257 IOPS, p50 4.8 ms). No urgent need to move. Could provision kyojin_databases sub-pool for fleet symmetry; optional.
theia — zagreus stays on theia_instances (healthy QVO raid10, 382 IOPS, p50 3.2 ms). Same as kyojin: optional sub-pool for symmetry.

kube.tf controller pool selection is currently branched specifically for hyperion — see the pool = expression in opentofu/kube.tf. Extending to other hosts means provisioning their sub-pools first (in their setup-<host>.yaml), then dropping the remote == "hyperion" guard.

Why kyojin + theia stay on `_instances` (deferred sub-pool)

Worth capturing because “why isn’t the tier fleet-wide?” is the obvious next question. Three considerations, two of them weighing against rolling the sub-pool out to kyojin + theia now.

Perf: not needed. Both kyojin_instances (EVO mirror, p50 4.8 ms) and theia_instances (QVO raid10, p50 3.2 ms) are well under etcd’s 10 ms p99 target. There’s no active perf issue to solve.

Symmetry / future-migration cleanliness: weakly in favor. If <host>_databases sub-pools existed on all three hosts today, the eventual migration to dedicated hardware would be a pure “swap the backing of the named pool” operation — TF unchanged, just delete-and-recreate the pool on the new drive + incus copy --refresh the VM. Without them, when hardware lands for kyojin / theia, the migration also has to (a) provision the new pool, (b) edit kube.tf to drop the remote == "hyperion" guard, (c) incus copy --refresh per host. That’s ~30 minutes of extra ceremony per host, deferred — but it’s deferred, not avoided.

Wear: weakly against. Routing etcd writes through the WDC OS-disk mirror instead of the existing _instances drives shifts the wear pattern:

kyojin: etcd would move OFF the EVO mirror (Samsung 870 EVO 2 TB = ~600 TBW per drive) ONTO the WDC mirror (WDS250G2B0A = 100 TBW per drive). The EVOs have ~6× the endurance budget of the WDCs. Moving etcd to the WDCs accelerates OS-disk wear without any perf benefit — clearly the wrong direction for kyojin.
theia: etcd would move OFF the QVO raid10 ONTO the WDC mirror. QVOs are QLC with low TBW; WDCs are TLC with modest TBW but mirrored (every write hits both drives, while raid10 stripes across pairs). Roughly a wash, maybe slight WDC wear increase. No strong signal either way.

Decision (2026-05-16): defer kyojin + theia sub-pools until dedicated _databases hardware arrives. Reasoning:

YAGNI — both controllers are happy on their current pools; no problem to solve.
Wear — on kyojin specifically, moving etcd from EVO → WDC is a real downgrade for hardware lifetime. The EVO mirror is the right home for etcd there until real _databases hardware lands.
Migration cost is bounded — the future ceremony is well-understood and cheap (~30 min per host). Not worth pre-paying it now.

When dedicated hardware does arrive, do all three hosts together — same operation each host, drop the asymmetric guard in kube.tf, fleet ends in the clean three-tier shape.

If you ever want fleet symmetry sooner (e.g. for documentation or testing purposes), the mechanics are: replicate hyperion’s post_tasks: block in setup-kyojin.yaml + setup-theia.yaml, drop the remote == "hyperion" guard in kube.tf. Reversible with incus storage delete + incus copy --refresh back to _instances if you change your mind.

Sub-pool gotchas + the OS-disk wear concern

The sub-pool approach is off the documented IncusOS path — sub-pools aren’t visible in incus admin os system storage show, only at the Incus layer. Operational consequences:

Wear on the OS disks. Etcd’s sustained WAL workload is roughly 100-500 KB/s per member at steady state, ~17 GB/day → ~6 TB/year of logical writes per host. With ZFS amplification (~2-3× from metadata + parity-equivalent ops; mirror doesn’t double per-drive TBW) → 15-20 TB/year physical writes per WDC. The WDC WDS250G2B0A spec is 100 TBW, so realistic drive lifetime under sustained etcd workload is 5-7 years. WDCs are already at ~11k-12k power-on hours, so some endurance is already burned, but most of the budget remains. Acceptable for an interim but worth tracking.
Monitoring gap. IncusOS storage API only exposes enabled / passed / power_on_hours SMART fields — doesn’t surface Total_LBAs_Written or Wear_Leveling_Count. Without those, we can’t directly measure wear progress. Coarse proxy: track power_on_hours and pool_allocated_space_in_bytes deltas over time. To get detailed wear data would need physical-console smartctl -a access or extending IncusOS API access.

Drive (hyperion) Power-on hours baseline (2026-05-16) Serial
boot drive 12,511 174676805520
mirror partner 11,373 174676800882

Re-capture weekly. If pool_allocated_space on local grows faster than expected (controllers shouldn’t be writing many GiB of new data, just churning the same WAL), investigate.
Scope discipline. Don’t migrate other DB workloads (MariaDB, Forgejo, OpenBao) onto hyperion_databases while it’s a sub-pool. Etcd alone is ~6 TB/year writes; piling on more DBs would shorten WDC lifetime materially. Wait for real _databases hardware before consolidating.
Survives make down + re-run — playbook tasks in setup-hyperion.yaml are idempotent. Would need re-registration after a factory-reset.
IncusOS upgrades could theoretically rearrange datasets under local. Low risk (the local/databases namespace is operator-created, not in IncusOS’s dataset prefix list) but worth verifying after each IncusOS upgrade.

Drive (hyperion)	Power-on hours baseline (2026-05-16)	Serial
boot drive	12,511	174676805520
mirror partner	11,373	174676800882

See Tier 1.5 below for the mechanics; see “Migration path” above for the eventual transition to dedicated hardware.

Migration path (when fast SSD arrives per host)

Add the new drive to each k0s-hosting IncusOS host (kyojin / theia / hyperion / atomsk when racked). Requires a free SATA port or M.2 slot in the chassis; otherwise the new drive replaces an existing one.
Refactor ansible/roles/incus/storage-pools/ to accept a list of pool definitions (currently it handles one instances_pool + instances_disks triple). Estimated ~30-line change — replace the single set of vars with a loop. Defer until the first new drive lands so the refactor and the new pool’s first creation happen on real hardware.

Update each ansible/playbooks/incus/setup-<host>.yaml to declare both pools under a single list:

instances_pools:
  - name: <host>_databases
    type: zfs-raid1            # or zfs-raid0 if single drive — see "Redundancy posture" below
    disks:
      - { device: <new fast drive>, id: <ata-...> }
  - name: <host>_instances
    type: zfs-raid10            # current 4× QVO arrangement
    disks: [...]

Re-run the per-host setup playbook — role creates and registers the new pool; existing pools untouched.
Update opentofu/kube.tf to point k0s controller VMs at <host>_databases (was local interim, was <host>_instances originally).
tofu apply — controllers get recreated on the new pool. For warm migration with etcd data continuity, use incus copy --refresh <src> <dst> instead and adjust TF state pointers afterward.
(Optional, follow-up) Migrate other DB workloads onto <host>_databases once verified happy with controllers: LAMP backend MariaDB volumes (lamp_site_db), Forgejo PVC, OpenBao PVC, etc. Done piecemeal as the workloads’ fsync latency becomes a measurable concern.

Capacity + redundancy posture

Drive count per host	Pool layout	Capacity (256 GB drives)	Redundancy
1× drive	`zfs-raid0` (single)	~256 GB	None at storage layer. Acceptable for k0s controllers — etcd quorum across 3 hosts provides cluster-level fault tolerance. NOT acceptable for irreplaceable DB workloads.
2× drives	`zfs-raid1` (mirror)	~256 GB	1-disk fault tolerance per host. Right answer for DBs that aren’t covered by an external cluster-layer quorum.
2× drives	`zfs-raid0` (stripe)	~512 GB	None. Don’t do this. The capacity gain isn’t worth the redundancy loss for state stores.

Recommendation: start with 1 drive per host (single-disk _databases pool, leans on etcd quorum). Add a second drive later if non-etcd DB workloads (MariaDB, Forgejo, OpenBao) migrate onto the tier and need their own in-host redundancy. Cheaper to grow than to over-provision up front.

Mitigations — ordered by leverage

Tier 0: Free, no infra change — node-pin by disk class

The cheapest possible fix for any specific fsync-sensitive workload. Doesn’t change the storage layer at all — it moves the pod to a host that has the right disks.

Steps:

Label k0s nodes by their underlying disk class:

# kyojin's workers (EVO)
kubectl label node connie sasha rrchnm.org/disk-class=evo
# theia + hyperion workers (QVO)
kubectl label node megaera thanatos thor loki rrchnm.org/disk-class=qvo

Add nodeSelector or affinity to fsync-sensitive workloads. For the chart-managed ones it’s a values-file edit:
```
# openbao-values.yaml, grafana Deployment, forgejo-values.yaml, etc.
nodeSelector:
  rrchnm.org/disk-class: evo
```
For Helm charts that don’t expose nodeSelector directly, use affinity or post-deploy kubectl patch on the StatefulSet/ Deployment template.

Trade-off: workloads concentrate on the EVO host (kyojin). If kyojin goes down, you lose those workloads’ availability instead of spreading them across the cluster. For prototype workloads this is fine; for production you’d combine with replica anti-affinity + a second EVO host.

Suggested initial pins (in order of fsync sensitivity):

OpenBao (platform/openbao-0) — secret store, every read/write is a fsync. Highest priority pin.
Future Postgres — fsync-per-COMMIT, classic transactional workload.
Grafana, when fresh-DB migrations matter — biggest win for first- install time.
Prometheus — if you start scraping at high rate.

zot’s PVC, Loki’s PVC, app ephemeral storage: leave on QVO. They’re sequential or low-fsync workloads that QVO handles fine.

Tier 1: Free, immediate

Server-side kubectl apply + --request-timeout=5m — applied to all k0s playbooks (2026-05-16). Raises client tolerance for slow etcd commits.
pre_tasks: wait_for_cluster_quiet at top of every post-cluster playbook — added (2026-05-16). Gates on /healthz/etcd green before issuing writes. Lives at infra/ansible/tasks/wait_for_cluster_quiet.yaml in the repo.
retries: 5, delay: 15s on every k8s/kubectl task — added.

etcd timeout tuning in k0s.yaml — raise heartbeat and election windows so a slow node doesn’t trigger leader elections:

spec:
  storage:
    type: etcd
    etcd:
      extraArgs:
        heartbeat-interval: "250"     # default 100
        election-timeout: "2500"      # default 1000

Doesn’t speed anything up; reduces flap risk under stress.

Tier 1.5: Free, requires `make down` first — deployed interim

These are the moves available today without buying any hardware. The sub-pool item is the deployed fix on hyperion; the raid10 rebuild stays in place as architecturally-correct shape regardless.

Rebuild hyperion_instances as zfs-raid10 (committed 2026-05-16, 415d11f). Architecturally correct — raidz1 was a bad fit for small-write workloads on a VM-host pool — but did NOT fix the etcd problem on hyperion. Post-rebuild benchmark across three runs gave 4–9 IOPS and p50 ~150 ms, essentially unchanged from raidz1. The drives themselves are the proximate cause. Rebuild stays because raid10 is the right shape regardless; capacity cost (5.4 TiB → 3.6 TiB usable) is paid for shape, not perf.
Provision <host>_databases as a sub-pool of local (committed 2026-05-16, 0a7b5ce, hyperion only). ZFS dataset local/databases, registered as a distinct Incus storage pool, naming consistent with the target architecture so the future migration to dedicated hardware is a one-line pool-name no-op. Etcd on hyperion now sees the WDC mirror’s fsync latency (1.97 ms p50, 531 IOPS) instead of the QVO mirror’s (~150 ms p50, 4–9 IOPS). End-to-end make up runtime ~27 min with zero etcd-health retries — same shape as previous catastrophic runs but no more retry storms.
- For hyperion: deployed.
- For kyojin/theia: optional — their _instances pools are already healthy enough for etcd. The sub-pool would be for fleet symmetry only, not for solving any active perf issue. Mechanics if you want them: replicate the post_tasks: block from setup-hyperion.yaml into the respective setup-<host>.yaml, then drop the remote == "hyperion" guard in opentofu/kube.tf’s pool-selection ternary.
- Transition out when <host>_databases lands on real dedicated hardware: pool name doesn’t change in the IaC. Stand up the real (IncusOS-managed) <host>_databases pool, incus copy --refresh the controller VM onto it, then drop the sub-pool tasks from setup-<host>.yaml. The TF pool = ... ? "hyperion_databases" : ... line stays put.
(Optional, diagnostic) Capture full smartctl -a per drive on hyperion to narrow down which QVO(s) are degraded. Won’t fix anything alone, but informs the eventual drive-replacement decision (replace just the worst, or all four).

Whole VM root on `_databases` vs etcd-only on `_databases` (variant choice)

The <host>_databases tier (sub-pool today, real pool tomorrow) can carry either the whole controller VM root or only etcd’s dataDir. Trade-offs:

	Whole VM root on `_databases`	Only etcd `dataDir` on `_databases`
TF change	`pool: "<host>_databases"` for controller root	Extra disk per controller, mounted at `/var/lib/k0s/etcd`
Cloud-init	None	Add an `fs_setup` + `mount` block per controller
`_databases` space per controller	~10–15 GiB (full VM root)	~1–5 GiB (etcd data dir only)
Operational complexity	Low (one pool per VM)	Higher (per-VM extra disk + mount)
Reversibility	`incus copy --refresh` back to old pool	Snapshot etcd, unmount, re-attach
Blast radius if `_databases` fills	Whole controller VM stalls	Only etcd stalls (which is the whole game anyway)

Deployed choice (2026-05-16): whole VM root on the sub-pool. The space cost is negligible (250 GiB underlying local mirror; 3× ~20 GiB cap per controller = ~60 GiB max out of 250, IncusOS itself <10 GiB), and the operational simplicity (one pool per VM, no cloud-init disk juggling) is worth more than the per-VM space savings of the etcd-only variant. Same reasoning will hold when the tier lands on real hardware.

Tier 2: Cheap, requires reboot/restart (hardware-touching but no new purchase)

Move etcd data dir to a dedicated partition. k0s dataDir config lets you pin the etcd data location. Variant of the Tier 1.5 etcd-only flavor above — useful if Tier 1.5 doesn’t go far enough.
Stop running etcd VMs on slow pools entirely. Already covered by Tier 1.5.

Tier 3: Hardware investment

Add a mirrored SLOG to the ZFS pools — a small Optane M.2 (16–58GB models still findable used) or 2× small enterprise NVMe in a mirror. Costs ~$100–300. ZFS routes all sync writes to the SLOG; etcd’s WAL writes hit Optane (<1ms fsync) instead of QVO (50–500ms). Game-changer for every sync-heavy workload on the pool, not just etcd.
Replace QVOs with TLC EVOs, WD Red SA500, or enterprise SSDs (Samsung PM893, Intel D3-S4520). The QVO line was never appropriate for VM hosts — they’re QLC drives designed for archival read workloads.

Tier 4: Architectural — only if hardware can’t be fixed

Switch to k3s + kine + external PostgreSQL. Doesn’t fix the underlying disk-fsync cost; just moves it. Adds a Postgres operational dependency. Marginal gain (~20–30%) at best. Not recommended unless hardware is genuinely stuck.
Reduce to single-controller + kine + SQLite. Loses HA entirely. Only acceptable for dev/test, not the production-prototype target.

Decision tree

Is a specific workload slow, or is etcd flapping?
├── Specific workload (openbao timeout, grafana migration takes 13 min,
│   forgejo git push hangs, etc.):
│   ├── Which node is the pod on?  (kubectl get pod -o wide)
│   ├── Is that node on QVO (megaera/thanatos/thor/loki)?
│   │   YES → Apply Tier 0 (pin to EVO node) — single yaml change,
│   │         takes effect at pod restart. Cheapest possible fix.
│   │         If pinning isn't possible (HA constraints), continue.
│   │   NO  → Disk class isn't your problem; look at app config /
│   │         resource limits / network.
│   └── Run the fio benchmark on the underlying pool (Diagnostic §3)
│       to confirm fsync p99 if you want hard numbers before pinning.
│
└── etcd / cluster-wide instability:
    └── Is /healthz/etcd green right now?
        ├── Yes → No active problem. Run fio benchmark on each pool
        │         periodically — drives wear, pools fill, performance
        │         drifts. A regression below p99=10ms is your warning.
        └── No  → Run fio on the pool hosting etcd data.
                 ├── p99 < 10 ms → Disk is fine. Look at cross-site
                 │                 network latency (RTT kyojin <-> theia),
                 │                 etcd member health, controller CPU.
                 ├── 10 ms < p99 < 50 ms → Marginal. Tier 1 playbook
                 │                         hardening may be enough.
                 └── p99 > 50 ms → Disk is the problem. Then:
                     ├── Does `<host>_databases` exist on the affected
                     │   host? → Move the controller VM to it
                     │           (single-line TF change).
                     ├── Is the pool raidz1 with small-fsync workload?
                     │   → Rebuild as raid10 (Tier 1.5, free). May not
                     │     fix it if drives are the proximate cause.
                     ├── Is `local` healthy (run benchmark)?
                     │   → Move controllers to `local` as interim bridge
                     │     (Tier 1.5, free). Migrate to `<host>_databases`
                     │     when a fast drive arrives.
                     ├── Spare TLC/enterprise SSD on hand + free slot?
                     │   → Stand up `<host>_databases` for real.
                     ├── Budget for SLOG (~$100-300)?
                     │   → Add SLOG to the existing data pool (Tier 3) —
                     │     masks slow drives for ALL sync workloads on
                     │     that pool, not just etcd.
                     └── Neither, urgent?
                         → Tier 1 playbook hardening only; accept slow
                           cluster.

For “we want fewer fsync-sensitive workloads on QVO going forward” — default to Tier 0 (nodeSelector) for any new fsync-heavy app. Cheap, reversible, gets you out of the immediate problem without buying hardware. Save SLOG/disk-replacement budget for the workloads that can’t be pinned (typically: DaemonSets, or HA setups requiring spread).

Acceptance criteria for “fixed”

kubectl get --raw=/healthz/etcd returns ok consistently
etcd journal shows no apply request took too long warnings during steady-state operation
make k0s completes end-to-end in < 25 min without any task hitting its retry limit
fsync p99 on /var/lib/k0s/etcd < 10ms under fio synthetic load

References

etcd hardware recommendations: https://etcd.io/docs/v3.5/op-guide/hardware/
k0s storage configuration: https://docs.k0sproject.io/stable/configuration/
kine backends (k3s context): https://github.com/k3s-io/kine
ZFS SLOG sizing: rule of thumb 5 seconds × max write throughput; for etcd workload, even 8–16 GB is overkill

k0s-cluster-rebalance — controller placement strategy, AQ114 reliability bias
k0s-gateway-placement — gateway IP plan
Related Ansible task include: infra/ansible/tasks/wait_for_cluster_quiet.yaml (idempotent gate on /healthz/etcd green)