ibuypwr brick

Historical postmortem — April–May 2026; closed September 2026

Written 2026-05-08, two days into the post-recovery stability window. That window has now held ~4 months, meeting the note’s own resolution criterion (“absent ~30+ days → resolved with an asterisk”) — suspect 1, the failing consumer boot SSD, stands as the effective conclusion. IaC paths reflect May 2026 (opentofu/gpu.tf → now fleet.yaml/fleet.tf; -target=incus_instance.logi → incus_instance.fleet["logi"]). This is a dated record and is not maintained.

Resolution status

Symptoms ceased after the 2026-05-06 recovery. Stable since. ibuypwr (now 10.112.113.90 on client subnet 3, hosting a single GPU VM logi) has run a sustained workload (NVIDIA RTX 2080 SUPER passthrough + Ollama inference) without recurrence of the ~45-70 min uptime brick that defined the 2026-04 incidents.

Root cause not definitively isolated. Three things changed at the same recovery event, which makes attribution to any single one impossible from this evidence:

1. Boot SSD replaced — old 480 GB SSD swapped for a fresh Samsung 870 QVO 1 TB.
2. CMOS battery replaced — pre-emptive, cheap to do alongside the SSD swap.
3. Fresh IncusOS install on the new SSD.

The leading suspect is (1) prior boot SSD failing (controller fault, NAND endurance hit, or firmware bug under sustained writes; the consistency of the ~45-70 min running-time pattern is consistent with a heat-soak or write-pattern failure mode). (2) and (3) are weaker candidates — see Leading suspects.

ibuypwr is no longer in the k0s topology (client-subnet DHCP can’t reserve stable IPs for cluster nodes), so even if the underlying hardware is marginal, the blast radius is bounded to losing the GPU VM. That changes the cost calculus from the 2026-04 era and is part of why the investigation was wound down rather than driven to root cause.

What changed since the original investigation

This file was previously ibuypwr-troubleshooting.md, written as an active-investigation log. It’s been reframed as a post-mortem because:

• The brick pattern stopped after recovery; no new evidence to act on.
• The diagnostic plans + “things to try” lists are moot.
• The host’s role changed (k0s controller → standalone GPU VM host), so future bricks if they recur have lower stakes.

What’s preserved: the timeline, the falsified hypotheses (real evidence about what isn’t the cause, useful precedent for any future hardware-suspect investigation), and the GPU-passthrough config-layer explainer.

Original symptoms

• Host became unreachable ~45-70 min after make up finished. TCP 8443 (Incus API) and TCP 22 (SSH) both timed out from kyojin.
• incus cluster list kyojin: showed ibuypwr OFFLINE with “No heartbeat for Xh…” (the cluster era — kyojin/theia/ibuypwr were a 3-member Incus cluster at the time).
• VMs on ibuypwr (njord, tyr, surtr in that era) went to ERROR state.
• Physical reboot resulted in IncusOS hanging indefinitely at “loading incus application”.
• Only known recovery was wipefs -a on every disk and reinstall — strongly suggesting on-disk corruption rather than memory or compute fault.

Timeline

Date	Deploy finish	Brick detected	Uptime before brick	GPU passthrough	VT-d (firmware)	IncusOS version
2026-04-20	~16:17	~16:34	~17 min	Yes	on	202604150351
2026-04-21	~16:14	~17:23	~1h 9min	Yes	on	202604150351
2026-04-22a	~09:30	~10:23–10:28	~55 min	No (surtr disabled)	on	202604202240
2026-04-22b	~11:00	~11:47	~47 min	No	off (BIOS)	202604202240
2026-05-06	(recovery event — boot SSD swap, CMOS battery, fresh install on Samsung 870 QVO 1 TB)
2026-05-06 to current	n/a	no recurrence	sustained (days)	Yes (logi)	on	202604282312

Both 2026-04 brick sessions occurred after make up / make workloads (Helm chart installs, especially Tetragon + Cilium operator activity). Run 2 had 4 GiB controllers + 8 GiB workers (vs 2+6 in run 1) and still bricked, ruling out VM-level memory sizing as cause.

Hypotheses ruled out

These were tested explicitly during the 2026-04 investigations. None caused the brick:

• VM-level OOM (controllers / workers at ~90% memory limit) — bumped limits, brick still happened.
• ZFS pool space exhaustion on local — 2026-04-22 test confirmed local was 412 GiB; peak usage during run was 4.3 GiB (~1%). Disk-fill implausible.
• Inherent Cilium/eBPF incompatibility with ibuypwr hardware — njord (controller) and tyr (worker) ran on ibuypwr for extended periods in other sessions without issue.
• GPU passthrough / VFIO — 2026-04-22a test disabled surtr + GPU passthrough entirely; brick still recurred at ~55 min.
• IncusOS version-specific bug (202604150351) — brick also happened on newer 202604202240.
• Host-level memory pressure on ibuypwr — last sample before brick showed only 5.9-6.1/32 GiB (~19%) used. ZFS ARC couldn’t explain a spike given this much idle RAM.
• IOMMU (VT-d / AMD-Vi) — 2026-04-22b test disabled VT-d in BIOS; brick recurred at ~47 min.

The 2026-04 conclusion was: every variable changeable via config has been cycled and the brick still recurs at ~45-70 min. That left only hardware-level or deep-system causes — none cheaply testable with the equipment available at the time.

Leading suspects (post-recovery)

Ranked by consistency with the evidence + the SSD-swap correlation:

1. Prior boot SSD was failing (most likely)

Consistent with:

• Sustained-uptime failure pattern — a heat-soaked or wear-leveling-saturated SSD controller can pass POST and run cleanly for tens of minutes before degrading under sustained writes (k0s + Cilium + Tetragon are write-heavy at deploy time).
• wipefs + reinstall recovery — re-laying the partition layout would temporarily mask a failing controller until the fault recurred. Matches the “recovers, bricks again, recovers” cadence.
• Symptoms ceased after replacing the SSD — strongest single piece of evidence.
• Consumer-grade SSD origin — the 480 GB original was a consumer SSD; the brick pattern is consistent with consumer-grade firmware/controller behavior under server-style sustained load.

Inconsistent with: nothing observed in the 2026-04 forensics flat-out contradicts this. The SMART data wasn’t captured pre-brick (no IPMI/BMC access, host went unresponsive too fast), so we can’t show post-hoc “the drive was throwing errors” — just that the symptom stopped after replacement.

2. Reinstall cleared corrupt IncusOS state (medium)

Consistent with: the only known 2026-04 recovery was full wipefs + reinstall; the recovery event in 2026-05-06 was also a fresh install. Reinstall on the same drive could have cleared the brick previously and just hadn’t been tested as a low-effort intervention. Possible that the on-disk corruption was the proximate cause and the SSD itself was fine — but corruption-without-failing-storage usually has a mechanism (kernel bug, sudden poweroff during write), and we have no signal pointing at one.

Inconsistent with: in the 2026-04 era, the recovery procedure was already wipefs+reinstall, and the brick recurred on the same hardware after each rebuild. So if reinstall alone fixed it, the 2026-04-21 recovery should have lasted longer than ~1h. It didn’t.

3. CMOS battery (low)

Consistent with: CMOS battery swap happened concurrently with the SSD swap. Worth recording as part of the recovery event.

Inconsistent with: CMOS-battery failure typically causes BIOS-settings drift across cold boots (lost VT-d, lost boot order, clock drift) — not a running-system brick after 45-70 min. Once POSTed, a running system doesn’t depend on the CMOS battery. Possible-but-strained mechanism: the battery was weak enough to revert BIOS settings between cold boots, and one of those reverts (e.g. AHCI/RAID mode, CPU C-states) interacted badly with the workload. Speculative; no direct evidence.

4. Other dormant hardware issue (always possible)

PSU instability under sustained load, non-ECC RAM bit-flip, CPU/VRM thermal event, motherboard chipset issue. None directly tested. Currently quiet; would re-emerge if hardware-level. The 2026-05-06+ stability period is short (days, not weeks) — not yet enough observation to definitively rule these out.

What to watch for

If the brick pattern returns on the new SSD, that:

• Falsifies suspect 1 (the new SSD is a different unit; if it bricks too, the fault wasn’t drive-specific).
• Strengthens suspect 4 (deeper hardware issue — PSU, RAM, board).
• Probably warrants either retiring the chassis or stripping it down to the bare metal and rebuilding (PSU swap, fresh RAM, etc.) to find the actual fault. At that point the cost-benefit of investigating further on consumer hardware probably loses to “buy a small refurbished SFF for whatever role this serves” — same logic as docs/plan-incus-operations-center.md.

If the brick pattern stays absent for ~30+ days under realistic workload, the simplest hypothesis is suspect 1 — the prior 480 GB SSD was failing — and the matter can be considered resolved with an asterisk (root cause never confirmed, but symptom permanently absent).

Operational status (as of 2026-05-08)

• Hardware: same chassis (MAC a8:a1:59:0a:fd:09 unchanged), new boot SSD (Samsung 870 QVO 1 TB single-disk, no mirror), new CMOS battery, surviving WDC WD10EZEX 1 TB data drive repurposed as the ibuypwr_instances ZFS pool.
• Network: DHCP on client subnet 3 (10.112.113.90 current lease). Cannot host k0s VMs (no MAC-bound DHCP reservations on this subnet).
• Role: standalone GPU host. Single VM logi runs with NVIDIA RTX 2080 SUPER passthrough + Ollama LLM serving.
• No clustering — the 2026-04 incidents predate the 2026-05 de-clustering decision (see incus-declustering). ibuypwr is now an independent IncusOS daemon, so any future brick affects only logi, not the rest of the fleet.

GPU passthrough config — where it lives

(Preserved from the original investigation log because it’s still useful operational reference. Not part of the brick analysis.)

Two layers, important to understand:

1. Host-level (IncusOS kernel config) — persists across VM destroy/rebuild: Set via incus admin os system kernel edit on ibuypwr, supplying:

   config:
       blacklist_modules:
         - nouveau
       pci:
         passthrough:
           - vendor_id: "10de"
             product_id: "1e81"
             pci_address: "0000:01:00.0"

   This unbinds the GPU from nouveau and binds it to vfio-pci. Automated in ansible/playbooks/incus/setup-ibuypwr.yaml (which also creates the ibuypwr_instances pool). Documented as a manual prerequisite in opentofu/gpu.tf’s header and in docs/incus-pci-passthrough.md. Cleared only by editing kernel config to empty pci.passthrough, or full IncusOS reinstall (wipefs resets OS-layer state).

2. VM-level (OpenTofu) — recreated on tofu apply: In opentofu/gpu.tf — logi’s device "gpu0" block attaches the VFIO-bound device at VM create time. Goes away on tofu destroy.

Implication: host-level VFIO binding survives VM destroy/rebuild. To inspect current state:

INCUS_CONF=/configs/incus incus admin os system kernel show ibuypwr:

Recovery procedure — historical, not current

The 2026-04 recovery procedure was Incus-cluster-aware. Don’t use it. The cluster has been dissolved; ibuypwr is standalone. If a future brick recurs:

1. Confirm the brick pattern matches (loss of API + SSH from kyojin’s client; reboot hangs at “loading incus application”).
2. Try incus admin os system reboot ibuypwr: from kyojin first. If that works, you have access to a reboot path that doesn’t require physical intervention; capture state via the IncusOS admin API before doing anything destructive.
3. If unrecoverable: physical access, wipefs -a boot SSD, reflash via install media (/configs/incus/seeds/install_ibuypwr.yaml has the install target — operator-side, outside repo). On post-install: ansible-playbook playbooks/incus/setup-ibuypwr.yaml (creates ibuypwr_instances, configures GPU passthrough), then tofu apply -target=incus_instance.logi (recreates the GPU VM).
4. Capture forensics this time if possible — SMART data on the new SSD via smartctl, kernel ring buffer via journalctl, IncusOS state from incus admin os system show ibuypwr:. The 2026-04 investigations were hampered by no pre-brick forensic capture.

Lessons learned

• Suspect storage early on consumer hardware. A four-incident pattern of “consistent uptime-bounded brick + unrecoverable boot hang + only wipefs recovers” should converge on storage as a leading suspect. We chased GPU/IOMMU/OOM first because they were cheaper to test in software; the storage hypothesis required physical intervention. Reorder priorities in future investigations.
• wipefs-recovery cycles are diagnostic noise. If recovery requires destroying on-disk state, you’ve lost most of the evidence with it. Prioritize forensic capture (SMART data, dmesg, IPMI SEL if available) before wipefs.
• Three changes at one recovery event is one too many. SSD swap + CMOS battery + reinstall together means we can’t attribute the fix. If the brick had recurred we’d have wanted to revert one change at a time; that’s now impossible. Prefer single-variable interventions when stakes allow.
• Hardware investigations on consumer chassis hit a cost ceiling fast. When IPMI/BMC isn’t available and the only forensic channel is an attached monitor, the marginal cost of each diagnostic round is high. After two unsuccessful rounds, “buy/use different hardware” becomes the right answer — and that’s effectively what happened here (host moved out of the k0s critical path).

Related docs

• incus-declustering — why ibuypwr is now standalone.
• incus-pci-passthrough — patterns for GPU passthrough (still applies to logi).
• incus-os-factory-reset.md (operator note, not published) — clean-slate reset procedure (and its limitations on mirrored local — N/A for ibuypwr’s single-disk setup).
• plan-incus-operations-center.md (operator note) — the “buy a dedicated SFF” pattern for hardware that isn’t worth fighting on.