IncusOS factory-reset

Scope

Key-custody specifics (where pre-reset key captures live, example key values) are deliberately operator-side; this page keeps the operational mechanics in full. Findings reflect the 2026-05 test cycles; the upstream gaps cited (#745, #814, the re-registration block) were still open as of 2026-09.

incus admin os system factory-reset returns an IncusOS host to its post-install starting state without physical access. It’s the right tool for de-clustering, restarting a botched bring-up, recovering from corruption, or decommissioning a host before transfer.

Treat factory-reset as effectively destructive across the board. Testing on kyojin (2026-05) established what actually happens vs. what the upstream wording suggests:

• The boot drive’s LUKS volumes (root, swap) are re-formatted with fresh keys. The old keys in any pre-reset backup will not unlock them — restore fails on systemd-cryptenroll with “Operation not permitted.” There is no path back to the old LUKS keys.
• The local pool’s IncusOS-side registry is wiped, and on a clean post-install layout (single boot drive, no mirror added) IncusOS auto-recreates local with a fresh ZFS encryption key on next boot. On a layout that has drifted from install defaults (e.g. mirrored to zfs-raid1), local does not auto-recreate — you’d see pools: null and would have to reflash.
• User-created pools on dedicated drives are genuinely untouched on disk, and import-pool with the saved key brings them back at the IncusOS layer. However, the Incus daemon currently refuses to register the imported pool because the on-disk datasets trigger its empty-pool safety check, and there’s no documented force/adopt flag. So even a textbook capture-then-import sequence leaves user pool data visible to IncusOS but unusable to Incus. See Re-registering imported pools with the Incus daemon is broken.

So the practical decomposition:

Thing	What factory-reset does	Recovery story
Boot-drive LUKS keys	Rotated	None. restore cannot bring back the old ones.
local pool	Wiped from registry; auto-recreates fresh on clean disk layout, otherwise gone	Reflash to provision fresh local; user data in old local is lost
User-created pools (e.g. kyojin_instances)	IncusOS-side registry wiped; on-disk data + labels untouched	import-pool brings the pool back at IncusOS layer, but Incus-daemon registration is currently broken — pool data effectively unusable post-reset
Soft state in state.txt (network, DNS, applications, recovery passphrase)	Wiped	restore should replay this; in our testing restore aborted before reaching this step
TLS server cert	Regenerated (localhost-only SAN unless static-IP network seed)	Re-trust on operator side
incusbr0	Regenerated with random subnet	Re-configure post-reset (e.g. via OpenTofu)

In short: factory-reset is fine for genuine clean-slate scenarios (decommissioning, restarting a botched bring-up, recovering an unreachable host). Use it when you don’t need to preserve any pool data. For anything else, plan to reflash and re-bootstrap rather than relying on backup/restore/import-pool to round-trip cleanly — the upstream story has multiple gaps that prevent a full recovery as of 2026-05.

How to reset

ansible-playbook playbooks/incus/ops/factory-reset.yaml -e host=<host> -e confirm=yes

Both host=<remote> and confirm=yes are required. The playbook:

1. Renders the four seed payloads (applications, incus, network, update) from Jinja2 templates under ansible/playbooks/incus/templates/ against the target host’s inventory vars + group_vars/incus_servers.yaml. The trusted client cert in the incus payload is supplied at runtime as -e input_incus_trusted_cert=... (the playbook asserts it’s set); deliberately not committed.
2. Verifies local is single-device — aborts if mirrored (see below).
3. Deletes every user-created pool (anything not named local) and wipes their drives. Destructive.
4. Posts factory-reset to /os/1.0/system/:factory-reset with the bundled seed payload.
5. Waits for the daemon to come back (~2 min), re-trusts the regenerated cert, verifies daemon responding.

After the playbook completes, re-run setup-{host}.yaml to recreate user pools and (if desired) re-mirror local.

Important constraint: the playbook aborts on hosts with a mirrored local. There is no IncusOS API path to unmirror a pool. We’ve exhausted the options:

• storage edit with <2 devices: refused (Error: only 1 devices provided in update, expected at least 2).
• wipe-drive on the mirror partner: refused (Error: cannot wipe drive belonging to pool 'local').
• Sub-dataset source paths: same empty-pool check (covered in the gotcha below).
• No :detach, :remove-device, or equivalent endpoint exists in the IncusOS storage API surface.
• No shell access on IncusOS to run zpool detach directly. Debug API is read-only (api_debug.go). No incus admin os system run/exec.

For mirrored hosts, the working paths are:

1. Drive-pull workaround — physically remove the mirror partner drive, then run the playbook. Validated end-to-end on kyojin (2026-05), see Drive-pull reset procedure below. Requires brief hands-on access, but completes in ~2.5 min and avoids reflash.
2. Reflash via install media — simpler if you don’t have physical access; ~10 min with BMC.

For kyojin and theia (both currently have mirrored local + dedicated user pool), the drive-pull procedure is the fastest in-place reset.

Drive-pull reset procedure (for mirrored hosts)

Validated 2026-05 on kyojin. Total wall time ~5 min (1 min hands-on, ~3 min playbook + boot, ~1 min setup-kyojin afterward).

1. Physically remove the mirror partner drive from local. This is the second drive of the boot-drive mirror — NOT the boot drive itself, NOT any user-pool drive. Identify by member_pool: local + boot: false in incus admin os system storage show <host>:. The IncusOS daemon stays up — it just sees local as DEGRADED.

2. Verify state: incus admin os system storage show <host>: should show local with state: DEGRADED, devices containing only the boot drive’s part11, devices_degraded listing the missing partner. The script’s mirror check looks at devices | length only — it’ll see 1 and proceed.

3. Run the reset:

   ansible-playbook playbooks/incus/ops/factory-reset.yaml -e host=<host> -e confirm=yes

   Deletes user pools, wipes their drives, posts factory-reset, waits ~2 min for daemon, re-trusts cert. Result: local auto-recreates as zfs-raid0 single-device on the (still-attached) boot drive partition; daemon comes up clean; no leftover state.

4. Re-insert the partner drive. Hot-plug or scheduled-downtime depending on chassis. IncusOS sees it as a separate drive (with possibly stale partition state from prior cycles).

5. Run setup-{host}.yaml — re-mirrors local (now sees raid0 + a second drive), recreates user pools, restores PCI passthrough. The playbook wipes the partner’s partition table before attempting the mirror (added 2026-05 after we hit sgdisk: Could not create partition on a re-inserted drive). Resulting host state matches pre-reset.

Caveats:

• local’s prior data is gone (boot drive’s part11 was reformatted, local was recreated fresh). Same as any factory-reset.
• User pool data is gone (we deleted those before reset). Same as any clean reset.
• incusbr0 gets a fresh random subnet (every factory-reset does this). If you depend on a specific CIDR, reconfigure post-reset.
• TLS server cert is regenerated; --accept-certificate once on the operator side (the playbook does this automatically).
• The re-inserted partner drive will have a stale partition table from its prior life. setup-{host}.yaml now wipes it before attempting the mirror attach — without that step, incus admin os system storage edit fails with sgdisk -n 11:...: Could not create partition. If you’re running the mirror step manually outside the playbook, do incus query <remote>:/os/1.0/system/storage/:wipe-drive -X POST -d '{"id":"<partner-drive-id>","secure_wipe":false}' --wait first.

The drive-pull is what makes this work — it converts the API-side “can’t shrink mirror” problem into an at-the-hardware-level fact that IncusOS just accepts. Once devices | length == 1, the rest of the script is unblocked.

What it actually does

Per the Backup/Restore docs, system_reset.go, and our 2026-05 testing:

Boot drive: re-formatted, re-keyed.

• LUKS volumes (root, swap) get fresh encryption keys. Old keys (e.g. in a pre-reset backup) won’t unlock them.
• The IncusOS install itself stays bootable — system partitions are recreated correctly with the new keys.
• The reserved local-pool partition (...-part11) is wiped and a fresh local ZFS pool is created on it during first boot, but only if the disk layout still matches install defaults (single-drive zfs-raid0). If you’ve added a mirror partner via setup-{host}.yaml’s mirror task, the layout has drifted and IncusOS won’t auto-recreate local — you’ll get pools: null and need to reflash.

IncusOS-side state: wiped.

• The pool registry (so local and any user pools disappear from incus storage list).
• Default profile (recreated empty by Incus on next boot — root-device pointing at the auto-recreated local if applicable).
• incusbr0 regenerated with a fresh random subnet.
• All server-side instances and images.
• The TLS server certificate (regenerated for <hostname>.<domain> + loopback only — not the LAN IP).
• TPM state (with optional allow_tpm_reset_failure flag for recovery).
• Existing seed data (if wipe_existing_seeds: true).

User-created pools on dedicated drives: untouched.

• The on-disk ZFS labels and data survive. The pool encryption keys do not (they were on the system drive).
• To bring a user pool back, use import-pool with the pre-captured encryption key — see Re-importing user-created pools.

Genuinely preserved without action:

• The IncusOS install itself.
• The trusted-client-certificate list baked into the IncusOS image at build time (so the operator’s existing incus client is still trusted on first connection post-reset).
• Per issue #745, the system’s degraded-security state (e.g. swtpm vs physical TPM) — relevant if you’re resetting to address security state changes.

Optional, controlled by the seeds field of the request:

• Apply a new seed payload that takes effect on next boot. This sets the LAN address (rendered from incus.yaml.j2), configures DNS/hostname/timezone (rendered from network.yaml.j2), enables apps (rendered from applications.yaml.j2), etc. The seeds get the daemon back online and reachable; they don’t affect pool state.

Pre-reset checklist

1. Capture every user-created pool’s encryption recovery key. Load-bearing for any pool whose data you want to keep (e.g. kyojin_instances, theia_instances). Run incus admin os system security show <remote>: and save the entire output to the operator-side key-capture location (it is a credential — custody details in the backup/key-custody operator note). The pool_recovery_keys map is what import-pool will need post-reset. Capturing local’s key is mostly cosmetic — local is on the boot drive, which gets re-formatted, so the saved key won’t unlock anything afterward (the old data is gone).
2. Take a system backup. incus admin os system backup <remote>: backup.tar.gz captures state.txt (network config, DNS, applications list, recovery passphrase, scrub schedule) plus all pool encryption keys plus per-volume LUKS keys for the boot drive. Of these, what restore can actually replay post-reset: state.txt soft state and user-pool keys. It cannot replay the LUKS keys — those collide with the post-reset re-keyed volumes and restore fails with systemd-cryptenroll: Operation not permitted. So backup is useful primarily as: (a) a snapshot to feed into a reflash’s seed payload, (b) the operational form of the user-pool keys for import-pool. See the backup/key-custody operator note; the instance-level DR half is published at incus-instance-export.
3. Confirm your client cert is in the IncusOS image’s trusted list. It should be — that cert is built into the image at IncusOS build time and survives factory-reset — but if you’ve been using a non-baked cert for trust, capture and re-add it post-reset.
4. Decide on the seed payload. Either prepare per-host seed files (see Per-host seed file structure) or pass the minimum seeds.incus.apply_defaults: true to get the daemon listening. The seeds get the daemon back online; pool re-import is a separate post-reset step.
5. Plan the unreachable window. ~2 minutes of daemon downtime while the host reboots into the reset state. Add a few more minutes per user pool you’re re-importing afterward, plus reflash time (~10 min with BMC) if you need to recover local and the layout had drifted from install defaults.

The proper procedure (with seeds)

The playbook bundles the right seeds for the named host and posts them to the API. The resulting payload looks like:

incus admin os system factory-reset <remote>: -d '{
  "allow_tpm_reset_failure": true,
  "wipe_existing_seeds": true,
  "seeds": {
    "applications": <rendered applications.yaml.j2 as JSON>,
    "incus":        <rendered incus.yaml.j2 as JSON>,
    "network":      <rendered network.yaml.j2 as JSON>,
    "update":       <rendered update.yaml.j2 as JSON>
  }
}'

Do not include an install seed in this payload. See Don’t bundle the install seed into factory-reset below for the failure mode and the upstream-source citation. The script enforces this — install_<host>.yaml lives outside the repo at /configs/incus/seeds/ for the install-media tarball, not for factory-reset.

After the reset:

• The Incus daemon comes up bound to :8443 (from incus.yaml.j2’s rendered core.https_address), the trusted client cert is restored, applications listed in applications.yaml.j2 are running.
• Hostname, DNS, timezone all set per the network seed.
• local pool state depends on prior disk layout:
  • If the boot drive layout matches install defaults (single drive, never mirrored): local auto-recreates as zfs-raid0 with a fresh ZFS encryption key. incus storage list shows it; default profile’s root device works.
  • If you’d previously mirrored local (added a second drive via setup-{host}.yaml): no auto-recreate. pools: null. You’ll need to reflash to provision a fresh local.
• User-created pools (e.g. kyojin_instances) do not auto-import. Their data and labels are intact on the dedicated drives; their IncusOS-side registry is gone. You re-import them with the saved key — see Re-importing user-created pools.
• Cert is regenerated with the LAN IP in the SAN list if the network seed pre-registers the LAN address with IncusOS before cert generation — which it does for hosts using static-IP network seeds; DHCP’d hosts (kyojin) still end up with localhost-only SANs. This only matters during the initial incus remote add call (Go’s TLS stack enforces SAN check on the unpinned first connection). Once the cert is pinned in /configs/incus/servercerts/<remote>.crt, subsequent connections validate by fingerprint only — SAN content is irrelevant, so you can use IP-literal remote URLs freely.
• incusbr0 is regenerated with a fresh random subnet (not whatever it was before) — the seed doesn’t pin it. Anything that hardcodes the bridge CIDR needs updating, or you set the bridge config via OpenTofu after the reset.

Re-importing user-created pools

For each user-created pool that existed pre-reset (e.g. kyojin_instances, theia_instances), call import-pool with the recovery key you captured in step 1 of the pre-reset checklist. Don’t try this for local — local’s on-disk pool was wiped and recreated by the reset (or wasn’t recreated at all if your layout drifted), so the saved key won’t unlock anything.

The SystemStoragePoolKey request struct (per system_storage.go) takes:

• name — pool name (kyojin_instances, theia_instances, etc.)
• type — the storage driver name. Use "zfs", not the layout name (zfs-raid1/zfs-raid10). The error message “unsupported pool type ‘zfs-raid1’” misleadingly suggests layout names; only the driver is accepted.
• encryption_key — the recovery key from pool_recovery_keys

Posted to the IncusOS API:

incus query "<remote>:/os/1.0/system/storage/:import-pool" -X POST \
  -d '{
    "name": "kyojin_instances",
    "type": "zfs",
    "encryption_key": "<the recovery key from your pre-reset capture>"
  }' --wait

After import succeeds, the pool shows up in incus admin os system storage show <remote>: (IncusOS view) with all its prior volumes and instances intact, ONLINE, encryption_key_status: available. The actual underlying RAID layout is preserved on disk; IncusOS infers it from the existing labels.

The pool will not appear in incus storage list <remote>: yet. Re-registering with the Incus daemon is a separate, currently-broken step — see Re-registering imported pools with the Incus daemon is broken below.

The improper procedure (no seeds)

echo yes | incus admin os system factory-reset <remote>:

The host comes up but is barely useful:

• Daemon binds only to the local Unix socket — no core.https_address means no LAN listener, so incus info <remote>: from any operator machine returns “connection refused”.
• No trusted client certs, so even if the daemon were reachable you couldn’t authenticate.
• No applications, no network seed (DHCP fallback only), no timezone — the IncusOS defaults take over.

Net: only reach a host this way if you’re sitting at its console with --force-local. For anything else, supply seeds.

Recovery paths after a reset

The right recovery depends on what you captured pre-reset and which thing you’re trying to recover.

Situation	Path
You used the playbook (with drive-pull or single-device local)	No recovery needed for local — auto-recreated fresh on the install-defaults layout the playbook left behind. Re-run setup-{host}.yaml to recreate user pools and re-mirror local if you want those back.
local is auto-recreated on next boot	Nothing to do for local. The disk layout matched install defaults; IncusOS provisioned a fresh local with a new key. Your old data is gone, but the pool exists.
local is pools: null (layout had drifted; reset run without removing mirror partner first)	Reflash via install media to provision fresh local. The previous local’s data is gone either way (boot drive was re-formatted). Requires physical/BMC access.
You have a user-pool recovery key in your pre-reset capture	import-pool per pool — see Re-importing user-created pools. Brings the pool back at IncusOS layer; re-registering with Incus daemon is broken, so practical usefulness is limited.
You have a backup tarball, want to replay soft state	incus admin os system restore <remote>: backup.tar.gz --skip encryption-recovery-keys,local-data-encryption-key. In our 2026-05 testing, restore aborted in every scenario we tried regardless of --skip set. Documented for completeness; don’t depend on it. See restore rejects backups it should accept.
Daemon never came up post-reset (preseed loop, bad seed, malformed config)	Console into the host, fix the seed template (or reflash). The local Unix socket usually still works for incus admin --force-local even when the network listener doesn’t. We hit this when incus.yaml.j2 had core.ui.enabled: "true" (not a valid Incus key — see Validate every key in incus.yaml.j2’s preseed).
You bundled the install seed and the host won’t boot	Reflash. The install code aborts before the daemon comes up, so there’s no API to talk to. See Don’t bundle the install seed into factory-reset.

Per-host seed file structure

Seeds split between two locations based on consumer:

In-repo (factory-reset playbook renders these from templates at runtime):

ansible/playbooks/incus/templates/
├── applications.yaml.j2     # IncusOS apps to install — driven by group_vars/incus_servers.yaml:incus_apps
├── incus.yaml.j2            # Incus daemon preseed — incus_https_address + cert from runtime extra-var input_incus_trusted_cert (factory-reset.yaml asserts it's set; deliberately not committed)
├── network.yaml.j2          # Per-host network — hostname=inventory_hostname, hwaddr=mac_addr; static-vs-DHCP from network_mode/prefix/gateway in inventory
└── update.yaml.j2           # Update settings — incus_update_auto_reboot, incus_update_check_frequency

The four templates render once per factory-reset.yaml -e host=<host> invocation against the target host’s inventory + group_vars context, then go straight into the seeds: dict of the factory-reset payload. No on-disk seed files; no per-host duplicated YAML.

Outside repo (operator-managed; install-media build + recovery-key capture):

/configs/incus/seeds/
├── install_<host>.yaml       # Per-host: install target (boot drive serial). Used at install-media build time.
├── (key captures)            # Per-host LUKS + ZFS pool recovery keys — credential custody per the operator note.
└── backups/                  # IncusOS system + per-instance export tarballs.

The set of files you bundle differs by use case:

Factory-reset payload (assembled by playbooks/incus/ops/factory-reset.yaml) — boot-time/runtime seeds only, all rendered from templates:

• applications.yaml.j2 (canonical name in seed: applications)
• incus.yaml.j2 (canonical name in seed: incus)
• network.yaml.j2 (canonical name in seed: network)
• update.yaml.j2 (canonical name in seed: update)

Install-media seed tarball (built when producing IncusOS install media) — adds the install seed:

• All of the above, plus
• /configs/incus/seeds/install_<host>.yaml (canonical name in seed: install — operator-side, outside repo)

install_<host>.yaml is only valid in the install-media tarball. Including it in a factory-reset payload causes IncusOS to refuse to boot — see the gotcha below.

The key capture is not a seed — it’s a snapshot of the host’s TPM state and pool recovery keys, useful pre-reset so you know which pools’ data is recoverable. It’s a credential, treat it like one (encrypt at rest; SOPS).

Gotchas

Don’t bundle the install seed into factory-reset

Bundling install_<host>.yaml into the seeds map of a factory-reset call is a guaranteed boot failure on the next cycle, not flaky behavior. We hit this on kyojin’s first reset.

The mechanism, from incus-osd/internal/install/install.go:

if installSeed != nil {
    contents, err := util.ReadEFIVariable("IncusOSInstallComplete")
    if len(contents) != 0 {
        return errors.New("install media detected, but the system is already installed")
    }
    ...
}

The presence of an install seed at boot tells IncusOS “you are install media, run the installer.” Already-installed systems set the IncusOSInstallComplete EFI variable, so the installer aborts with the error above and the host doesn’t come up. Per issue #814, there’s no first-class re-install path yet — until that lands, install seeds are install-media-only.

The factory-reset API itself (system_reset.go) has no whitelist on seed names — Seeds is map[string]json.RawMessage. You can put any seed name in the payload and the API will write it to the seed area. The constraint is enforced one boot later by the install code, not by the reset API. So a malformed reset request “succeeds” — the host then refuses to come up. This is what makes it look like an outage rather than a rejected request.

playbooks/incus/ops/factory-reset.yaml enforces the right set (applications, incus, network, update); the install seed is intentionally not loaded. If you’re constructing a reset payload by hand, don’t add it.

local post-reset is one of three states

Behavior depends on the boot drive’s pre-reset layout:

1. Layout matches install defaults (single boot drive, never mirrored): IncusOS auto-recreates local as zfs-raid0 on the reserved partition with a fresh ZFS encryption key. Confirmed in our 2026-05 testing — kyojin reset from clean post-install state came back with local present, key rotated to a fresh value (key prefixes redacted here). No action required to get local working again; the previous data is gone.
2. Layout drifted (mirror partner added via setup-{host}.yaml): local does not auto-recreate. incus admin os system storage show returns pools: null. The boot drive’s part11 has stale ZFS labels from the wiped pool, and IncusOS can’t auto-import (the key is gone). Reflash via install media is the only recovery; install-time disk prep wipes the partition cleanly.
3. You attempted manual storage edit to recreate local: fails with is part of exported pool 'local' because of the same stale-labels issue from case 2. The IncusOS storage API doesn’t expose -f for force-create. Don’t try this — it’s a dead end if local didn’t auto-recreate.

For user-created pools: factory-reset never auto-imports them, regardless of layout. They wait for a manual import-pool with the saved key. See Re-importing user-created pools.

Re-registering imported pools with the Incus daemon is broken

Once you’ve used import-pool to bring a user-created pool back at the IncusOS layer post-reset, the next step should be to register it with the Incus daemon so incus storage list, profiles, and instance creation see it. There is currently no working API for this.

What we tried (kyojin, 2026-05, across multiple test cycles):

• incus storage create kyojin:kyojin_instances zfs source=kyojin_instances — fails with Provided ZFS pool (or dataset) isn't empty, run "sudo zfs list -r kyojin_instances" to see existing entries. The on-disk datasets that import-pool brought back (containers/, custom/, images/, virtual-machines/, buckets/, deleted/) trigger Incus’ empty-pool safety check.
• Direct API call to POST /1.0/storage-pools with the same body — same server-side error. The check is in the daemon, not just the CLI.
• incus admin recover — only enumerates pools the Incus daemon already tracks. The just-imported pool isn’t in the daemon’s database, so recover can’t find it.

There’s no documented --force, --adopt, --accept-existing flag on storage create, and no recover mode that reaches into IncusOS-managed pools the daemon doesn’t know about.

The failure is independent of local’s state (verified across test B with mirrored local and test C with unmirrored local) — it’s the Incus ZFS driver’s empty-pool check on the user pool’s own datasets, not anything to do with local. Removing the local mirror does not help.

The failure is also independent of source path style. We tested registering with source=kyojin_instances/incus (a sub-dataset path, mirroring how IncusOS registers local in app_incus.go’s applyDefaults — source: local/incus). The theory: maybe Incus’ empty check would target the sub-dataset rather than the root zpool, and post-import that sub-dataset would be empty enough to adopt. Test result (kyojin 2026-05): the sub-dataset comes back populated by import-pool along with the rest of the zpool, and the empty check fails the same way: Provided ZFS pool (or dataset) isn't empty, run "sudo zfs list -r kyojin_instances/incus". Sub-dataset paths are not a workaround.

So in practice: after factory-reset, even with all keys captured pre-reset and a successful import-pool, your user pools’ data is visible to IncusOS but unusable to Incus. The pool exists, the volumes exist, instances within would still be on disk — none of it is reachable through incus CLI operations. This is a real gap in the upstream factory-reset → restore story.

Workarounds we know about, none clean:

• Reflash and start over — abandon the imported data; provision a fresh pool via setup-{host}.yaml.
• Delete the IncusOS pool, recreate fresh, restore data from somewhere else — defeats the point of the import.
• Manually mutate Incus’ state DB via incus admin sql — undocumented, risky, doesn’t survive upgrades.

This deserves an upstream issue: there’s no way to complete the documented workflow on a host with user-created pools that had real data. Until it’s resolved, treat user-created pools as effectively destroyed by factory-reset, even if you captured every key the docs ask for.

Practical viability matrix for factory-reset

Tested on kyojin across three reflash cycles (2026-05):

local mirrored?	User pool present?	Test	Result
No	No	A	✅ Works end-to-end. local auto-recreates fresh; nothing else to restore.
Yes	Yes	B	❌ Both halves fail. local doesn’t auto-recreate (layout drift), user pool can’t be re-registered with Incus daemon.
No	Yes	C	❌ User pool registration fails (Incus daemon empty-pool check). local is fine.
Yes	No	(inferred)	❌ Same local drift problem as B/C.

Plain factory-reset (no special handling) is only fully viable for hosts with no user-created pools and no local mirror. For hosts with user pools but single-device local, the playbook handles the user-pool deletion and lets factory-reset auto-recreate local cleanly. For hosts with a mirrored local, no API path exists to reshape it back to single-device — IncusOS storage edit rejects any pool definition with fewer than 2 devices and there’s no detach endpoint. The playbook detects this and aborts; the workaround is the Drive-pull reset procedure.

For kyojin/theia (mirrored local + dedicated user pool): the drive-pull procedure is the in-place reset path that round-trips cleanly. Reflash via install media is the alternative when physical access isn’t available.

restore rejects backups it should accept

The incus admin os system restore API rejects valid-looking backups in scenarios it should handle. Tested 2026-05 against a backup containing a populated state.txt and all expected key files:

• Without --skip: at least one recovery passphrase must be provided. The backup’s state.txt clearly contains a populated EncryptionRecoveryKeys[0]. Rejected anyway.
• With --skip encryption-recovery-keys: same error — makes sense (skip means don’t pull the backup’s keys; existing host has none post-reset; result is zero keys; validation fails).
• With --skip encryption-recovery-keys,local-data-encryption-key: open /var/lib/incus-os.bak/zpool.local.key: no such file or directory. The skip implementation copies the existing local key from a backup directory; if local doesn’t exist (because the layout drifted and reset didn’t auto-recreate it), there’s no key to copy.

Suspected cause for the first case: the captured state.txt includes System.Security.State.EncryptionRecoveryKeysRetrieved: true, which we hit because incus admin os system security show was called pre-backup. IncusOS may treat a “retrieved” key as consumed and not eligible for re-enrollment via restore. This is unconfirmed.

Practical impact: in our test scenario (kyojin, post-setup state with mirrored local and populated kyojin_instances), no --skip combination produced a working restore. We fell back to manual import-pool for kyojin_instances, which succeeded at the IncusOS layer (but then hit the re-registration gap above).

restore cannot undo a factory-reset’s boot-drive re-keying

The intuitive sequence “backup → reset → restore” doesn’t fully roll the system back. Factory-reset reformats the boot drive’s LUKS volumes (root, swap) with fresh keys; restore tries to enroll the backup’s old keys against the new LUKS volumes via systemd-cryptenroll, which fails:

Error: unable to process state from backup:
Failed to run: systemd-cryptenroll --unlock-key-file=/var/lib/incus-os/recovery.swap.key
  --password /dev/sda9: exit status 1 (Unlocking via keyfile failed: Operation not permitted)

The backup has the old recovery.swap.key, recovery.root.key, and zpool.local.key. None of them match what’s on the boot drive after the reset. Restore aborts before applying anything else from the tarball.

To restore the replayable portions of the backup (state.txt’s network config, applications list, recovery passphrase, plus user-pool encryption keys), pass --skip encryption-recovery-keys,local-data-encryption-key to skip the LUKS/local re-enroll attempts:

incus admin os system restore <remote>: backup.tar.gz \
  --skip encryption-recovery-keys,local-data-encryption-key

This is the documented contract of --skip flags: tell restore to leave the post-reset keys in place and only replay the rest. Verify the result with incus admin os system security show afterward — multi-skip parsing has been reported buggy in some IncusOS versions, and the restore could silently fail to skip what you asked.

So the practical recipe with backup is:

# Pre-reset
incus admin os system backup <remote>: backup.tar.gz

# Reset
ansible-playbook playbooks/incus/ops/factory-reset.yaml -e host=<host> -e confirm=yes

# Post-reset, replay only what's replayable
incus admin os system restore <remote>: backup.tar.gz \
  --skip encryption-recovery-keys,local-data-encryption-key

# Then re-import user-created pools (restore should do this, but verify)
# See: Re-importing user-created pools

Restore is an opt-in post-reset step because the right --skip set depends on what state you want to preserve.

delete-pool destroys on-disk data, not just unregisters

incus admin os system storage delete-pool (and the API endpoint :delete-pool) does not just remove the pool from IncusOS’s registry — it runs zpool destroy on the underlying ZFS pool, wiping all on-disk data. Confirmed empirically (kyojin, 2026-05): after a delete-pool call for kyojin_instances, a subsequent import-pool failed with cannot import 'kyojin_instances': no such pool available — the on-disk pool was gone.

Implications:

• This is the right behavior for the playbook’s “delete user pools before reset” step. Intentional data loss.
• This is dangerous if you confuse delete-pool for “unmount this pool from Incus” — it doesn’t unmount, it destroys. There’s no --keep-data or similar flag.
• The API call sometimes returns a stray error like drive '...' doesn't exist while the destroy succeeds anyway. Don’t trust the exit code alone — verify pool absence via storage show if it matters.

If you want to remove an Incus-side pool registration without destroying the underlying ZFS data, the only path is the regular incus storage delete <remote>:<pool> (Incus daemon side), which removes the pool from Incus’ DB but leaves the zpool intact. The IncusOS-side delete-pool has no such gentler equivalent.

Validate every key in incus.yaml.j2’s preseed

Modern Incus has no core.ui.* config namespace. The web UI is enabled simply by having UI files at /opt/incus/ui/, which IncusOS ships already. We had core.ui.enabled: "true" in the incus seed for months without noticing; on factory-reset, IncusOS applies the preseed and bombs on the unknown key, rolling back the entire core.* config block including core.https_address. The daemon then comes up bound to the local Unix socket only, and IncusOS retries the seed in a loop. Symptoms: TCP connection refused on :8443, console messages like:

cannot set core.ui.enabled to true, unknown key
storage pool local is missing encryption key

Recovery requires console access to fix the seed (and at that point you may as well reflash with the corrected one).

If you add anything to incus.yaml.j2’s preseed.config, validate it against incus config show on a known-good host first, or sanity-check it against the server config docs — silent acceptance during a working session doesn’t mean the key is real, only that it survived being rejected at apply time.

CLI confirmation prompts read from /dev/tty, not stdin

incus admin os system factory-reset and incus admin os system storage wipe-drive both prompt with “Are you sure? (yes/no)”. They read the answer from /dev/tty, not stdin, so echo yes | <cmd> doesn’t work in non-interactive contexts (CI, scripts, agents). It looks like the command is hung — the prompt is printed, and waits forever.

Workaround: hit the IncusOS API directly via incus query, which has no prompt. The IncusOS admin API is proxied through the Incus daemon at /os/1.0/.... So instead of:

incus admin os system factory-reset <remote>: -d "$payload"     # blocks

use:

incus query "<remote>:/os/1.0/system/:factory-reset" -X POST -d "$payload" --wait

playbooks/incus/ops/factory-reset.yaml uses this pattern throughout. Don’t fall back to the CLI form in scripts.

Cert SAN: hostname yes, LAN IP no — only matters for first remote add

After factory-reset the regenerated server cert’s SAN includes:

• DNS:<hostname>.<domain> (e.g. theia.rrchnm.internal)
• IP:127.0.0.1
• IP:::1

It does not include the LAN IP that the host actually listens on. Connecting via the LAN IP during the initial incus remote add --accept-certificate <ip> fails TLS hostname verification:

tls: failed to verify certificate: x509: certificate is valid for 127.0.0.1, ::1, not 10.112.12.104

--accept-certificate only handles fingerprint mismatch, not hostname mismatch — so this fails on the very first connection, before the cert can be pinned.

Remediation (pick one — only one needs to succeed; afterward all subsequent connections work via fingerprint pinning regardless of URL form):

• Easiest: incus remote add <name> https://<hostname>.<domain>:8443 --accept-certificate from a host that resolves <hostname>.<domain> (campus DNS, /etc/hosts, or wherever). The remote-add succeeds, the cert is pinned, and you can immediately switch the remote’s addr to the IP literal in /configs/incus/config.yml if you want — fingerprint-pinned trust persists. This is what the standard operator workflow does.
• DNS path: if your network has a DNS server that already resolves <hostname>.<domain>, the easiest variant works without any local config. We tried this; campus DNS did not resolve *.rrchnm.internal from this jump box, so we add a one-line /etc/hosts entry only as long as needed for the remote-add call (then remove it).
• Proper-IncusOS-side fix: re-seed IncusOS with an install seed that bakes the LAN IP into the host’s known network config. The cert is regenerated with the LAN IP in the SAN on next boot — but only for static-IP hosts (DHCP hosts still get localhost-only SANs).

After the cert is pinned in /configs/incus/servercerts/<remote>.crt, the SAN content stops mattering — Incus client validates by fingerprint match, not hostname match. So addr: https://10.112.113.202:8443 works fine for an existing remote, even though the cert SAN says kyojin.rrchnm.internal.

Pre-reset cluster member’s cert may name the leader’s hostname

A clustered IncusOS host’s cert SAN list may include the cluster leader’s hostname rather than its own — e.g. theia’s cert had DNS:kyojin.rrchnm.internal while theia was a member of kyojin’s cluster. After factory-reset the cert is regenerated correctly with the host’s own hostname. Benign while clustered, but confusing when reading certs.

Sources

• Backup/Restore — IncusOS documentation
• Storage — IncusOS documentation
• System recovery — IncusOS documentation
• Installation seed — IncusOS documentation
• Server configuration — Incus documentation (no core.ui.* keys exist)
• incus admin os system factory-reset — manpage
• system_reset.go (IncusOS source)
• system_storage.go (IncusOS source) — wipe-drive / import-pool API
• install.go (IncusOS source) — install-seed-on-installed-system check
• Issue #745 — Properly handle factory reset for degraded security states
• Issue #814 — Provide a way to re-install a system