De-clustering IncusOS

Historical record — May 2026

The de-clustering is complete and permanent (the fleet runs five standalone IncusOS daemons). Repo paths named below reflect May 2026: kube.tf and the per-purpose .tf files have since consolidated into fleet.yaml/fleet.tf, and bridge management evolved to tfbr0. One rationale below has aged badly: incus copy --refresh is upstream-blocked on IncusOS — see incus-warm-migration for what actually works. This is a dated record and is not maintained.

This document captures the procedure for taking an IncusOS host out of an Incus cluster, returning it to a clean standalone state. It records the gotchas encountered when we de-clustered theia and kyojin in 2026-05.

For the underlying factory-reset behavior (what it wipes, what it preserves, the seeds payload, and recovery if you forget to pass seeds), see incus-os-factory-reset.md (operator note, not published). For warm-migration between standalone hosts after de-clustering, see incus-warm-migration.

When to use this

You want to dissolve an Incus cluster on IncusOS and return its members to standalone operation. Common reasons:

• The cluster ceremony (cluster-wide pools with stub source overrides on each member) doesn’t pay for itself when you only have a couple of hosts and don’t need live migration.
• Warm migration (incus copy <src> <dst> --refresh) works between standalone remotes just as well as between cluster members and doesn’t require shared storage. Live migration is the only thing clustering would buy you, and on IncusOS its operational ergonomics may not be worth the trouble.
• An IncusOS-specific quirk leaves system custom volumes (backups, images, logs) as orphan ZFS datasets on cluster members at the disk level without a corresponding entry in Incus’s volume database — see “Gotcha: orphan datasets” below — which blocks clean cluster removal.

Procedure

Standby member (e.g. theia in a kyojin-leader cluster)

If the member you’re removing is not the cluster leader, do this from the leader:

# 1. Try clean removal first. If it fails with "ZFS pool has leftover datasets",
#    continue to step 2; if it succeeds, you are done.
incus cluster remove <leader>:<member>

# 2. Force-remove. The docs warn that this leaves the removed member's database
#    in an inconsistent state requiring "full reinstall" — for IncusOS,
#    "full reinstall" === factory-reset, which is the next step.
incus cluster remove --force <leader>:<member>

# 3. Factory-reset the member with a seeds payload (see the operator note). Wipes
#    its IncusOS state including the orphan datasets, and re-seeds local pool.
incus admin os system factory-reset <member>: -d '<seeds payload>'

After ~1–2 minutes the member’s daemon will be reachable again, but now standalone and with a regenerated cert. See incus-os-factory-reset.md (operator note) for the seeds payload and reconnection steps.

Cluster leader (e.g. kyojin as the only remaining member)

If the member you’re removing is the cluster leader (and especially if it’s the only member), there’s nothing to clean up cluster-side first — factory-resetting dissolves the cluster:

incus admin os system factory-reset <leader>: -d '<seeds payload>'

If there are still other members at this point, they will be orphaned. Force-remove or factory-reset them before doing the leader.

Gotchas

incus cluster remove blocks on orphan ZFS datasets

When IncusOS hosts join a cluster, the IncusOS layer auto-creates per-member datasets for system custom volumes (local/incus/custom/default_backups, …/default_images, …/default_logs). The cluster’s volume database, however, only registers these volumes on the originating member (whichever node was active when they were first created). The result: ZFS datasets exist on disk on member B with no Incus database record on either member.

incus cluster remove <member> runs a precondition check that walks the member’s pool for unknown datasets and refuses if any exist:

Error: ZFS pool has leftover datasets: custom/default_backups

The forum-suggested workaround (incus storage volume create to claim the orphan, then delete) doesn’t work here, because Incus’s create path issues zfs create local/incus/custom/default_backups which fails: the dataset already exists at the ZFS layer. There’s no API surface to “adopt” an existing dataset, and IncusOS doesn’t expose a shell to run zfs destroy directly.

Workaround: force-remove (which bypasses this check) followed by factory-reset (which wipes the orphan datasets along with everything else on the system drive).

--force removal docs are scarier than they need to be (for IncusOS)

The Incus docs say --force “leaves the member’s database in an inconsistent state… it will not be possible to re-initialize Incus later, and the server must be fully reinstalled.” For a generic Incus install this implies a destructive OS-level reinstall. For IncusOS, incus admin os system factory-reset is “full reinstall” — it resets the system drive to clean install state without physical access. So --force removal followed by factory-reset is a supported path, despite how the warning reads.

Older copies of setup-cluster.yaml overstate the brick risk

The repo previously had a setup-cluster.yaml header that warned “clustering is IRREVERSIBLE on IncusOS, NEVER disable clustering via the API”. That warning conflated two things:

• PUT /1.0/cluster {"enabled":false} — genuinely irreversible without reinstall (don’t do this).
• incus cluster remove --force <leader>:<member> — recoverable via factory-reset.

The playbook is now deleted (we de-clustered the repo too), but the conservative reading of that warning held us up earlier. The actual --force path is supported per the docs.

What was changed in the repo when we de-clustered

After de-clustering both hosts in 2026-05, these were the supporting changes:

• Makefile — dropped the setup-cluster.yaml line from make infra.
• ansible/playbooks/incus/setup-cluster.yaml and tasks/cluster-join-member.yaml — deleted; cluster bootstrap is no longer relevant.
• ansible/playbooks/incus/setup-{kyojin,theia}.yaml — dropped the --target <member> source-define tasks and the cluster-wide register task. In standalone Incus, a single incus storage create <remote>: <pool> zfs source=<source> creates the pool directly, no two-phase create-then-finalize.
• opentofu/kube.tf — removed the target = "..." field from each incus_instance. In standalone Incus there’s no concept of cluster member targeting; the remote = "..." field is what selects which daemon TF talks to.
• opentofu/system_images.tf — images are no longer auto-shared between hosts; each remote needs its own image cache. Now declares parallel incus_image resources per remote via for_each.
• opentofu/system_network.tf — added explicit incus_network "incusbr0" resources per host with stable CIDRs (default behavior is a random /24 per host on each cluster init).
• opentofu/system_network.tf — added incus_profile "bridged" per host to provide eth0+root devices, so the test instances can opt in via profiles = ["bridged"] rather than relying on the auto-managed default profile.