Incus warm migration

This document covers moving an Incus instance from one host to another. Two patterns work today on IncusOS:

• Cold migration — stop source, full copy, start destination. Simple; downtime equals the full transfer time. Documented below.
• Pre-stage + delta cutover — copy disk while source running (via --stateless), let dest sit stopped, sync final delta at cutover through the file/application layer. Near-zero cutover downtime. Documented below.

The third pattern — incus copy --refresh for storage-driver-native incremental warm migration — is blocked on IncusOS by mandatory ZFS encryption. It’s preserved later in the doc as an aspirational reference for non-IncusOS hosts with unencrypted ZFS pools.

The repo’s test3 VM (defined in opentofu/scratch.tf, lives on theia by default) exists to exercise these workflows.

Status (2026-05-17): --refresh incremental migration is upstream-blocked on IncusOS — see ZFS encryption breaks --refresh (linked to lxc/incus-os#721). Cold migration + pre-stage warm migration both verified working on alpine container tests theia↔kyojin.

Prerequisites

Client remotes must use IP literals, not hostnames

In /configs/incus/config.yml, the addr: for each IncusOS remote must be an IP-literal URL (e.g. https://10.112.113.202:8443), not a *.rrchnm.internal hostname. Reason: incus copy’s push-mode hands the destination’s URL from the client to the source daemon, which then opens its own TLS connection. IncusOS hosts resolve only via GMU campus DNS (magda/ruth/eve), which doesn’t serve the internal rrchnm.internal zone — so any hostname-based URL gets NXDOMAIN on the source-daemon side and the copy fails (or, before this discovery, silently fell back to import/export). The Incus client pins server certs by fingerprint (/configs/incus/servercerts/<remote>.crt), so URL-vs-SAN mismatches don’t matter.

This matches the guidance already in AGENTS.md’s “Prerequisites” mount table; the fix here is just to enforce it for the daemon-to-daemon dial too. The five IncusOS remotes (IPs per fleet.yaml incus_hosts — the source of truth if these drift):

Remote	addr:
every IncusOS host	https://<host LAN IP>:8443 — current addresses in fleet.yaml (incus_hosts)

Pointing the IncusOS hosts at the internal CoreDNS resolvers (azelf/mesprit/uxie) would also solve the resolution side, but creates a circular bootstrap dependency (mesprit runs on kyojin; azelf on theia; uxie on atomsk — a host can’t resolve anything until its own CoreDNS VM has booted). IP literals avoid the dependency entirely.

When to use warm migration

Use it when you want to move an instance between Incus hosts without:

• Setting up shared/distributed storage (Ceph, NFS, iSCSI)
• Joining the hosts into an Incus cluster
• Tolerating zero downtime — warm migration has a small downtime window during cutover

It’s cross-cluster and cross-architecture compatible: the destination just needs to be a reachable Incus daemon with a compatible storage pool driver. Different CPU vendors are fine, different memory sizes are fine, different network configs are fine — because the VM is restarted on the destination, runtime state (memory, open sockets, ARP tables) doesn’t have to survive transit.

Why not live migration?

Live migration (QEMU live state transfer + storage live migration) requires:

1. Both hosts in the same Incus cluster. Standalone remotes can’t live-migrate to each other.
2. The destination on the same L2 broadcast domain as the source. Live migration’s value is preserving the VM’s IP/MAC mid-flight so existing TCP connections survive. Migrating across subnets defeats this.
3. Compatible CPU and QEMU versions between source and destination. CPU feature flags and QEMU memory-state ABI must match.
4. Shared storage or live-storage-migration capability, supported by the storage driver on both ends.

In our homelab — kyojin and theia are de-clustered (separate Incus universes) and on different L2 subnets — live migration is structurally not available, so warm migration is the only option. Even if we re-clustered, the cross-subnet topology would still defeat live migration’s main benefit. For zero-downtime workloads, that would mean reorganizing both clustering and L2 layout; warm migration sidesteps the question.

The mechanism

incus copy <src-remote>:<instance> <dst-remote>: --refresh:

1. Snapshots the source volume (ZFS snapshot under the hood for ZFS pools).
2. Ships the snapshot to the destination’s pool. For ZFS, this uses zfs send over the migration protocol, with zfs send -i (incremental) on subsequent refreshes — only the delta since the last refresh transfers.
3. Idempotent — running it once does an initial copy; running it again with the same args ships only what’s changed since the last run.

The source instance stays running throughout. Each refresh on a running source captures point-in-time state (whatever happened to be on disk at the moment of the snapshot), so any in-flight writes after that snapshot are picked up on the next refresh.

For the final cutover, the source is stopped first so the last refresh captures everything, then started on the destination.

Cold migration on IncusOS

The IncusOS-compatible equivalent of XCP-ng’s cold migration — downtime equal to the bulk transfer time, no incremental sync. Plain incus copy (no --refresh) on a stopped instance only does zfs receive to create a fresh dataset; the -F rollback flag that trips the encryption restriction is never invoked.

# 1. Stop the source.
incus stop theia:foo

# 2. Full copy. No --refresh, so destination is created fresh.
#    Pick a destination pool; for instance-class workloads use the dedicated
#    *_instances pool to avoid filling the boot drive.
incus copy theia:foo kyojin:foo --storage kyojin_instances

# 3. Start on the destination.
incus start kyojin:foo

# 4. Validate, then delete the source.
incus delete theia:foo

Downtime is full-transfer wall clock + instance start time. For a 16 GiB container that’s seconds-to-minutes; for a 200 GiB VM over the campus LAN, expect tens of minutes. There’s no incremental option that survives the encrypted-pool restriction (see caveats).

Verified working 2026-05-17 with an alpine container theia → kyojin: copy succeeded in ~1s, started successfully on kyojin with the right tfbr0 IP (the bridged-profile resolves per-remote, so the migrated copy automatically picks up the destination host’s tfbr0 without reconfiguration). Profile-resolution behavior is the same as in the aspirational walkthrough below.

What about running containers without --refresh?

Plain incus copy src:foo dst:foo against a running container triggers Incus’s live state transfer path, which invokes CRIU. On IncusOS that fails with:

Error (criu/util.c:642): execvp("iptables-restore", ...) failed: No such file or directory

IncusOS is nftables-only — iptables-restore isn’t shipped, CRIU’s network-locking helper can’t run, the live dump errors out. So always incus stop first before a cold copy.

Pre-stage + delta cutover (warm migration on IncusOS)

This is the IncusOS-compatible equivalent of XCP-ng’s warm migration. The source keeps running for the entire bulk-transfer phase; downtime is just the final-delta sync at cutover — typically seconds to a minute, regardless of disk size.

The key insight: incus copy --stateless skips the live state-transfer path (CRIU for containers, QEMU live-state for VMs), so it can run against a running source without hitting the IncusOS CRIU error. The destination gets a point-in-time disk snapshot but stays stopped — ready to start, but stale until the cutover delta fills it in. The cutover delta runs at the file/application layer, not the storage layer, so it sidesteps the zfs receive -F block entirely.

The sequence

# === Phase 1: pre-stage (source keeps running) ===
# Copies the disk only (no live state). Source unaffected.
# Use --storage to pin destination pool; *_instances for VM-class workloads.
incus copy theia:foo kyojin:foo --stateless --storage kyojin_instances

# Source has been writing the whole time; dest holds a snapshot from time T0.
# Dest exists in STOPPED state — do NOT start it yet.

# === Phase 2: divergence window (minutes to days, your call) ===
# Source serves traffic normally. Dest stays stopped.
# Optionally, do periodic warm-up syncs of high-change directories so
# the final cutover delta is small:
incus file pull theia:foo/var/lib/important /tmp/relay
incus file push /tmp/relay kyojin:foo/var/lib/important
# (or for production: in-guest rsync between source and dest IPs over ssh —
#  see "Reachability for in-guest rsync" below.)

# === Phase 3: cutover ===
# 1. Drain services in source (so no new writes happen).
incus exec theia:foo -- systemctl stop important-service

# 2. Final delta sync of the data dirs that matter.
incus file pull theia:foo/var/lib/important /tmp/relay
incus stop theia:foo
incus file push /tmp/relay kyojin:foo/var/lib/important

# 3. Start dest.
incus start kyojin:foo

# 4. Verify, swap traffic (DNS, caddy upstream, etc.), retire source.
incus delete theia:foo

Downtime = the time between step 3.1 (stop services on source) and step 3.3 (start dest). For typical change rates that’s seconds-to-a-minute — vs the tens of minutes a cold migration would take for a multi-GB VM.

Verified working 2026-05-17 on an alpine container theia → kyojin: pre-stage copy took 1.2s, divergence window ran for ~6s during which 3 marker updates were written on source, cutover-delta + start completed in seconds, dest finished with all 4 marker entries (T0 baseline + 3 T1 updates from the divergence window).

What’s transferred when

Phase	Bytes moved	Source state
Pre-stage	Full disk (one-time, slow)	Running, unaffected
Divergence	Zero (or periodic warm-ups)	Running, unaffected
Cutover delta	Only changed paths since pre-stage	Briefly drained

This is why pre-stage saves cutover downtime: the bulk transfer happens while the source is up and serving.

Reachability for in-guest rsync

The incus file pull/push relay through the workspace works for the test (and for small/infrequent deltas), but it’s slow for large directories — every byte passes through the workspace’s TCP fans. For production cutover, prefer direct guest-to-guest rsync over ssh:

• If source and dest can both reach a shared LAN (e.g. both VMs on macvlan-profile attached to their host’s eno1), they’re routable across campus subnets and can ssh directly.
• If source and dest are on per-host NAT’d bridges (tfbr0, incusbr0), they’re isolated. Either re-attach them to macvlan for the migration window, or use the workspace relay.

Setup pre-cutover: drop the ansible SSH public key into the dest guest, install rsync + openssh in both, verify connectivity, then drive rsync from source via incus exec theia:foo -- rsync -avx --delete /var/lib/important/ root@<dest-ip>:/var/lib/important/.

Note on incus file pull/push vs rsync. The sequence above uses incus file pull/push to keep things self-contained and demo-friendly, but those commands transfer the entire path every invocation — no delta detection, every byte goes through the workspace twice (source → workspace → dest). Fine for the cutover (one-shot, services drained) and for tiny files. For periodic warm-up syncs during the divergence window, use real rsync between the guests — its delta detection is what makes the warm-up cheap, and direct guest-to-guest avoids the workspace relay hop.

What to sync vs. what to leave alone

The pre-stage gives the destination a complete OS-layer copy. The cutover delta only needs to cover workload state — the directories where the application writes data that must survive the migration. A bunch of stuff actively shouldn’t be synced because it represents the destination guest’s own identity or is transient.

Sync	Don’t sync	Why “don’t”
/var/lib/<service>/ (postgres, mariadb, redis, etcd, app-specific)	/etc/machine-id, /var/lib/dbus/machine-id	Host identity — dest got its own at pre-stage; collision causes journal/D-Bus weirdness
/srv/www/<site>/, /srv/data/	/etc/ssh/ssh_host_*	SSH host keys — clients would see a key change
/var/spool/<service>/ (postfix, cron)	/etc/hostname, /etc/hosts	Host identity
App data dirs (/opt/<app>/data/, /home/<user>/)	/var/log/, /var/cache/, /tmp/, /var/tmp/	Transient; logs should restart fresh on dest
Runtime-mutable config in /etc/<service>/ (rare — most config is static, set at deploy time)	Network config (/etc/network/, /etc/netplan/, NetworkManager state)	IP/MAC/gateway differ on dest
	/var/lib/systemd/random-seed	Per-host entropy seed
	/proc, /sys, /run, /dev	Kernel/runtime pseudo-filesystems — never on disk anyway

Use one of two rsync patterns:

• Whitelist (recommended for known workloads): rsync -avx --delete /var/lib/mariadb/ root@dest:/var/lib/mariadb/. Sync only the explicit directories you care about. Safest — anything not listed stays at pre-stage version on dest.
• Blacklist (for unknown workloads): rsync -avx --delete --exclude=/etc/machine-id --exclude=/etc/ssh/ssh_host_* --exclude=/var/log/ --exclude=/tmp/ ... /etc/ /var/ /srv/ /opt/ /home/ root@dest:/. Riskier — easy to miss an exclude.

Whitelist is much smaller and easier to verify for most workloads (databases tend to keep everything under one directory). Use blacklist only when you genuinely don’t know what the workload touches.

Constraints + gotchas

• Dest MUST stay stopped during divergence. Otherwise source and dest will diverge with no reconciliation path — both writing to their respective copies of the data. Start dest only at step 3.3.
• Per-workload knowledge required. You need to know which directories carry state worth syncing (/var/lib/<service>, /srv/<site>, /etc/<service> typically). Things like /var/log, /tmp, /var/cache usually don’t matter and can be skipped or reset.
• MAC + IP handoff is separate. The dest comes up with the source’s MAC (copied at --stateless time). If both source and dest are on the same LAN simultaneously (during pre-stage + divergence + you accidentally start dest), they will collide. This is why dest stays stopped, and why the source incus stop precedes the dest incus start.
• No --refresh on the cutover. Don’t be tempted — same -F block applies. The cutover delta must go through the file/application layer.
• Stateless databases need application coordination. If the workload has a database with open transactions, a plain rsync of /var/lib/mysql mid-flight can capture inconsistent state. Either drain the DB cleanly before the final rsync (systemctl stop mariadb, then rsync) or use the database’s native replication + promote-replica path (the Pattern A option referenced under “When this pattern is the right choice” below).

When this pattern is the right choice

• The VM or container is too large for cold-migration downtime to be acceptable
• The workload has clearly-bounded data directories (you can enumerate them)
• You can briefly drain the workload at cutover (seconds-to-a-minute is OK)

If the workload has native HA (k8s pods drain-and-reschedule, replicated DBs, stateless services behind a load balancer), prefer the application-layer warm migration path instead — it’s cleaner and downtime can be zero. Use this pre-stage pattern for pet VMs that don’t have native HA but do have well-defined state.

Aspirational walkthrough: --refresh-based warm migration (blocked on IncusOS)

This pattern does not currently work on IncusOS. It’s preserved here as a record of what --refresh is supposed to do, and remains the expected mechanism on non-IncusOS hosts with unencrypted ZFS pools. The first invocation succeeds (creates a fresh destination dataset); every subsequent invocation fails on zfs receive -F. See ZFS encryption breaks --refresh.

Migrating test3 from theia → kyojin

# 1. Initial copy. test3 stays running on theia; the volume is snapshotted
#    and zfs-sent to kyojin's local pool. Takes roughly the disk-size worth
#    of transfer time.
incus copy theia:test3 kyojin: --refresh --storage local

# 2. (Optional) Write some state inside test3 on theia so we can verify it
#    propagates to the destination.
incus exec theia:test3 -- bash -c 'date > /root/migrate-marker.txt'

# 3. Refresh. Only the delta is shipped (zfs send -i). Fast.
incus copy theia:test3 kyojin: --refresh --storage local

# 4. Cutover. Stop the source, ship the final delta, start the destination.
incus stop theia:test3
incus copy theia:test3 kyojin: --refresh --storage local
incus start kyojin:test3

# 5. Verify state preservation on the destination.
incus exec kyojin:test3 -- cat /root/migrate-marker.txt

# 6. Once you're confident the destination is healthy, delete the source.
incus delete theia:test3

--storage <pool> selects the destination pool. Without it, Incus uses the destination’s default profile pool. For test3 we pass --storage local to be explicit; for VM-class workloads (64 GiB+ root disks), pass the dedicated instance pool name instead — e.g. --storage kyojin_instances — so the volume doesn’t fill the boot drive.

Caveats

ZFS encryption breaks --refresh

IncusOS encrypts every ZFS pool by mandate (no opt-out at the IncusOS storage-layer level). Incus’s --refresh flow is:

1. Initial invocation: zfs receive creates a fresh destination dataset. Succeeds.
2. Every subsequent invocation: zfs receive -F to roll the destination back to the common snapshot before applying the incremental delta.

ZFS refuses -F against encrypted destination datasets:

cannot receive new filesystem stream: zfs receive -F cannot be used to destroy
an encrypted filesystem or overwrite an unencrypted one with an encrypted one

This applies to every IncusOS pool — local, *_instances, anything created via the incus/storage-pools role — because they all sit on IncusOS-encrypted block devices. Tested 2026-05-17 against both kyojin:local and kyojin:kyojin_instances; identical failure.

Incus has no flag to skip the -F step or to fall back to rsync between two ZFS pools (rsync transport only kicks in when source and destination drivers differ, e.g. zfs↔dir). --mode={pull,push,relay} only controls connection direction. --refresh-exclude-older doesn’t help — same -F failure on the second invocation.

Paths forward (none casual):

1. Non-encrypted ZFS pool outside IncusOS’s mandate — build an Incus storage pool backed by a block device that bypasses IncusOS’s encryption layer (raw disk passthrough or an external mount). Loses at-rest encryption on that pool’s volumes; needs threat-model review for whatever workloads land there. Probably the most practical option for a dedicated migration scratch pool on the destination host.
2. Mixed-driver migration scratch pool — a dir-driver pool on a non-encrypted mount as the destination. Forces rsync transport, bypasses -F. Slower; loses ZFS snapshot semantics on the destination.
3. Wait for upstream Incus to expose a “skip optimized zfs path” knob (none exists today).
4. Don’t use IncusOS for hosts that need warm migration — regular Debian Incus would let you choose unencrypted pools. Architectural regression.

Until one of the above is in place — or upstream issue lxc/incus-os#721 lands (proposed Dec 2025: switch IncusOS to encrypted datasets on unencrypted pool root, which would unblock zfs receive -F) — the IncusOS-compatible options are cold migration (full-transfer downtime) and pre-stage + delta cutover (near-zero cutover downtime via file-layer delta).

The VM gets a new IP after migration

test3 uses the bridged-profile, which attaches eth0 to the per-host NAT’d bridge (tfbr0, the TF-managed one with a stable CIDR). kyojin’s tfbr0 and theia’s tfbr0 are separate networks with separate DHCP. The VM’s MAC is preserved across migration, so it picks up a fresh DHCP lease from the destination’s bridge. Existing TCP connections to/from the old IP break at cutover. Profile names are scoped per-remote, so the migrated copy automatically resolves bridged-profile against the destination host’s profile of the same name without reconfiguration.

For VMs on host-LAN networking (macvlan) the same applies whenever the source and destination LANs differ — the lease is invalid on the new subnet, the VM has no working network until reconfigured inside the guest.

Cloud-init does not re-run

Anything baked into the cloud-init seed at first-boot time (users, packages, write_files, network-config) stays as-is on the destination. Editing tofu’s cloud-init.user-data and re-applying does not change the running guest — cloud-init only consumes the seed once, and the seed travels with the volume.

If post-migration the VM needs different network config (e.g. switching from theia’s bridge subnet to a static IP on a host LAN), apply that inside the guest with whichever network manager it uses (netplan, systemd-networkd, NetworkManager).

Don’t manage migrated state via tofu

After incus copy theia:test3 kyojin:, kyojin’s test3 is a manual instance, not tracked by tofu. The theia entry in tofu state still points at the (now-deleted-or-stale) source. Two ways to reconcile:

• One-shot test: treat the migration as a throwaway. incus delete kyojin:test3 after testing, and tofu apply recreates a fresh theia:test3 from spec.
• Persistent move: tofu state rm incus_instance.test3, change the resource in misc.tf to target kyojin, and tofu import the migrated instance into the new resource address. Heavy.

For routine migration, the manual instance pattern (don’t manage it as IaC after migration) tends to be simpler than fighting tofu state reconciliation.

Cross-pool-driver migration may need extra flags

If source and destination pools use different storage drivers (e.g. ZFS → btrfs), the ZFS-native incremental send doesn’t apply — Incus falls back to rsync over the migration protocol. Slower, but works. Add --mode=relay if direct migration fails (forces traffic through the local incus client instead of host-to-host).

Production-pattern checklist

For workloads with state you care about. Tailored to the pre-stage + delta cutover pattern (the IncusOS-working one); the same principles transfer to --refresh on non-IncusOS hosts.

• Pre-stage well ahead of cutover. The longer the divergence window, the larger the delta — but pre-staging early gives you time to validate the destination’s environment (storage, profile, attached devices) without a clock running. A pre-stage hours-to-days before cutover is reasonable for large or important workloads.
• Run warm-up syncs during the divergence window if the data changes a lot. Each warm-up rsync shrinks the final-cutover delta. For an active database, a warm-up every few minutes for the last 20–30 minutes before cutover keeps the final delta small.
• Always stop services in source before the final delta sync. Otherwise writes during the final sync window are lost or land in inconsistent state — the source keeps writing while you’re rsyncing, the cutover happens, and those last writes never made it to dest. Drain the service (systemctl stop, application-level quiesce) before the final rsync.
• Schedule the cutover during a quiet window. Downtime is the final-delta-sync time + incus stop source + incus start dest. For a workload with frequent warm-ups, this is seconds to a minute. Pick a window where that’s tolerable.
• Keep the source around briefly after cutover as a fallback. Once you’ve validated the destination is healthy (services answering, traffic flowing, no errors), delete the source — but giving it a 24h grace period is cheap insurance. Be careful not to start it accidentally during the grace period (MAC collision with dest).
• For stateful services with active connections (databases, message queues, websocket fan-out), the cutover drops connections. Either coordinate with the application (graceful drain → cutover → reconnect) or use application-layer replication (e.g. PostgreSQL streaming replication with switchover, MariaDB primary-replica) instead of disk-layer warm migration. Application-layer is usually a better fit for these workloads.
• Validate traffic-routing handoff explicitly. Caddy upstream changes, DNS A records, k8s service routing, etc. — write down the steps + verify after cutover that traffic actually shifted. Source serving stale data because traffic didn’t move is a worse failure than visible downtime.