Warm-migration validation

Question this document answers: Is incus copy --refresh (warm migration) good enough to replace XCP-ng’s migration workflows for our use case, or do we need to take on the cost of running an Incus cluster?

Working hypothesis: warm migration is sufficient. Clustering’s primary value is live migration (zero-downtime), which on Incus requires shared/distributed storage (Ceph, NFS, iSCSI, FibreChannel) — infrastructure we don’t have and don’t currently plan to deploy. Without shared storage, even a clustered Incus deployment falls back to warm-migration semantics for cross-host moves. So clustering wouldn’t actually buy us live migration, just operational complexity.

This doc is the test plan to validate that hypothesis hands-on, using test3 as the migration subject. Pass/fail criteria are explicit so future re-evaluations can re-run the same tests.

Why this matters

We’re prototyping a replacement for XCP-ng. XCP-ng admins are used to:

Cold migration — VM stopped, disk copied to destination, started. This is the XCP-ng default for hosts in different pools or without shared storage. Downtime = stop time + copy time.
Warm migration — XCP-ng has this as a paid feature in some editions: snapshot → copy → delta sync → cutover. Downtime = final delta + start.
Live migration — only possible within a pool with shared storage. Downtime ≈ none.

incus copy --refresh is functionally equivalent to XCP-ng’s warm migration. The XCP-ng users we’re migrating from largely rely on cold migration for cross-host moves; warm migration is a strict upgrade in their experience. Live migration is not a workflow they currently use cross-pool, so we’re not regressing on capability.

If warm migration validates here, the conclusion is: stay de-clustered, accept that any future “we need live migration” requirement triggers a separate decision (do we deploy shared storage, and is it worth the cost?).

When clustering would actually be worth it

Document these triggers so the decision can be revisited later:

Trigger	Why it changes the calculus
Deploying shared/distributed storage (Ceph, NFS-backed volumes, iSCSI, etc.)	Unlocks real live migration. Clustering becomes the API to use it.
HA-required workloads with strict zero-downtime SLAs	Warm migration’s seconds-to-minutes downtime becomes unacceptable. Need live migration → need clustering + shared storage.
Fleet grows past ~10 hosts with mobile workloads	Manual `incus copy` per workload doesn’t scale; clustering’s distributed scheduler starts paying off.
Multi-tenant / strong network isolation requirements	Clustering + OVN gives you the SDN primitives (network ACLs, floating IPs, multi-tenant networks). Warm migration alone doesn’t address this.

Until one of those is true, warm migration is the right tool.

Test subject: `test3`

Defined in opentofu/misc.tf:

VM type, 1 vCPU, 2 GiB RAM, 16 GiB root on local
Initial host: theia
Profile: theia:bridged-profile (gets eth0 on theia’s tfbr0 + DHCP from 10.114.104.0/24)
Cloud-init image: vm_debian (Debian Trixie cloud)

Small enough that initial copy is fast (~minutes, not hours) and iteration is quick.

Test plan

Five tests in increasing difficulty. Run in order; each builds on the previous. Record pass/fail and downtime measurements.

Test 1: Basic round-trip migration

Goal: Verify incus copy --refresh works end-to-end between theia and kyojin.

# Initial copy (test3 keeps running on theia)
incus copy theia:test3 kyojin: --refresh --storage local

# Verify the copy exists on kyojin
incus list kyojin: test3 -c L,n,s,4

# Cutover
incus stop theia:test3
incus copy theia:test3 kyojin: --refresh --storage local
incus start kyojin:test3

# Verify it's running on kyojin
incus list kyojin: test3 -c L,n,s,4

# Round-trip back to theia
incus stop kyojin:test3
incus copy kyojin:test3 theia: --refresh --storage local
incus start theia:test3
incus delete kyojin:test3

Pass criteria:

All commands succeed without manual intervention
VM runs healthily on the destination after each cutover
No data corruption (ext4/xfs mounts cleanly)

Fail signals: copy errors, VM fails to start on destination, kernel oops on boot.

Test 2: State preservation across migration

Goal: Verify disk state survives migration — files written on the source persist on the destination.

# Write a marker file on theia:test3
incus exec theia:test3 -- bash -c '
  date -Iseconds > /root/migrate-marker.txt
  echo "host: $(hostname)" >> /root/migrate-marker.txt
  dd if=/dev/urandom of=/root/random-1mb.bin bs=1M count=1
  sha256sum /root/random-1mb.bin > /root/random-sha256.txt
'

# Migrate theia → kyojin (cutover)
incus stop theia:test3
incus copy theia:test3 kyojin: --refresh --storage local
incus start kyojin:test3
incus delete theia:test3

# Verify state on kyojin
incus exec kyojin:test3 -- cat /root/migrate-marker.txt
incus exec kyojin:test3 -- bash -c 'sha256sum -c /root/random-sha256.txt'

Pass criteria:

Marker file content matches what was written on theia
1 MB random file’s SHA256 still verifies on kyojin

Fail signals: missing files, content mismatch, sha256 mismatch.

Test 3: Downtime measurement

Goal: Quantify the cutover window — how long is the VM unreachable?

# Pre-cutover: get a baseline. Start a continuous ping from another VM on the
# same destination host, against test3's `tfbr0` IP. (Both pings must originate
# from the same host as the destination, since `tfbr0` is per-host NAT.)

# On kyojin, start a netshoot container pinging test3's eventual kyojin IP:
# (This is a sketch — adapt to actual IP after kyojin DHCP assigns one.)

# Better: instrument from inside test3 itself with a simple HTTP server.
# Pre-migration, on theia:test3:
incus exec theia:test3 -- bash -c '
  cat > /root/server.py <<EOF
import http.server, time
class H(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200); self.end_headers()
        self.wfile.write(f"{time.time()}\n".encode())
    def log_message(self, *a): pass
http.server.HTTPServer(("0.0.0.0", 8080), H).serve_forever()
EOF
  nohup python3 /root/server.py >/tmp/srv.log 2>&1 &
'

# From a vantage point reachable to BOTH hosts' `tfbr0`s, hit the server in a
# loop and time the gap during cutover. Since `tfbr0` is per-host NAT, this
# vantage point has to be on each host respectively. In practice: instrument
# WITHIN test3 by writing a heartbeat to a file on a separate persistent volume
# that survives migration, and look at the gap in timestamps.

# Simpler: measure cutover wall-clock time directly:
START=$(date +%s.%N)
incus stop theia:test3
incus copy theia:test3 kyojin: --refresh --storage local
incus start kyojin:test3
END=$(date +%s.%N)
echo "Cutover took: $(echo "$END - $START" | bc) seconds"

# Wait for VM to be reachable (cloud-init done):
until incus exec kyojin:test3 -- true 2>/dev/null; do sleep 1; done
END_AVAIL=$(date +%s.%N)
echo "VM available after cutover: $(echo "$END_AVAIL - $START" | bc) seconds"

Pass criteria:

Cutover (stop → copy → start) completes in under 60 seconds
VM exec-reachable within 90 seconds total

Acceptable downtime targets for this prototype:

Stateless workloads: any value < 5 min is fine
Services with active connections (SSH sessions, HTTP keep-alives): < 30 sec preferred so retries succeed cleanly

Test 4: Repeated round-trip stability

Goal: Verify there’s no accumulating state corruption across multiple migrations.

# Run the round-trip 3 times. Each round, increment a counter file inside the VM.
incus exec theia:test3 -- bash -c 'echo 0 > /root/counter.txt'

for i in 1 2 3; do
  echo "=== round $i: theia → kyojin ==="
  incus exec theia:test3 -- bash -c 'echo $(($(cat /root/counter.txt) + 1)) > /root/counter.txt'
  incus stop theia:test3
  incus copy theia:test3 kyojin: --refresh --storage local
  incus start kyojin:test3
  incus delete theia:test3

  echo "=== round $i: kyojin → theia ==="
  incus exec kyojin:test3 -- bash -c 'echo $(($(cat /root/counter.txt) + 1)) > /root/counter.txt'
  incus stop kyojin:test3
  incus copy kyojin:test3 theia: --refresh --storage local
  incus start theia:test3
  incus delete kyojin:test3

  incus exec theia:test3 -- cat /root/counter.txt
done

# After 3 rounds, counter should be 6.

Pass criteria:

Counter ends at 6
VM remains healthy through all rounds (no kernel issues, no filesystem errors)
Total runtime under 30 min for the full sweep

Test 5: Incremental refresh efficiency

Goal: Verify that subsequent --refresh calls send only deltas, not full disk.

# Initial copy and time it
echo "=== initial copy ==="
time incus copy theia:test3 kyojin: --refresh --storage local

# Make a small change inside test3 on theia (just write a small file)
incus exec theia:test3 -- bash -c 'date > /root/refresh-test.txt'

# Refresh — should be much faster
echo "=== refresh after small change ==="
time incus copy theia:test3 kyojin: --refresh --storage local

# Make a larger change (write 100 MB)
incus exec theia:test3 -- bash -c 'dd if=/dev/urandom of=/root/100mb.bin bs=1M count=100'

# Refresh — should be roughly 100 MB worth of transfer time
echo "=== refresh after 100 MB change ==="
time incus copy theia:test3 kyojin: --refresh --storage local

# Cleanup the dest copy
incus delete kyojin:test3

Pass criteria:

Initial copy takes ~16 GiB worth of transfer time
Refresh after small file: under 30 seconds (mostly snapshot + protocol overhead)
Refresh after 100 MB file: scales with the data size, not the disk size — i.e., much less than initial copy

Fail signals: Every refresh takes initial-copy duration (zfs incremental not engaging); refresh takes wall-clock minutes when it should take seconds.

Recording results

After running, capture the actual numbers in this doc (replace this section with the table). Future re-evaluations of “should we cluster?” can use these data points.

Test	Result	Notes
1. Basic round-trip
2. State preservation
3. Downtime measurement	cutover: __s, available: __s
4. Repeated round-trip
5. Refresh efficiency	initial: __s, small delta: __s, 100 MB delta: __s

Conclusion criteria

If all 5 tests pass: warm migration is validated for our prototype. Stay de-clustered. Document this decision in the production-readiness review.

If test 1, 2, or 4 fail: something fundamental is wrong with the migration workflow — debug before deciding. Failures here are usually network (firewall blocking the migration protocol port) or storage (zfs send/receive incompatibility); they’re not reasons to switch to clustering.

If test 3 (downtime) exceeds acceptable targets for a specific workload: that workload may need either application-level HA (preferred — survives any underlying restart) or, eventually, real live migration (which means shared storage + clustering — a separate, larger decision). Don’t let one demanding workload force a re-cluster; isolate it.

If test 5 (refresh efficiency) fails (every refresh takes full-copy time): zfs incremental isn’t working between source and destination pools. Check that both local pools are zfs (they should be — IncusOS-managed). If different drivers somehow, fall back to rsync mode (--mode=relay) — slower but functional.

Re-evaluation triggers (future you, please re-read)

Re-run this validation if:

We deploy shared/distributed storage (Ceph, etc.) — clustering becomes potentially worth it for live migration
We add a workload with strict zero-downtime requirements that can’t tolerate Test 3’s cutover window
The fleet grows past ~10 hosts and unified scheduling/management starts to matter
Migration becomes routine (multiple per day) — automation around incus copy --refresh is needed; if that automation gets complex, clustering’s API may be cleaner

Until one of those happens, the answer remains: warm migration is enough.

Warm-migration validation

Why this matters

When clustering would actually be worth it

Test subject: test3

Test plan

Test 1: Basic round-trip migration

Test 2: State preservation across migration

Test 3: Downtime measurement

Test 4: Repeated round-trip stability

Test 5: Incremental refresh efficiency

Recording results

Conclusion criteria

Re-evaluation triggers (future you, please re-read)

Test subject: `test3`