Instance export & DR

Instance export is the VM-level counterpart to IncusOS’s host-level system backup (which, with its key-custody handling, is an operator-side doc — not published). Different tool, different artifact, different use case:

	IncusOS system backup (this doc)	Instance export
Command	`incus admin os system backup <remote>:`	`incus export <remote>:<inst>`
Captures	OS-level: encryption keys, state.txt, hostname/network config	Instance-level: VM/container config + root volume content
Size	~1 KiB (no application data)	GBs (root volume snapshot)
Restore command	`incus admin os system restore` (broken in 2026-05; operator note)	`incus import <remote>: <file>.tar.gz` (works)
Use case	Pre-reset capture of host-level state	Pre-teardown VM snapshot, DR for unreproducible VM state

Instance exports are the restore-from-tarball workflow operators reach for when a TF-managed VM has accumulated state that can’t be reproduced from declarative config. Best practice in this codebase says you shouldn’t need this routinely — design data planes outside the VM (passthrough devices, separate incus_storage_volume resources, network mounts) so destroy/recreate via TF is safe. But the workflow is there for genuine disaster scenarios, and worth documenting.

When instance export/import is the right tool

Scenario	Right tool?
Pre-teardown safety net before a risky change	✓ yes (cheap insurance)
Whole-host failure, time-critical recovery on a different host	✓ yes (faster than re-Ansible’ing from scratch)
Forensics / point-in-time inspection	✓ yes (operational archaeology)
Routine “restore TF-managed VM” workflow	✗ no — fix the data plane separation instead
Backing up data that lives inside the VM	✗ no — use an application-layer backup tool (etcd snapshots for k0s, borg for filesystems, db-native dumps for databases)

Capture command

mkdir -p /configs/incus/seeds/backups/instances
incus export kyojin:backup2 \
  /configs/incus/seeds/backups/instances/$(date -u +%Y%m%dT%H%M%SZ)_kyojin-backup2.tar.gz \
  --optimized-storage \
  --instance-only

--optimized-storage uses ZFS snapshot / zfs send rather than tar-walk: ~10× faster, no VM downtime. --instance-only skips snapshots if any.

Naming convention: <ISO_DATETIME>_<INCUS_HOST>-<INSTANCE_NAME>.tar.gz. Documented in /configs/incus/seeds/README.md (operator-side, outside the repo).

Restore runbook

The mechanical restore is straightforward. Re-adopting into TF state has caveats — see “TF reconciliation” below.

# 1. If the original instance still exists, detach TF and remove it
tofu -chdir=opentofu state rm incus_instance.<name>
incus stop <remote>:<name>
incus delete <remote>:<name>

# 2. Import the tarball
incus import <remote>: /configs/incus/seeds/backups/instances/<file>.tar.gz
incus start <remote>:<name>

# 3. (Optional) Re-adopt into TF state
tofu -chdir=opentofu import 'incus_instance.<name>' '<remote>:<name>'

Step 1’s tofu state rm is needed before incus delete because TF would otherwise see the resource missing on next plan and propose to recreate it (which would race with the manual import). State-removing first decouples them.

Step 3 is optional — if the imported VM is meant to be a temporary recovery instance (e.g., to extract a one-off file or finish an in-progress workload), you can skip TF re-adoption and just delete it manually when done.

TF reconciliation: the lifecycle-hooks gotcha

After step 3’s tofu import, tofu plan will likely propose to replace the imported VM. Two reasons:

Image fingerprint drift. The imported VM was built from a specific image fingerprint at the time of capture. Today’s incus_image.vm_debian["<host>"] resource resolves to a fresh fingerprint (auto-pulled on each tofu apply). TF sees the difference as image: "<old-fp>" -> "<new-fp>" # forces replacement and wants to destroy + recreate.
Cloud-init formatting. Incus normalizes the cloud-init.user-data config string on read (whitespace handling). TF’s HCL <<-EOT heredoc renders slightly differently. This is an in-place update only (no replacement), so cosmetic but noisy.

If you let TF apply this, the imported VM gets destroyed and you’re back to a TF-fresh instance — defeating the import. To prevent that, add lifecycle.ignore_changes to the resource definition for the duration of the recovery:

resource "incus_instance" "backup2" {
  # ... existing fields ...

  lifecycle {
    ignore_changes = [
      image,                              # don't churn on image fingerprint drift
      config["cloud-init.user-data"],     # don't churn on cloud-init normalization
    ]
  }
}

After the lifecycle hook is committed, tofu plan shows zero drift on the imported VM. TF tracks devices, profiles, metadata going forward; it just won’t re-image or re-cloud-init.

This is a recovery-mode patch, not a permanent codebase state. Once the incident is resolved (e.g., you’ve extracted the data you needed and run setup-<host>-servers.yaml to bring the VM to current state), remove the lifecycle hook so subsequent applies work normally.

Why this isn’t TF-clean

OpenTofu’s mental model is “config is the source of truth; resources reflect config.” Imported state has bits TF didn’t compute (volatile fields, image fingerprints, cloud-init formatting), so reconciliation is always partly manual. The lifecycle hook is the official escape valve for this case, but it’s a patch, not a feature.

The cleaner long-term answer for any resource with restore-worthy data: separate the data plane from the VM lifecycle (passthrough devices, separately-TF-managed volumes, external storage) so destroy/recreate via TF is safe. Then the tarball is purely DR insurance, never routine restore.

In this codebase:

backup2 — JBOD on PCI passthrough; ZFS pool auto-imports via drive metadata; no VM-level state worth preserving. Recreate via TF; pool comes back automatically. ✓
logi — separate incus_storage_volume.logi_ollama for Ollama models; recreate VM, volume preserved. ✓
k0s VMs — etcd state on root, but k0s has k0s backup/k0s restore for that layer. Don’t use incus export for k0s; use the k0s tool. ✓
Test instances — pure cattle. Destroy + recreate, who cares. ✓

So the instance-export workflow is DR-only in this codebase. Worth knowing, rarely needed.

Validation: the cycle works

Verified end-to-end on 2026-05-06:

incus export kyojin:backup2 → 978 MB tarball at /configs/incus/seeds/backups/instances/20260506T215505Z_kyojin-backup2.tar.gz
tofu state rm incus_instance.backup2 + incus stop + incus delete kyojin:backup2
incus import kyojin: <tarball> (~90 seconds)
incus start kyojin:backup2 → VM RUNNING at original DHCP-reserved IP 10.112.113.252
Cloud-init packages preserved (openssh, nftables, zfs-dkms, borgbackup, restic, smartmontools, etc.)
JBOD ZFS pool auto-imported by zfs-zed on first boot — 29.1 TB pool, 18 TB borg backup data immediately accessible. No Ansible run required for data recovery.

The auto-import behavior means even setup-backup-servers.yaml’s --tags zfs step is redundant for an imported VM — the only thing the playbook still needs to do is --tags clients to render the borg client allowlist.