Backup strategy
This document captures what gets backed up, where it goes, how often, and how recovery actually works. It records the architecture decisions and the test plan for verifying restores work — because untested backups are documentation, not insurance.
Why this matters
The current state is dangerous: backup2 exists as an Incus VM on kyojin with USB JBOD passthrough (~30 TB raw across 4 drives) but there is no documented backup pipeline using it. A site-level event (fire, theft, multi-disk failure, malicious tenant, accidental tofu destroy) loses everything. The prototype is past the point where this is acceptable for a production-replacement.
Three categories of risk this is meant to mitigate:
| Risk | Failure mode | Recovery requirement |
|---|---|---|
| Hardware failure (single disk, single host) | A drive dies, a host bricks (see ibuypwr) | Restore individual workloads to surviving infra in hours |
Operator error (tofu destroy, accidental delete, bad migration) | Recent loss, otherwise infra healthy | Restore specific items (pod data, repo, config) in minutes |
| Site-level event (fire, theft, ransomware on backup2 itself) | All on-site copies inaccessible | Restore from off-site copy in days, with degraded service acceptable |
Each requires a different copy. 3-2-1 rule: at least 3 copies, on 2 different media, with 1 off-site.
What gets backed up
Categorize by what it is, where it lives, and what software produces it:
| Data | Location now | Backup target | Why it matters |
|---|---|---|---|
| OpenTofu state | opentofu/terraform.tfstate (committed to git) | Git remote + backup2 + off-site | Source of truth for Incus resources; loss = re-import nightmare |
| SOPS-encrypted secrets | k0s/platform/*-values.yaml (committed to git) | Git remote + backup2 + off-site | Re-deriving credentials is painful, sometimes impossible |
| Age private key | operator-side, outside the repo | Encrypted off-site (e.g. password manager + cold storage) | If lost, every encrypted secret is unrecoverable |
| Ansible inventory + roles | git-tracked, pushed to Forgejo | Git remote + backup2 + off-site | Same as Tofu state |
| k8s etcd | Inside k0s controllers | backup2 (snapshot via k0s etcd snapshot) + off-site | Cluster state; without it, recreating the cluster from scratch |
| OpenBao unseal/recovery keys | operator-held | Off-site only — multiple custodians | Sealed Bao is bricked without these |
| Forgejo repositories | PVC inside k0s | backup2 (rsync/restic from Forgejo data dir) + off-site | Source code; the repo this infra lives in. Backing up Forgejo to itself is circular. |
| Zot OCI images | PVC inside k0s | backup2 (restic of registry storage) | Recoverable by re-pulling from upstream, but slow and creates external dependencies |
| Prometheus TSDB | 50Gi PVC, 90d retention | Optional — large, recoverable from re-scrape | Historical metrics; nice to have, not critical |
| Loki log chunks | 20Gi PVC | Optional — same as Prometheus | Historical logs; same trade-off |
| Grafana dashboards | Inside Grafana DB (sqlite/PVC) | backup2 (sqlite dump) + git (export-as-json on commit) | Cheap to back up, painful to recreate |
| Application data (Actual Budget, OWASP Juice Shop, etc.) | PVCs | backup2 + selective off-site | Per-app prioritization |
| VDI persistent state | PVCs in vdi namespace | backup2 only (per-user, large, low-priority) | User home dirs in webtops |
| IncusOS pool recovery keys | operator-side, outside the repo | off-site backup of operator workstation | Without them, factory-reset = data loss on user pools |
| Ansible SSH key | operator-side, gitignored | Off-site (encrypted) | Lost = regenerate + re-key every host |
Software choices
Three layers, each with its own tool
| Layer | Tool | Use |
|---|---|---|
| Block-level instance snapshots | incus snapshot (ZFS-backed, instant) | Per-VM point-in-time, lives on the same host’s pool. First line of defense — quick rollback after a bad migration. |
| Block-level cross-host replication | ZFS send/receive (via incus copy --refresh or zrepl) | Replicate snapshots from kyojin/theia to backup2. Block-efficient (delta-only). Same pattern as warm-migration but with snapshots as the unit. |
| File-level dedup’d encrypted backup | Restic | Volume contents → backup2 + off-site. Encrypted, deduplicated, supports many backends (local, S3, B2, SFTP). Granular file-level restore. |
This layered approach gives:
- Fast rollback via Incus snapshots (seconds, no data movement)
- Host loss tolerance via ZFS replication to backup2 (minutes-to-hours, depends on data volume)
- Site loss tolerance via Restic to off-site (hours-to-days, fully encrypted at rest)
Why Restic specifically
| Tool | Encryption | Dedup | Backends | Verdict |
|---|---|---|---|---|
| Restic | Yes (AES-256) | Yes (chunk-level) | Local, SFTP, S3, B2, Azure, GCS, REST | Pick. Modern, well-maintained, scriptable, content-addressed |
| Borg | Yes | Yes | Local, SFTP | Comparable; ecosystem smaller, no native S3 |
| Bup | Yes | Yes (git-pack format) | Local | Mature but quieter project |
| Duplicity | Yes | No (incremental tarballs) | Many | Old-school; less efficient |
tar + gpg | Manual | No | Anywhere | Reliable but high-effort to manage retention |
Restic’s S3/B2 support makes off-site cheap (Backblaze B2 is ~$0.005/GB/month).
Where backups live
On-site: backup2 VM with USB JBOD
backup2 is an Incus VM on kyojin with PCI passthrough of the ASMedia ASM2142 USB controller (PCI 0000:17:00.0). It owns the 4-drive JBOD enclosure (~30 TB raw, configured as ZFS — exact topology TBD).
- Restic repos stored on the JBOD pool — one repo per data category (e.g.
forgejo,zot,etcd,vm-snapshots) - ZFS receive targets — replicated
*_instancessnapshots from kyojin and theia - Quarterly/yearly archives on slower or cold-storage media if needed
backup2 itself needs to be backed up via off-site (otherwise it’s the SPOF for the on-site copy).
Off-site: cloud object storage
Backblaze B2, Wasabi, or AWS S3 (Glacier Deep Archive for cheap cold storage):
- Restic repo replicated from
backup2(viarestic copy) - Critical secrets (Age key, OpenBao recovery keys) — separately encrypted, multiple custodians, manual procedure (not part of the automated pipeline)
- Quarterly snapshots of the entire backup2 Restic repo as cold-storage anchors
Off-site: git remote (for code/config)
Already in place via Forgejo + GitHub mirror (if configured). OpenTofu state, SOPS-encrypted values, Ansible inventory are all in git — pushing to a remote outside our infra IS off-site backup for them.
Mirror to a second git remote (GitHub, GitLab, another Forgejo) so a Forgejo outage doesn’t lose access. The repo’s primary identity stays local Forgejo; the mirror is read-only safety.
Schedule and retention
| Backup type | Frequency | Retention | Where |
|---|---|---|---|
| Incus instance snapshots | Daily | 7 daily, 4 weekly | Same host |
| ZFS send to backup2 | Daily | 30 daily, 12 weekly, 12 monthly | backup2 JBOD |
| Restic to backup2 (k8s PVC dirs) | Hourly for hot data, daily for cold | Restic policy: 24 hourly, 14 daily, 8 weekly, 12 monthly | backup2 JBOD |
| Restic copy to off-site | Daily | 90 days hot tier, then 365 days cold | B2/S3 |
| Age key / recovery keys | On rotation | Indefinite | Secure off-site (multiple custodians) |
| Git push to mirror | On every commit | Indefinite | Second git host |
Restic’s forget --keep-* policy handles automatic pruning.
Orchestration
Where the backup runs from:
- Cron jobs on each Incus host for instance-level work (snapshots, ZFS send to backup2)
- CronJobs in k0s for k8s-internal work (etcd snapshot, Restic of PVC contents — using a sidecar pattern or dedicated backup pod)
- Cron on backup2 for the off-site copy (Restic to B2/S3)
- Ansible playbook for the initial setup of all the above; idempotent so re-running fixes drift
Implementation: an ansible/playbooks/backup/ directory with playbooks per concern. The Restic repo init step is one-time, manual (need to securely handle the Restic password); afterwards everything is automated.
Restore testing
Untested backups are documentation, not insurance. Schedule restore drills:
| Drill | Frequency | What you verify |
|---|---|---|
| Full Forgejo restore to a scratch namespace | Monthly | k8s PVC restore + service start works end-to-end |
| Single-VM restore from instance snapshot | Monthly | Incus snapshot/copy mechanics work |
| etcd restore to a fresh k0s cluster | Quarterly | k8s state can be recovered after total cluster loss |
| Off-site restore (B2 → backup2 → target) | Quarterly | Bandwidth and credentials are working; no surprises |
| Bare-metal recovery: kyojin loses all disks → reinstall + restore | Annually | Full DR runbook works; secret recovery flow works |
Document each drill’s outcome (in a sibling doc, e.g. docs/backup-drill-log.md). The drills aren’t optional — they catch silent backup corruption, expired credentials, missing dependencies. A backup you’ve never restored is Schrödinger’s data.
Phased plan
Phase 1: in-cluster backup (get something working)
- Stand up Restic repo on backup2’s JBOD (Ansible playbook to init the pool, install Restic, generate password)
- Add CronJobs in k0s for: etcd snapshot → write to PVC → Restic to backup2; Forgejo data dir → Restic to backup2
- Daily Restic of OpenBao data
- Test single-PVC restore to a scratch pod
Outcome: covers the most painful “operator error / single-PVC corruption” recovery scenarios.
Phase 2: instance-level backup
- Cron on kyojin/theia: nightly
incus snapshot create+zfs sendof the instance pools to backup2 - Cron on backup2: receive, retain per policy
- Document the per-instance restore procedure (
incus copy backup2:<snap> kyojin:<new-name>)
Outcome: covers “host loss” — losing kyojin or theia while preserving instance state.
Phase 3: off-site
- Provision B2/S3 bucket with credentials in OpenBao
- Cron on backup2: nightly
restic copyfrom local repo to remote - Bandwidth test; tune schedule and dedupe targets if needed
- First restore drill from off-site
Outcome: covers site loss.
Phase 4: ongoing drills
Schedule the drill calendar; build the runbooks; train a second operator on them.
Decision log
- Restic over Borg. Cloud backend support (B2/S3) is the deciding factor. Borg ecosystem is solid but lacks native cloud targets —
borg + rcloneworks but adds another moving part. - ZFS send/receive for block-level, Restic for file-level — both, not either. Different recovery scenarios benefit from different tools. ZFS is faster for full-VM restores; Restic is granular for “I need that one file from last Tuesday.”
backup2on-site as the primary aggregation point. Single backup target simplifies orchestration (everything pushes to one place). Trade-off: backup2 is a SPOF for on-site backups; mitigated by off-site copy.- Off-site to commercial S3-compatible (B2/Wasabi), not another DC. RRCHNM presumably has a second site option (HPC?) but cloud is cheaper and faster to stand up. Revisit if compliance forbids cloud or if egress costs become significant.
- No Velero (k8s-native backup tool). Velero is heavier (operator + CRDs) and overlaps with what Restic already provides. Worth revisiting if we adopt CSI volume snapshots and want their cluster-aware orchestration.
Re-evaluation triggers
- A backup fails or a restore drill fails → emergency redesign of the failing piece
- Storage cost on backup2 or B2 grows beyond budget → reconsider retention, dedup, or move cold tier to truly-cold storage (Glacier Deep Archive, tape)
- Compliance regime requires audit logs, immutability, multi-tenant isolation → Restic’s append-only mode + S3 Object Lock; or switch to a tool with stronger guarantees
- Recovery time objective (RTO) becomes formal and measurable → may need warm replicas (continuous replication) instead of nightly snapshots
- A critical secret loss event happens → retrospective on Age key and OpenBao recovery key handling, custodian model, secure storage process