Skip to content
RRCHNM Infra

Incus Operations Center

Status: planned. Will deploy on a dedicated SFF (small form-factor) PC running IncusOS as a bare-metal host, when the hardware lands. Earlier 2026-05 attempts at IncusOS-on-Incus VM and Debian-VM-on-ibuypwr deployments are abandoned — see Decision log below.

Goal

A single UI + REST API for centrally managing the four standalone IncusOS hosts (kyojin, theia, hyperion, ibuypwr) without re-clustering Incus. Specifically: cross-host inventory, fleet-wide updates, system + per-app backup/restore, single-pane-of-glass diagnostics. Per the upstream reference/server docs, OC manages individual IncusOS instances as servers independent of any cluster abstraction — our four standalone daemons fit that model directly. No Incus-native re-clustering required.

This is separate from the OVN integration question in plan-incus-ovn-integration.md. OC manages the IncusOS layer; OVN would manage VM-network plumbing.

Key concepts (from upstream docs)

  • Server vs. cluster. A “server” is one IncusOS instance registered with OC. A “cluster” is a group of servers with shared workloads (implies Incus-native clustering). Servers can exist in OC without being in any cluster — that’s our deployment shape.
  • Two CLIs. incus (existing, talks to managed daemons) and operations-center (new, talks to the OC daemon). The operations-center binary needs to be installed on the operator’s workstation.
  • Provisioning tokens. OC generates pre-seeded IncusOS install ISOs that bake in a registration token. New hosts boot from the token-flashed ISO and self-register with OC on first boot — no manual seed crafting, no boot-drive-serial discovery in advance.
  • OC’s own trust model. Separate from the operator’s existing Incus client cert. Trust is established by registering the cert fingerprint via operations-center system security edit. Reusing the existing operator cert material is fine, but the OC-side fingerprint registration is a distinct step.
  • Port 8443 when run as IncusOS app, 7443 when standalone. Conflicts with incus’s 8443 — OC needs its own IncusOS instance, hence the dedicated bare-metal host.

Target deployment shape

  • Hardware: dedicated SFF PC (TBD — sourcing pending). Modest specs sufficient: 4 vCPU / 8 GiB RAM / 256 GiB SSD is comfortable. No GPU, no JBOD, no special I/O.
  • OS: IncusOS, with operations-center as the primary application (replaces incus on this host — they conflict on port 8443 and only one primary app can run per IncusOS instance).
  • Network position: RH461 campus subnet (10.112.113.128/25), DHCP-reserved IP. Reachable from operator clients and from the four managed IncusOS hosts.
  • Naming convention: Norse, like the rest of the fleet. mimir (the keeper of wisdom whose well sits at Yggdrasil’s roots) is the working candidate — picks up the “watches everything” connotation cleanly.

Bring-up procedure

Two phases: bootstrap the OC instance itself, then onboard the four production hosts to it.

Phase 1 — Bootstrap OC

  1. Coordinate DHCP reservation with network admin. Allocate a 10.112.113.x IP in the RH461 reservation pool.
  2. Build install media via the IncusOS web customizer (https://incusos-customizer.linuxcontainers.org/ui/):
    • Primary application: operations-center (NOT incus)
    • Optional: debug
    • Network: hostname + MAC + DHCP4
    • Trusted client cert: re-use the operator’s Incus client cert at /configs/incus/client.crt (the public half of the operator identity SOPS-decoupled from the repo; see ansible/playbooks/incus/ops/factory-reset.yaml’s input_incus_trusted_cert assert for how it’s passed at runtime). The browser will need this cert imported as PKCS#12 to access OC’s web UI.
  3. Add the host to ansible/inventory.yaml under incus_servers (mac_addr, ansible_host, network_mode=dhcp). Documentation only — IncusOS hosts are managed via the Incus admin API, not SSH.
  4. Flash the customized install image to a USB drive, boot the SFF PC, install proceeds unattended.
  5. Install the operations-center CLI on the operator’s workstation. The incus CLI is already there; both are needed.
  6. Establish trust: convert the operator’s Incus client cert to PKCS#12, register its fingerprint with OC via operations-center system security edit, import the PKCS#12 into the browser for UI access.
  7. Capture security state per the factory-reset runbook’s pre-reset key-capture step (operator-side, outside the repo).

Phase 2 — Onboard the four production hosts

Upstream docs are thin on this. Two paths to investigate during bring-up:

  • (a) Manual server registration via OC API. If OC supports adding existing IncusOS daemons by URL + cert fingerprint without a token-flashed ISO, this avoids re-installing our hosts. Likely path; needs CLI / API exploration during deploy.
  • (b) Token-flashed re-install. Generate a token via operations-center provisioning token add, build a per-host pre-seeded ISO via operations-center provisioning token get-image <token> <out.iso> --architecture x86_64 --type iso, factory-reset each host with the token-baked image. Destructive — only fall back to this if (a) doesn’t exist.

Document the working path here once validated.

Why not the alternatives we tried

Option (rejected, 2026-05): IncusOS-on-Incus VM on ibuypwr

Tried 2026-05-07. Got blocked at the install-seed plumbing layer:

  • The IncusOS install ISO embeds its seed at the start of partition 2 (seed-data GPT partition). When booted from a CD-ROM-typed Incus volume, the kernel mounts iso9660 and the GPT partitions are masked, so IncusOS can’t find the seed at install time.
  • Workaround: attach a separate SEED_DATA-labeled FAT partition as a second disk. Requires parted/mkfs.fat/xorriso to construct, none of which are available in our control-node environment without significant package-source maneuvering.
  • Forum-blessed alternative: use the IncusOS customizer to bundle the seed at download time. That works for bare-metal USB flashing but didn’t apply in our Incus-VM context — Incus presented the volume as a CD-ROM regardless of customizer output.

The upstream OC tutorial uses the same --type=iso + boot-media disk shape we tried, suggesting the issue may be context-specific or resolvable with closer attention to the customizer flags. If we revisit this path to avoid the SFF PC hardware ask, the next attempt should:

  • Test on a host with capacity (probably ibuypwr or theia post-setup-theia.yaml)
  • Diff our 2026-05 customizer output against a current upstream-tutorial-shaped ISO for the seed payload location
  • File an upstream issue if the seed-discovery failure recurs with current images

Until that retest happens, the SFF PC plan stands.

Option (rejected, 2026-05): Debian VM running operations-centerd as a Go service

Tried 2026-05-07. Hit an upstream limitation: the operations-center GitHub release assets ship only the CLI client (operations-center), not the daemon (operations-centerd). Per the Makefile, the daemon is built by go build ./cmd/operations-centerd — i.e. it’s compiled from source as part of the IncusOS application image construction, not distributed separately.

Could be built from source on Debian (Go 1.25.6 + a vanilla go build, no special CGO), but introduces a build pipeline for a single binary, drifts from the upstream-blessed deployment pattern, and forfeits OC’s own update story. Not worth the divergence.

Why a dedicated SFF PC + IncusOS is the right answer for now

  • Matches the upstream-blessed pattern. OC IS designed to ship as an IncusOS application — bypasses the seed-plumbing fight from the IncusOS-on-Incus attempt, gets the daemon ship-shape with no source-build pipeline.
  • Customizer handles seed embedding cleanly. USB flash → boot → done. No FAT/GPT yak-shaving. (This is the path that hit problems in the VM context but not in the bare-metal context.)
  • Isolation. OC’s reach into all four production IncusOS hosts means the management-plane host is itself security-sensitive. A dedicated box gives a clean fault domain — losing the management plane doesn’t lose any production capacity, and vice versa.
  • No bootstrap inversions. Doesn’t depend on ibuypwr (or any other production host) being up to manage the rest of the fleet.
  • Hardware cost is modest. SFF PC suitable for OC is in the $200–400 range new, less used.

Decision log

DateDecisionReason
2026-05-07Test OC on a Debian VM (mimir) on ibuypwrQuickest path to validate the standalone-Incus-management hypothesis
2026-05-07Abandon Debian-VM approachDaemon binary not in releases — would need source builds, drifting from upstream pattern
2026-05-07Test IncusOS-on-Incus VM on ibuypwrBlessed pattern attempt without new hardware
2026-05-07Abandon IncusOS-on-Incus approachHit install-seed plumbing fight against CD-ROM-typed boot media; no local FAT/GPT tooling
2026-05-07Commit to dedicated SFF PC + IncusOSRight tool, right pattern, manageable cost
2026-05-09Re-read upstream OC docsConceptual corrections incorporated (server vs cluster, two CLIs, token bootstrap, OC’s own trust model). SFF PC plan stands; left a note in the IncusOS-on-Incus rejection section that a deliberate retest is the only way to revisit that decision.