Skip to content
RRCHNM Infra

Incus client cert rotation

This document describes how to rotate the operator’s Incus client cert across the fleet’s IncusOS hosts. Two flavors are covered:

  • Add a new identity alongside the old (frequent, low-risk): bring a new operator onboard, or replace your client laptop without losing access. Both certs work in parallel until the old is retired.
  • Replace a compromised cert (rare, higher-risk): the old client.key may be exposed; rotate fast and revoke.

There is no central authority — each IncusOS host runs an independent Incus daemon with its own trust store (since the cluster was dissolved, see incus-declustering). The fleet currently has five daemons: kyojin, theia, hyperion, atomsk, ibuypwr. Every step that mutates trust runs per remote, five times.

There is no Ansible role or script that automates this end-to-end; the judgment calls (which cert to add, when to revoke the old, whether the install seed needs updating) stay manual and the commands are run by hand.

What gets touched

  • Server-side trust store on each of the 5 remotes — visible via incus config trust list <remote>:. The entry name (e.g. init, operator) is local-only metadata; the fingerprint is what authenticates.
  • Client-side identity at /configs/incus/client.{crt,key} (mounted into the workspace container from the host). The private key never leaves wherever it was generated.
  • Pinned server certs at /configs/incus/servercerts/<remote>.crt — populated by incus remote add for each remote. These pin the server’s identity by fingerprint, independent of TLS SAN.
  • Install seed cert — passed to playbooks/incus/ops/factory-reset.yaml at runtime as -e input_incus_trusted_cert=.... Whatever cert is supplied there becomes the initial trusted client on a fresh factory-reset.

Run incus config trust list <remote>: against each remote to see the live state.

Avoids any window where you can’t authenticate. Both old and new certs work in parallel on every remote until you confirm the new one and then revoke the old.

1. generate new identity (token flow from one already-trusted remote)
2. propagate new cert to the other 4 remotes (add-certificate, using old cert)
3. update the install seed reference to the new cert
4. verify the new identity authenticates to all 5 remotes
5. revoke old cert from each remote (5×)
6. retire old client.{crt,key} on disk

If anything goes wrong between 1 and 5, the old cert still works on at least some remotes — you can revert by skipping the revoke step.

Strategy: atomic replace (faster, riskier)

Use only when client.key is known-compromised and you need to cut the old cert immediately. Bootstrap the new identity per-remote via console-issued tokens (one per remote, requires shell access to each IncusOS host), then revoke the old cert from each in one pass.

1. console into each remote (or use --force-local on the daemon host)
2. per remote: incus config trust add operator --force-local → token → consume from new client
3. revoke old cert from each remote
4. retire old client.{crt,key}

Don’t do this unless you have to — needing console/SSH access to five hosts is more painful than the add-then-revoke procedure’s single re-use of the old cert.

Pre-rotation checklist

  • Old client.{crt,key} accessible — either still at /configs/incus/ or available from a host-side backup. Without it, you’re in the “lost old cert” recovery flow below, not add-then-revoke.
  • All 5 remotes currently reachable: for r in kyojin theia hyperion atomsk ibuypwr; do incus info $r: >/dev/null && echo "$r OK"; done
  • Snapshot current trust state: for r in kyojin theia hyperion atomsk ibuypwr; do echo "=== $r ==="; incus config trust list $r:; done
  • Decide whether to update the install seed (input_incus_trusted_cert) — needed if you want post-factory-reset hosts to trust the new cert without manual re-add.
  • If rotating for compromise: revoke immediately upstream of any other action (no parallel window).

Procedure: add-then-revoke

1. Generate new identity via token

Pick one remote that currently trusts your old cert. Issue a trust token; that token is one-shot and expires after consumption.

Terminal window
incus config trust add kyojin:operator
# Server prints a base64 token. Copy the entire string.

Set up a fresh INCUS_CONF for the new identity (don’t overwrite /configs/incus/ until you’ve verified the new cert works). Consume the token:

Terminal window
mkdir -p /configs/incus-new
INCUS_CONF=/configs/incus-new incus remote add kyojin <PASTE_TOKEN>

This generates /configs/incus-new/client.{crt,key} (fresh keypair, never leaves disk) and registers the public half with kyojin as operator. Verify:

Terminal window
INCUS_CONF=/configs/incus-new incus info kyojin: >/dev/null && echo "new identity authenticates to kyojin"

2. Propagate the new cert to the other 4 remotes

The new identity is only trusted on kyojin. To trust it on theia/hyperion/atomsk/ibuypwr without separate console access, push the cert from your old still-trusted client:

Terminal window
# Using the existing /configs/incus (old cert, still trusted on all 4 remotes):
for r in theia hyperion atomsk ibuypwr; do
  incus config trust add-certificate $r: /configs/incus-new/client.crt --name operator
done

add-certificate accepts a .crt file and registers it as trusted on the target remote. No token, no separate auth handshake — it works because the old cert is already trusted on those remotes.

Argument order is [<remote>:] <cert.crt> — remote first, then cert. Putting the remote at the end (after the cert path or after --name) causes the CLI to drop back to the default-remote (which is local/unix and unreachable from the workspace container), producing a misleading “This client hasn’t been configured to use a remote server yet” error.

Add those remotes to the new INCUS_CONF too so the new client knows where they are:

Terminal window
for r in theia hyperion atomsk ibuypwr; do
  INCUS_CONF=/configs/incus-new incus remote add $r https://$r.rrchnm.internal:8443
done

The first incus remote add to a new server fingerprint-pins it; you may need to confirm the fingerprint interactively or supply --accept-certificate.

3. Update the install seed reference

If a host gets factory-reset after rotation, IncusOS will only trust whatever cert was in input_incus_trusted_cert at reset time. Update wherever you store that runtime extra-var (operator workstation, password manager, vars file outside the repo) to point at the new client.crt. The cert is deliberately not committed (see ansible/playbooks/incus/ops/factory-reset.yaml for the assert).

If you’re in mid-rotation and you also need to factory-reset a host before revoke is done, you can pass both certs in a preseed.certificates list — IncusOS supports multiple trusted clients in the seed.

4. Verify the new identity end-to-end

The reliable test is the auth field on /1.0 — trusted means the cert authenticated, untrusted means it didn’t:

Terminal window
for r in kyojin theia hyperion atomsk ibuypwr; do
  echo -n "$r: "
  INCUS_CONF=/configs/incus-new incus query $r:/1.0 | jq -r .auth
done

All 5 should print trusted. If any prints untrusted, step 2 missed that remote or the cert push errored — fix before proceeding.

Don’t use incus info <remote>: as a verification test. Incus’s /1.0 GET endpoint serves a stripped public view to unauthenticated clients (api_extensions subset, auth: untrusted, no config block). incus info will succeed and print something even for a fully-revoked cert — it just prints less. The jq -r .auth check is unambiguous; incus info’s exit code is not.

5. Revoke the old cert from each remote

Look up the old cert’s fingerprint on each remote:

Terminal window
for r in kyojin theia hyperion atomsk ibuypwr; do
  echo "=== $r ==="
  INCUS_CONF=/configs/incus-new incus config trust list $r: | grep init
done

Then revoke:

Terminal window
for r in kyojin theia hyperion atomsk ibuypwr; do
  INCUS_CONF=/configs/incus-new incus config trust remove $r: <OLD_FINGERPRINT>
done

(Fingerprints may differ per remote if the old cert was added separately to each — list and check before bulk-removing.)

Confirm the revoke took effect by checking with the old cert’s INCUS_CONF — it should now report untrusted:

Terminal window
for r in kyojin theia hyperion atomsk ibuypwr; do
  echo -n "$r: "
  INCUS_CONF=/configs/incus incus query $r:/1.0 | jq -r .auth
done

All 5 should print untrusted. Admin commands against any remote with the old cert should also fail — but the failure mode is non-obvious:

  • incus config trust list <remote>: → Error: not authorized (clean error, server-side rejection)
  • incus list <remote>: → Error: The server is missing the required "api_filtering" API extension
  • incus storage list <remote>: → Error: The server is missing the required "storage" API extension

The “missing extension” errors are client-side refusals: the server advertises a reduced api_extensions list to unauthenticated clients (no api_filtering, no storage, etc.), and the CLI checks the list before sending the request. So an untrusted client never even attempts the privileged call — it errors out locally. These extension-missing errors look like server problems but are reliable signals that the cert is no longer authenticated.

6. Retire the old client files

Move /configs/incus-new/* into /configs/incus/ (replacing the old contents), or update whatever mount/config makes /configs/incus/ resolve to the new identity. Securely destroy or archive the old client.key per your compromise posture.

Procedure: recovery (you’ve lost the old client cert)

If client.key is gone and no host-side backup is recoverable, you can’t use add-certificate to push a new cert — that path requires being already trusted. You need console/SSH access to each IncusOS host and use --force-local against the local Unix socket:

Terminal window
# On each IncusOS host (5 times):
incus config trust add operator --force-local
# Take the printed token, consume from your new client:
INCUS_CONF=/configs/incus-new incus remote add <host> https://<host>.rrchnm.internal:8443 --token <TOKEN>

This bypasses TLS auth by using the daemon’s local socket directly — no client cert needed. Once one remote trusts the new cert, you’re back in the add-then-revoke flow for the remaining 4 (push via add-certificate).

The IncusOS console is accessible via incus console <host>: if you still have an Incus client cert on a workstation that can reach the host. If you’ve lost trust on every workstation, you’re down to physical console / IPMI.

Verification commands

Terminal window
# Per-remote trust list (look for old vs new fingerprints)
incus config trust list <remote>:


# Pending tokens (should be empty after a consume)
incus config trust list-tokens <remote>:


# Local cert fingerprint
openssl x509 -in /configs/incus/client.crt -fingerprint -sha256 -noout


# Is the current INCUS_CONF's cert authenticated on each remote?
# trusted = cert in trust store; untrusted = not (revoked, wrong cert, or never added)
for r in kyojin theia hyperion atomsk ibuypwr; do
  echo -n "$r: "
  incus query $r:/1.0 | jq -r .auth
done

Don’t use incus info <remote>: for auth verification — it returns the public /1.0 view to untrusted clients without erroring. Always check the auth field via incus query.

Rollback

If a step fails mid-rotation:

  • Between step 1 and 5: both certs are trusted on at least kyojin (always) and progressively on the others. Set INCUS_CONF=/configs/incus (old cert) to roll back to the pre-rotation identity. The new trust entries are harmless to leave in place; you can remove them later if you decide not to retry.
  • Between step 5 and 6: old cert is revoked from one or more remotes but new cert is verified working. Continue forward, don’t try to roll back — re-adding the old cert means handing the old (potentially compromised) public key back to those servers, which defeats the rotation purpose.
  • Stuck with new cert unverified on some remote: that remote’s trust store still has the old cert from before rotation started; INCUS_CONF=/configs/incus keeps working there.

The danger zone is “old cert revoked, new cert never propagated to one remote, no console access to that host.” Avoid by completing step 2 in full before any step-5 revoke.

Special cases

Rotating only one remote’s trust set without touching the others

incus config trust add-certificate and incus config trust remove are per-remote operations. There’s nothing structural preventing per-host divergence — useful if you’re decommissioning a single host or onboarding a host-specific operator. Just be explicit about the per-remote trust state and document it (cluster-wide-identical trust is convention, not a constraint).

Adding a second operator (instead of replacing)

Same as add-then-revoke, but skip step 5 and 6. Each operator has their own client.{crt,key} at their own /configs/incus/. Server-side: each gets a distinct trust entry (e.g. operator-alice, operator-bob). All operators have full admin rights — there’s no project-scoped restriction by default. Use --restricted --projects <list> on incus config trust add if you need project isolation.

Server cert SAN mismatch on a fresh remote

After factory-reset, IncusOS regenerates its server cert. The SAN list excludes the LAN IP (see incus-factory-reset). The first incus remote add after reset must use the hostname URL (https://<host>.rrchnm.internal:8443), not the IP literal — Go’s TLS stack enforces SAN on the unpinned first connection. After remote-add, the cert is pinned by fingerprint in servercerts/<host>.crt and SAN content is irrelevant.

Differentiating per-cert vs per-token issuance

incus config trust add <name> issues a one-shot token (server expects token consumption to register a new keypair generated client-side). incus config trust add-certificate <cert> registers an already-existing public cert directly (no token, no client-side keygen). Use the first when bootstrapping a fresh identity from scratch; use the second when propagating an existing identity to additional remotes.