ansi key rotation

This document describes how to rotate the operator’s Ansible SSH automation keypair across every VM/container in the fleet. Two flavors of “rotation” are covered:

Operator handoff / hygiene (frequent, low-risk): bring a new admin’s pubkey alongside the existing one, or retire an operator who’s leaving. The fleet’s existing ansi user identity doesn’t change; only the set of pubkeys in its authorized_keys does. Multiple pubkeys can coexist indefinitely — /configs/ansible/*.pub is glob-slurped wherever it’s read.
Key replacement for compromise (rare, higher-risk): the ansible_ed25519 private key may be exposed; rotate fast and remove the old pubkey from every VM.

The mechanism is the same in both cases: /configs/ansible/*.pub is the single source of truth, and rotation is just editing that directory and re-running the sync. Drop a .pub in to add a key fleet-wide; delete a .pub to prune it. The flavors differ only in whether parallel validity of old+new is acceptable while you transition.

Status

The declarative tooling (make sync-ansible-ssh-keys, added 2026-06-11) is live, and adding a key has been exercised on this fleet — the cicd_ed25519.pub was added 2026-05-30 for the dedicated CICD runner identity (sop-cicd-identity-bootstrap.md, operator note) and propagated to existing hosts via the init role’s authorized-keys refresh. Since corrected: the cicd keypair was later relocated to /configs/ansible/cicd/ — deliberately OUTSIDE the *.pub glob — so it is no longer part of this SOP’s key set. What has not yet happened on this fleet is a full credential replacement (the compromise path: prune the old key fleet-wide). The original /configs/ansible/ansible_ed25519{,.pub} has been in continuous use since IaC bring-up. Treat the removal/prune half of this SOP as written-ahead-of-first-use.

The canonical mechanism: `make sync-ansible-ssh-keys`

The whole rotation now runs through one op (ansible/playbooks/ops/sync-ansible-ssh-keys.yaml, invoked from infra/):

make sync-ansible-ssh-keys                          # enforce /configs/ansible/*.pub fleet-wide
make sync-ansible-ssh-keys EXTRA="--check --diff"   # preview each host's authorized_keys delta, change nothing

It reconciles every non-IncusOS host’s ansi (and service-user) authorized_keys to exactly the keys in /configs/ansible/*.pub — the same directory OpenTofu’s cloud-init slurps into new VMs (opentofu/system_cloud-init.tf). One source of truth, two consumers: tofu seeds new VMs at birth, this op reconciles existing ones on demand.

The sync is exclusive (ansible.posix.authorized_key exclusive=true). That’s what makes “delete a .pub from the dir, re-run = key removed from every host” work — an add-only push could never retire a key. Exclusivity is also why the op carries two localhost pre-flight guards that keep it from locking the fleet out:

Refuse an empty dir. If /configs/ansible/*.pub matched nothing, an exclusive sync would strip every authorized_keys and lock everyone out. The op aborts instead.
Refuse to prune the key the run connects with. The fingerprint of ssh_active_private_key (default /configs/ansible/ansible_ed25519, i.e. ansible.cfg’s key) must be present among the dir’s pubkeys. This stops the classic “deleted the old .pub while ansible.cfg still points at the old key” self-lockout. To prune the old key you must first connect with the new one.

Override the active key when you run with a non-default --private-key: make sync-ansible-ssh-keys EXTRA="-e ssh_active_private_key=/configs/ansible/ansible_ed25519_new".

Guard #2 is the mechanism that structurally enforces the safe add-then-remove ordering below — you physically cannot get to a partial-lockout state through this op.

What gets touched

The pubkey flows into VMs via two independent paths, both of which the rotation addresses:

Path	Where it’s read	Mutation surface	Retroactive?
Cloud-init at VM creation	`opentofu/system_cloud-init.tf:5-8` (`fileset("/configs/ansible", "*.pub")`)	`/configs/ansible/*.pub` (operator-side)	No — one-shot at first boot. New VMs created after the rotation pick up whatever’s in `/configs/ansible/` at that moment; existing VMs are unaffected.
Live VM authorized_keys	`~ansi/.ssh/authorized_keys` (and mirrored service users) on every fleet VM	`make sync-ansible-ssh-keys` (exclusive add and prune); the init role’s `authorized-keys` tag (add-only refresh)	Yes — runs against the current fleet.

The live-state mutation surface has two tools with different powers — know which you need:

make sync-ansible-ssh-keys — exclusive reconcile. The only mechanism that can prune. Use it for both halves of a rotation (add and remove).
The init role’s authorized-keys tag (roles/init/tasks/main.yaml, runs on every make up/init) — add-only. Its “Ensure all /configs/ansible/*.pub are in ansi’s authorized_keys” task (lines 123-131) adds every dir pubkey to ansi with state=present (not exclusive); “Mirror ansi’s authorized_keys to service users” (lines 133-144) copies the result onto each service user; “Append per-user authorized keys” (lines 151-159) layers on per-user extras. Adding a key will therefore propagate on the next ordinary init run too — but init can never remove a key, so prunes always require the sync op.

And the consumers / referencing config:

/configs/ansible/ansible_ed25519 — private half. Referenced by ansible/ansible.cfg:11’s private_key_file =. Operator’s own laptop typically also holds a copy (since the workspace mount comes from the host).
/configs/ansible/ansible_ed25519.pub — public half. Glob-matched by cloud-init, the init role, and the sync op.
ansible.cfg private_key_file — only needs editing if the rotation changes the filename. Rotate-in-place (same filename, new contents) needs no cfg edit. Note that switching ansible.cfg to the new key is also what satisfies the sync op’s guard #2 before you prune the old key.
Per-VM service-user authorized_keys — every init_service_users entry has its authorized_keys reconciled to (dir keys ∪ that user's declared extras). Today only moby on the docker hosts qualifies (defined in group_vars/docker_servers.yaml); no other group declares init_service_users. The sync op handles service users in the same pass as ansi, so there’s no separate per-play step to run.
.gitignore — keeps *ed25519* excluded; new keypair filenames must contain ed25519 (or you risk accidental commit).

The backup user is NOT in scope. On backup hosts, ~backup/.ssh/authorized_keys is a borg forced-command allowlist, not an init_service_users mirror — the sync op and init role both leave it untouched. Don’t expect the ansi key to authenticate as backup, and don’t add it there as part of this rotation.

Out of scope — the five IncusOS hosts (kyojin, theia, hyperion, atomsk, ibuypwr). They have no ansi user; Ansible reaches them via incus exec (see the incus_servers plays in setup-*.yaml), and incus exec authenticates via the Incus client cert at /configs/incus/, not via SSH. Both the sync op and the init role scope themselves all:!incus_servers for exactly this reason. The Incus client cert is rotated separately (incus-cert-rotation) and also functions as the emergency backdoor if SSH access to every VM is lost — see Procedure: recovery.

Enumerate the live fleet that gets touched:

# Every VM/container the rotation must push to (anything in inventory
# that's NOT an IncusOS host):
ansible all:!incus_servers --list-hosts | tail -n +2 | wc -l
ansible all:!incus_servers --list-hosts

Strategy: add-then-remove (the only safe path, and what the tooling enforces)

Both old and new pubkeys are present in every VM’s ~ansi/.ssh/authorized_keys simultaneously; the old is removed only after the new is confirmed working across the whole fleet. There’s never a window where Ansible can’t authenticate.

1. generate new keypair                     ← new private key exists on operator workstation
2. add new .pub to /configs/ansible/        ← cloud-init for NEW VMs immediately picks up
                                              both keys; existing VMs unaffected so far
3. make sync-ansible-ssh-keys               ← both keys now present on every live VM
                                              (ansi + service users), in one pass
4. distribute new private key to operators  ← out-of-band
5. verify new key authenticates everywhere  ← prove it works before retiring the old
6. point ansible.cfg at new key             ← if new filename; also satisfies guard #2
                                              for the pruning sync in step 8
7. remove old .pub from /configs/ansible/   ← future VMs no longer get the old key
8. make sync-ansible-ssh-keys               ← exclusive sync prunes the old key fleet-wide
9. verify old key no longer authenticates
10. retire old private key on disk

If anything goes wrong between steps 2 and 8, both keys still work everywhere, so you can revert by deleting the new .pub and re-running the sync.

Note there is no separate “atomic replace” strategy any more. The old SOP documented a one-pass exclusive=true replace that wiped the old key while adding the new — its failure mode was locking yourself out of hosts that lost the old key before the new one landed. The sync op’s guard #2 makes that footgun unreachable: it refuses to prune the key you’re connecting with, so the only way to retire the old key is to first connect with the new one (steps 5-6 above). For a compromise rotation you still follow add-then-remove — just move fast through steps 3-8. The old key remains valid only for the (short) window between the two syncs, which you control.

Pre-rotation checklist

Old private key works against every fleet VM. Verify:
Terminal window
```
ansible all:!incus_servers -m ping
```
Don’t rotate from a broken state — fix the unreachables first. An UNREACHABLE host won’t get synced, so it’d be left on the old keyset.
Dry-run the sync to see the current fleet delta (and confirm the guards pass) before touching anything:
Terminal window
```
make sync-ansible-ssh-keys EXTRA="--check --diff"
```

Snapshot current authorized_keys on a representative VM for rollback reference:

ansible <one-vm-per-group> -m shell -a 'cat ~/.ssh/authorized_keys' > /tmp/pre-rotation-authorized_keys.txt

Confirm Incus client cert at /configs/incus/ is currently trusted on all 5 IncusOS hosts (the recovery backdoor). Verify:

for r in kyojin theia hyperion atomsk ibuypwr; do
  echo -n "$r: "; incus query $r:/1.0 | jq -r .auth
done   # expect all five: trusted

If rotating for compromise: also plan to audit recent activity on the controller and any host the key could have been used from. Key rotation alone doesn’t undo whatever the compromised key already did.
Decide on filename: rotate in place (keep ansible_ed25519, replace contents — no ansible.cfg edit needed) or new filename (e.g. ansible_ed25519_2026, update ansible.cfg). In-place is simpler; named-by-year is more auditable. Either way, the pruning sync (step 8) requires that the key ansible.cfg/--private-key connects with is still in the dir at that moment.

Procedure (add-then-remove)

1. Generate a new keypair

# On the operator workstation (NOT in /workspace — accidental git add risk):
ssh-keygen -t ed25519 -a 100 -C "ansible@rrchnm $(date +%Y-%m-%d)" -f ~/ansible_ed25519_new

Outputs ~/ansible_ed25519_new (private, mode 0600) and ~/ansible_ed25519_new.pub. The -a 100 raises the KDF rounds for the on-disk encryption (mild defense against offline cracking if the file’s ever exfil’d). Don’t commit either half.

2. Add the new pubkey alongside the old

Drop the new .pub (only) into /configs/ansible/:

cp ~/ansible_ed25519_new.pub /configs/ansible/ansible_ed25519_new.pub
ls /configs/ansible/*.pub   # should show both: ansible_ed25519.pub + ansible_ed25519_new.pub

From this point, any new VM created via tofu apply will have both keys in ~ansi/.ssh/authorized_keys at first boot. Existing VMs are unaffected until step 3.

3. Sync the new pubkey onto every live VM

make sync-ansible-ssh-keys EXTRA="--check --diff"   # preview: expect every host to gain the new key
make sync-ansible-ssh-keys                          # apply

One pass handles both ansi and the mirrored service users (moby on docker hosts today). The add is non-destructive — the old key stays in place, since the dir still contains both .pub files. Watch the recap for any unreachable or failed host — those still have only the old key and must be reached before step 8. Re-run once they’re back; the op is idempotent.

(Equivalently, the next ordinary make up/init run would add the new key too, since the init role’s authorized-keys refresh is add-only. But run the sync explicitly so you’re not waiting on an unrelated deploy — and so the verification in step 5 reflects a known state.)

4. Distribute the new private key

Out-of-band — encrypted email, secure file share, in-person USB. Never via git, never via the workspace mount, never the same channel that delivered the old key if rotating for compromise.

Each operator saves it at their /configs/ansible/ansible_ed25519_new (mode 0600) — or wherever their ansible.cfg’s private_key_file points if they’ve customized it.

5. Verify

The conclusive test is that Ansible can run with only the new key configured:

# In a fresh shell — don't let ssh-agent silently fall back to the old key:
ssh-add -D                                   # purge any cached identities
SSH_AUTH_SOCK=                               # disable agent entirely
ansible all:!incus_servers -m ping \
  --private-key=/configs/ansible/ansible_ed25519_new

All hosts should return pong. Any Permission denied (publickey) failure means step 3 missed that host — re-sync before continuing.

For an extra check, also verify a service-user login works (those were reconciled in the same step 3 pass):

# pick one docker host (moby is the only mirrored service user today):
ssh -i /configs/ansible/ansible_ed25519_new moby@<docker-host> true && echo moby ok

6. Switch ansible.cfg to the new key

Required before step 8 unless you rotated in place. If you chose a new filename in the pre-rotation checklist (e.g. ansible_ed25519_2026), edit ansible/ansible.cfg:11:

private_key_file = /configs/ansible/ansible_ed25519
private_key_file = /configs/ansible/ansible_ed25519_2026

This isn’t just cosmetic: the sync op’s guard #2 refuses to prune the key the run connects with, so the pruning sync in step 8 will abort unless your active key (ansible.cfg’s private_key_file, or whatever you pass via -e ssh_active_private_key=/--private-key) is the new key whose .pub is still in the dir.

If rotating in place (same filename, new contents), ansible.cfg already points at the right path — just make sure the new private key’s bytes are the ones at /configs/ansible/ansible_ed25519 on the workstation running the sync.

7. Remove the old pubkey from `/configs/ansible/`

mv /configs/ansible/ansible_ed25519.pub /configs/ansible/ansible_ed25519.pub.retired

The .pub.retired rename keeps it visible-but-not-glob-matched in case you need to roll back step 8 (*.pub won’t match it). Or delete it outright. Tofu doesn’t need the file to exist for already-created VMs.

Future tofu apply creates of new VMs will now seed only the new key. Existing VMs still have both keys until step 8.

8. Sync the removal to every live VM

make sync-ansible-ssh-keys EXTRA="--check --diff"   # preview: expect every host to LOSE the old key
make sync-ansible-ssh-keys                          # apply the exclusive prune

Because the sync is exclusive, dropping the old .pub out of the dir is all it takes — the op reconciles every ansi (and service-user) authorized_keys down to exactly the remaining keys, pruning the old one fleet-wide in one pass. If guard #2 aborts here, you skipped step 6: your active key isn’t in the dir. Point ansible.cfg/--private-key/-e ssh_active_private_key= at the new key and re-run.

9. Verify the old key no longer authenticates

ssh-add -D; SSH_AUTH_SOCK= \
  ansible all:!incus_servers -m ping --private-key=/configs/ansible/ansible_ed25519.pub.retired
# (point --private-key at the retired PRIVATE key on your workstation, not the .pub)
# expect every host: Permission denied (publickey).

Every host should fail auth. If any host still accepts the old key, step 8 missed it (likely it was unreachable) — re-sync once it’s back.

10. Retire the old private key

If rotating for compromise: securely destroy every copy (shred -u the old private key on every workstation; coordinate with other operators to do the same). The retired .pub.retired is harmless but you can delete it now too.

If rotating for hygiene: archive the old keypair offline. Useful if someone ever needs to read backup material that was authenticated with it at the time.

Procedure: recovery (locked out of every VM)

If somehow Ansible can’t authenticate anywhere via SSH (e.g. an out-of-band edit bypassed the sync guards, or every host was unreachable during a prune and later came back on the wrong keyset):

The Incus client cert at /configs/incus/ is an independent identity that authenticates to the IncusOS daemons (not to VMs directly). From the workspace container, incus exec <remote>:<vm> runs commands inside the VM via the daemon’s pty channel — no SSH needed.

# From the workspace container, find the host:vm pairing for the locked-out VM:
incus list <remote>: | grep <vm>

# Drop a working pubkey back into ansi's authorized_keys:
NEW_PUB="$(cat /configs/ansible/ansible_ed25519_new.pub)"
incus exec <remote>:<vm> -- sh -c "echo '$NEW_PUB' >> /home/ansi/.ssh/authorized_keys && chown ansi:ansi /home/ansi/.ssh/authorized_keys"

# Then re-verify ssh works for that vm:
ssh -i /configs/ansible/ansible_ed25519_new ansi@<vm-ip> true && echo recovered

This works as long as the Incus client cert at /configs/incus/ is still trusted on the IncusOS host running the VM. If that’s also broken, see incus-cert-rotation’s recovery procedure — which uses console / --force-local on the IncusOS host itself. The keys-of-last-resort chain is: SSH key → Incus client cert → console / IPMI on the IncusOS bare-metal host.

Once ansi’s authorized_keys has a working key on enough hosts to authenticate, the regular make sync-ansible-ssh-keys reconciles the rest of the fleet back to the dir. For a full fleet lockout, iterate the incus exec repair over every VM in ansible all:!incus_servers --list-hosts first; there’s no scripted automation for the hand-repair — it’s a row-by-row job.

Verification commands

# Which pubkeys does /configs/ansible/ currently provide (the source of truth)?
ls -l /configs/ansible/*.pub

# Preview what the fleet would converge to WITHOUT changing anything:
make sync-ansible-ssh-keys EXTRA="--check --diff"

# Fingerprint of the local active private key:
ssh-keygen -lf /configs/ansible/ansible_ed25519.pub

# Live state on one VM (which pubkeys does ansi actually trust?):
ansible <one-vm> -m shell -a 'awk "{print \$1, \$2}" ~/.ssh/authorized_keys | ssh-keygen -lf -'

# Whole-fleet reachability under the current ansible.cfg key:
ansible all:!incus_servers -m ping

# Whole-fleet reachability under a SPECIFIC key (the no-false-positive check):
ssh-add -D; SSH_AUTH_SOCK=; ansible all:!incus_servers -m ping --private-key=/path/to/key

Don’t use ansible all -m ping for verification of the SSH key — it walks the IncusOS hosts too, which use incus exec and will appear to succeed regardless of SSH state. Always scope with all:!incus_servers.

Also don’t rely on agent-cached identities (SSH_AUTH_SOCK unset, ssh-add -D first) when verifying — the agent will silently try every loaded key and a “success” may be the old key still working, not the new one.

Rollback

If a step fails mid-rotation:

Between step 2 and step 7: both keys are present in /configs/ansible/. Roll back by deleting /configs/ansible/ansible_ed25519_new.pub and re-running make sync-ansible-ssh-keys — the exclusive reconcile removes the new key, and the old key remains trusted everywhere. The world is back to pre-rotation. (Make sure ansible.cfg is still pointing at the old key so guard #2 is satisfied — if you’d already done step 6, point it back first.)
Between step 7 and step 8: new pubkey is in /configs/ansible/, old is renamed .pub.retired, but you haven’t synced the prune yet. Live VMs still have both. Restore the old by mv-ing .pub.retired back to .pub — no sync needed; the world is back to dual-trust state.
Stuck after step 8 with the old key working on some VMs and not others (a host was unreachable during the prune): don’t try to undo. Just re-run make sync-ansible-ssh-keys once the stragglers are reachable — the exclusive reconcile converges them to the dir’s keyset. If a host is hard-down and you need access before the prune completes, use the recovery procedure to hand-place a working key.

The dangerous “exclusive replace locked me out mid-fleet” failure mode of the old manual procedure is gone: the sync op never prunes the key it’s riding on (guard #2) and never reconciles to an empty set (guard #1), so a partial run leaves the controller’s own key intact on every host it reached.

Special cases

Multi-operator steady state (adding a second pubkey, not replacing)

Same as add-then-remove, but stop after step 5 (verified working) and never run steps 7-10. The fleet now trusts both keys indefinitely. Each operator’s pubkey lives as a separate .pub file in /configs/ansible/ (filenames must contain ed25519 for the gitignore safety net). Cloud-init, the init role, and the sync op all glob *.pub, so any number of files just work. Each operator has their own private key on their workstation; nothing has to be coordinated beyond making sure every operator’s .pub lands in /configs/ansible/ and a sync is run.

Rotating only one operator’s pubkey (granular handoff)

Sub-case of the multi-operator pattern. Add the new operator’s pubkey (steps 2-3), then remove the departing operator’s .pub (steps 7-8). Because the sync reconciles to exactly the dir, removing one .pub prunes only that key; the other operators’ keys are untouched. Just make sure the key you run the sync with isn’t the one being removed (guard #2 will stop you if it is).

Filename change without content change (renaming the key file)

Cloud-init, init, and the sync op all glob *.pub, so a rename of the on-disk filename — same key bytes — has no effect on live state; every host already trusts that fingerprint. Only ansible/ansible.cfg:11’s private_key_file needs to point at the new path. Useful when reorganizing /configs/ansible/ but not actually rotating the cryptographic identity.

Existing VM created before the new key was added — needs the new key but the sync skipped it (unreachable)

Use the recovery procedure to push directly via incus exec. Once ansi’s authorized_keys has the new pubkey and the host is reachable again, a normal make sync-ansible-ssh-keys reconciles it (ansi + service users) the rest of the way.

Service user with extra per-user keys (e.g. a CI runner with its own keypair)

The sync op’s target for a service user is (dir keys ∪ that user's init_service_users[*].authorized_keys), so per-user extras survive the exclusive reconcile — they’re folded into the enforced set, not clobbered by it. Declare such extras on the authorized_keys field of the user’s init_service_users entry (see the schema comment in roles/init/tasks/main.yaml); today only moby in group_vars/docker_servers.yaml has the field (currently empty). The ansi rotation only changes the dir keys half; the extras half is owned by group_vars.