CICD identity

2026-06-07 — the Incus credential was REMOVED. cicd is now a two-credential identity (age + SSH). The runner is websites-only CI (SSH + SOPS); infra provisioning (make plan/-j infra/tofu/incus) is operator-run, so the runner has no Incus job. The cicd-runner trust was revoked on all 4 daemons, /configs/incus/cicd/ deleted, and incus-cert dropped from cicd_seed_sources. Phase C below (and every other Incus mention) is obsolete — do not run it. See plan-agent-secret-isolation.md.

The cicd runners (containers on nidavellir — GitHub runners brokkr/sindri plus the Forgejo runner eitri, all on the host’s rootless docker daemon) need two credentials to do their job:

Credential	What it does	Earlier mistake
age private key	decrypts SOPS files during playbook runs (cicd is a recipient of `vars/{slack,websites-lamp}` only)	original setup-cicd-servers.yaml seeded the operator’s `/configs/sops/age/keys.txt` into the runner. Runner compromise = whole secret tree leaked as operator.
SSH private key	SSHes to fleet hosts to run ansible	original seeded the operator’s `/configs/ansible/` (ansi key). Runner compromise = root access to every host as operator.
~~Incus client cert~~	~~talks to Incus daemons for `tofu apply` etc.~~	REMOVED 2026-06-07 — cicd no longer touches Incus (websites-only); trust revoked on all 4 daemons, files deleted.

The 2026-05-30 bootstrap rotated CICD onto its own dedicated identity, giving it the minimum scope it needs without sharing any operator identity. This SOP documents the procedure for both the initial bootstrap and any future rotation.

Status

Reference procedure. Initial bootstrap landed 2026-05-30 (see changelog.md); the Incus credential was retired 2026-06-07. The CICD credentials live under operator-side mount paths that are SEPARATE from the operator’s own:

/configs/sops/cicd-age/keys.txt                 ← cicd age private key
/configs/ansible/cicd/cicd_ed25519{,.pub}       ← cicd SSH keypair (the cicd/ SUBDIR — see below)
(/configs/incus/cicd/ — REMOVED 2026-06-07; cicd no longer uses Incus)

Location corrected 2026-09: the SSH keypair was relocated into the /configs/ansible/cicd/ subdir — deliberately OUTSIDE the fleet-wide /configs/ansible/*.pub glob (which is non-recursive), so cloud-init, the init role, and sync-ansible-ssh-keys never see it. The key is distributed only to inventory group cicd_target_servers. Phase B below records the ORIGINAL glob-based distribution; it no longer applies.

The operator’s personal credentials at /configs/sops/age/keys.txt and /configs/ansible/ansible_ed25519 are NEVER seeded into the runner.

What gets touched

Surface	File(s)	Mutation	Notes
age recipient list	`ansible/.sops.yaml`, `k0s/.sops.yaml`	Add `cicd` as 3rd recipient anchor + reference in every `age:` list	Co-equal with `init` + `operator`. (A 4th anchor, `agent`, has since been defined — reserved, no rule uses it today.)
age data-key envelopes	every file matching the `.sops.yaml` `path_regex` rules	re-wrapped by `rotate-age-key.py updatekeys` to include the cicd recipient	values not changed; only the envelope around each file’s data key
cloud-init pubkey slurp	`opentofu/system_cloud-init.tf`	~~picks up `cicd_ed25519.pub` automatically~~ SUPERSEDED — the key moved to the `cicd/` subdir, invisible to the glob	distribution is now scoped to `cicd_target_servers` only
existing-VM authorized_keys	`ansible/roles/init/tasks/main.yaml`	new “Ensure all /configs/ansible/*.pub are in ansi’s authorized_keys” task added before the “Mirror to service users” task	refreshes the keylist on every init-role run; idempotent; first user is this cicd rotation but covers any future operator-key add-or-rotate too
Incus trust store	every Incus daemon (kyojin / theia / hyperion / ibuypwr)	`incus config trust add-certificate` adds cicd cert as a `client`-type trusted identity named `cicd-runner`	non-destructive (operator’s existing identity stays trusted)
cicd-runner secret volumes	named docker volumes on ptah (`sops-age` / `ansible-ssh` / `incus-cert`)	`setup-cicd-servers.yaml` Play 3 tmpfs-stages the cicd-specific files from `/configs/.../cicd*` and `cp`s into the volumes	the moby user owns the volumes via rootless docker; only the cicd-runner container reads them

Out of scope — github_token_manage_runners (separate concern, lives in group_vars/all.sops.yaml and is independently rotatable via the SOPS flow), the inner-runner container image, the github actions registration itself.

Bootstrap procedure (one-time, first identity install)

Pre-flight: make cicd-servers is wired to nidavellir (the cicd_servers host; TAGS=runtime converges all three runners), make docker has been run (rootless docker daemon under moby owner), make add-trusted-cert exists but is interactive — we use raw incus config trust add-certificate below because it’s easier to script.

Phase A — age key

mkdir -p /configs/sops/cicd-age && chmod 700 /configs/sops/cicd-age
./scripts/rotate-age-key.py keygen --out /configs/sops/cicd-age/keys.txt
# Capture the printed "public key: age1..." — you'll paste it next.

Edit ansible/.sops.yaml and k0s/.sops.yaml — add a third recipient anchor under keys::

keys:
  - &init     age104nvh3yfxk9r9x3ule38py7y32q4r7xm79ak7zlwjf4ymxuknv9suncwea
  - &operator age10etyhxdpdw9yyll73qk78us80y3fhjlfezn9kdnj64efzm62t4ds27jcpk
  - &cicd     age1g4rsxsk8w0znert63f5vhyyc9ghe4e0aear9ryaa3ccxcg654anslw9d0p   # ← new

Then append - *cicd to every creation_rules[*].age: list (3 in ansible/.sops.yaml, 1 in k0s/.sops.yaml).

Re-wrap data keys + verify:

./scripts/rotate-age-key.py updatekeys
./scripts/rotate-age-key.py verify                                       # operator key still works
SOPS_AGE_KEY_FILE=/configs/sops/cicd-age/keys.txt \
  ./scripts/rotate-age-key.py verify                                     # cicd key also works

Phase B — SSH keypair (distribution model since revised — see the Status note)

ssh-keygen -t ed25519 -N '' -C 'cicd-runner' -f /configs/ansible/cicd/cicd_ed25519

The matching .pub lands in the same dir, where:

New VMs auto-pick it up via cloud-init’s fileset("/configs/ansible", "*.pub") slurp into ansi’s authorized_keys. Zero playbook action needed.
Existing VMs need an active refresh — push to every host’s /home/ansi/.ssh/authorized_keys via ansible ad-hoc:

CICD_PUBKEY=$(cat /configs/ansible/cicd_ed25519.pub)
ANSIBLE_CONFIG=ansible/ansible.cfg SOPS_AGE_KEY_FILE=/configs/sops/age/keys.txt \
  ansible -i ansible/inventory.yaml incus_instances \
    -m ansible.posix.authorized_key \
    -a "user=ansi key='$CICD_PUBKEY' state=present"

The init role’s authorized-keys tag (ansible-playbook playbooks/setup-X-servers.yaml --tags authorized-keys) now picks up ALL /configs/ansible/*.pub files on subsequent runs — the cicd key re-asserts itself idempotently. This belt-and-suspenders covers the operator-key add-or-rotate case too.

Phase C — Incus client cert (REMOVED 2026-06-07 — do not run)

Obsolete. cicd no longer uses Incus (websites-only CI). The trust was revoked on all 4 daemons and /configs/incus/cicd/ deleted. This phase is kept only as historical record of how the (now-removed) Incus identity was set up.

mkdir -p /configs/incus/cicd && chmod 700 /configs/incus/cicd

cd /configs/incus/cicd
openssl req -x509 -newkey rsa:2048 \
  -keyout client.key -out client.crt \
  -days 3650 -nodes -subj "/CN=cicd@workspace"
chmod 600 client.key

# Trust the cert on every Incus daemon (using the OPERATOR's existing
# trust to authenticate the add request):
for h in kyojin theia hyperion ibuypwr; do
  INCUS_CONF=/configs/incus incus config trust add-certificate $h: \
    /configs/incus/cicd/client.crt \
    --name cicd-runner \
    --description "cicd-runner identity (generated $(date +%Y-%m-%d))" \
    --type client
done

# cicd needs the operator's server-cert pinning + remote definitions:
cp /configs/incus/config.yml /configs/incus/cicd/config.yml
cp -r /configs/incus/servercerts /configs/incus/cicd/

# Sanity-check the cert works:
INCUS_CONF=/configs/incus/cicd incus list kyojin: -c n | head

Phase D — refresh cicd-runner volumes

make cicd-servers

Play 3’s tmpfs-staging-then-volume-cp pipeline picks up the cicd-specific files from /configs/sops/cicd-age/, /configs/ansible/cicd_ed25519{,.pub}, and /configs/incus/cicd/ and populates the three named volumes (sops-age, ansible-ssh, incus-cert). The volumes are wiped + repopulated each run, so re-running after a rotation auto-rolls the runner.

Verify (end-to-end)

# Volume contents are cicd-only:
ssh -i /configs/ansible/cicd/cicd_ed25519 ansi@<nidavellir-ip> \
  'sudo -u moby DOCKER_HOST=unix:///run/user/1001/docker.sock \
   docker run --rm -v sops-age:/x -v ansible-ssh:/y -v incus-cert:/z busybox \
   sh -c "ls /x; echo ---; ls /y; echo ---; ls /z"'

# Expected:
#   /x: keys.txt                                      (cicd age key only)
#   /y: cicd_ed25519, cicd_ed25519.pub                (cicd ssh only)
#   /z: client.crt, client.key, config.yml, servercerts  (cicd incus only)

Operator’s keys MUST NOT appear in any of these listings. If they do, cicd_seed_sources in setup-cicd-servers.yaml is still pointing at the wrong paths — check Phase D’s invocation.

Rotation procedure

Same shape as the bootstrap, with two differences:

age: regenerate /configs/sops/cicd-age/keys.txt (use --force or rm first), update the &cicd anchor pubkey in BOTH .sops.yaml configs (sed-replace works), updatekeys, verify, then make cicd-servers --tags runtime to refresh the sops-age volume.
ssh + incus: regenerate the keys/certs in the same paths, make cicd-servers --tags runtime to refresh the ansible-ssh + incus-cert volumes. The cicd ssh pubkey rotation also needs the ansible ad-hoc broadcast (Phase B) to push the new pubkey to existing hosts. The Incus rotation also needs revoking the OLD trust entry on each daemon (incus config trust remove <host>: <old-fingerprint>).

Incident response — exposed private key

If a cicd private key ends up in a log, terminal scrollback, screen recording, or any other channel:

Rotate the affected credential per the rotation procedure above.
Old key remains valid on disk until you remove it from the recipient list. updatekeys re-wraps to the new recipient set, so any sops -d from then on will only succeed with one of the current recipients’ keys. The compromised file on disk can be shreded if it’s still around.
For Incus: also revoke the old cert from each daemon’s trust store, otherwise the compromised key can still authenticate.
Document the incident in changelog.md.

Worked example: the 2026-05-30 cicd-bootstrap session itself produced the first incident — a verbose cat /sops-age/keys.txt | head -3 inadvertently dumped the cicd age private key to terminal output. Rotation took ~2 min (regen + sed-replace pubkey + updatekeys + make cicd-servers --tags runtime); old key now on zero recipient lists + not in any volume.