SOPS key rotation

This document describes how to rotate the age key(s) that decrypt SOPS-encrypted files in this repo. Two flavors of “rotation” are covered:

Recipient management (frequent, low-risk): adding a new operator’s age key to the recipient list, or removing an operator who’s leaving. The encrypted data doesn’t change; only the list of who can decrypt the per-file data-key envelope changes. Multiple recipients can coexist indefinitely — the multi-recipient steady state (today: init + operator + cicd in the ansible config, plus a reserved agent anchor; init + operator in the k0s config) is the normal mode.
Key replacement (rare, higher-risk): the old key is compromised or you want a fresh one. Generate new, re-encrypt to it, retire the old.

The mechanism is the same; the operational shape differs. The bulk-mechanical work is automated by scripts/rotate-age-key.py. The judgment calls (which key to add, when to retire the old, who gets the private key) stay manual.

Status

Reference procedure. The multi-recipient steady state was reached early 2026-05 via the Adding the second operator from scratch flow, and has since grown: ansible/.sops.yaml carries four anchors (init, operator, cicd — active on a subset of rules — and agent, reserved with no rule using it); k0s/.sops.yaml remains two-recipient (init + operator). No full key replacement has been performed; all keys are considered uncompromised. ./scripts/rotate-age-key.py list is the live roster.

What gets touched

Surface	File(s)	Mutation	Consumers
Recipient declaration	`ansible/.sops.yaml` (6 `creation_rules`), `k0s/.sops.yaml` (1 rule)	Edit the `age:` list under each `creation_rules` entry — recipients are YAML anchors at the top, referenced via `init` / `operator` / `*cicd`	`sops` CLI reads these to know who to encrypt new files for and to wrap data keys for during `updatekeys`
Encrypted file data-key envelopes	every file matching the `path_regex` patterns — today: `ansible/host_vars/.sops.yaml` (`armin`/`groudon`/`kyogre`; field-level regex), `ansible/group_vars/.sops.yaml` (incl. `all.sops.yaml`, `step_ca_servers.sops.yaml`), the `ansible/vars/` rules, and `k0s/platform/secrets/.sops.yaml` (whole-file — `openbao-keys`, `initial-kv-seeds`). `rotate-age-key.py list` enumerates live	The per-file metadata block (data-key envelope, encrypted to each recipient) gets re-wrapped by `updatekeys`. The actual encrypted secret values do not change.	`sops -d` / community.sops vars plugin / explicit `lookup('pipe', 'sops -d …')` in playbooks
Operator private key	`/configs/sops/age/keys.txt` (or wherever each operator’s `SOPS_AGE_KEY_FILE` env var points)	Replaced if the operator’s identity is rotated; appended if multi-recipient	sops decrypt path for every operator action

Enumerate the live state — these are the source of truth for “what .sops.yaml files exist and which encrypted files inherit them”:

./scripts/rotate-age-key.py list      # who's a recipient where; what files match each rule
./scripts/rotate-age-key.py verify    # every encrypted file decrypts with the configured key

Out of scope — secrets stored outside SOPS (Ansible SSH key at /configs/ansible/, Incus client cert at /configs/incus/, IncusOS pool recovery keys at /configs/incus/seeds/). Those have their own rotation SOPs (sop-ansible-ssh-key-rotation.md, sop-incus-trusted-cert-rotation.md, …); SOPS recipient rotation does not touch them.

Strategy: add-then-remove (recommended)

Avoids any window where decryption is broken. Both old and new recipients work in parallel until you confirm the new one and then remove the old.

1. generate new key                  ← new private key exists, public key in hand
2. add new recipient to all .sops.yaml configs (alongside old)
3. updatekeys                        ← every encrypted file's data-key envelope
                                       now wraps for both recipients
4. distribute new private key to operators
5. verify with the new key           ← prove it works
6. remove old recipient from .sops.yaml
7. updatekeys                        ← strip old recipient from data-key envelope
8. retire the old private key file   ← optional, depends on rotation reason

If anything goes wrong between 2 and 7, both keys still work, so you can revert.

Strategy: atomic swap (faster, riskier)

Use only when the old key is known-compromised and you can’t tolerate a second of overlap. Replace the recipient in one shot; if anything fails midway, you’re stuck with files encrypted to a recipient nobody has the private key for.

1. generate new key
2. distribute new private key
3. replace old recipient with new in all .sops.yaml configs
4. updatekeys                        ← old key can no longer decrypt
5. verify
6. retire old private key

Don’t do this unless you have to. The add-then-remove flow is only a few minutes longer.

Pre-rotation checklist

SOPS_AGE_KEY_FILE is set in your shell (or you’re invoking via the workspace-container default /configs/sops/age/keys.txt).
Terminal window
```
echo "$SOPS_AGE_KEY_FILE"; test -r "$SOPS_AGE_KEY_FILE" && echo ok
```
Everything currently decrypts. Don’t rotate from a broken state.
Terminal window
```
./scripts/rotate-age-key.py verify
```

Snapshot the current recipient list for rollback reference.

./scripts/rotate-age-key.py list | tee /tmp/sops-pre-rotation-recipients.txt

Decide strategy: add-then-remove (almost always) or atomic swap (compromise only). Document why.
If replacing for compromise reason, plan to also rotate the actual secret values (passwords, htpasswds, intermediate CA materials, etc.) afterward — recipient rotation alone doesn’t change what’s encrypted, only who can read it. The compromised key has presumably already been used to read all current values.
Confirm at least one operator can run updatekeys. That command needs a key from the CURRENT recipient set to decrypt the data-key envelope before re-wrapping; if no operator has a working old key, you can’t updatekeys and the rotation is stuck in the recovery procedure instead.

Procedure (add-then-remove)

1. Generate a new keypair

./scripts/rotate-age-key.py keygen --out new.key

Outputs the new private key file (mode 0600) and prints the public key. Don’t commit the private key.

2. Add the new public key to each `.sops.yaml`

Manual edit. Each .sops.yaml has a keys: anchor block at the top + one or more creation_rules blocks; each rule’s age: field references the anchors as a list. Add the new key as a third anchor and reference it from every rule:

Before (top of ansible/.sops.yaml / k0s/.sops.yaml):

keys:
  - &init     age104nvh3yfxk9r9x3ule38py7y32q4r7xm79ak7zlwjf4ymxuknv9suncwea
  - &operator age10etyhxdpdw9yyll73qk78us80y3fhjlfezn9kdnj64efzm62t4ds27jcpk

creation_rules:
  - path_regex: ...
    encrypted_regex: ...
    age:
      - *init
      - *operator

After:

keys:
  - &init     age104nvh3yfxk9r9x3ule38py7y32q4r7xm79ak7zlwjf4ymxuknv9suncwea
  - &operator age10etyhxdpdw9yyll73qk78us80y3fhjlfezn9kdnj64efzm62t4ds27jcpk
  - &newop    age1xxxnewkeyxxx...                                              # YYYY-MM-DD added for <reason>

creation_rules:
  - path_regex: ...
    encrypted_regex: ...
    age:
      - *init
      - *operator
      - *newop

Comments next to the anchor help future-you understand which key is which. Repeat for both .sops.yaml files — ./scripts/rotate-age-key.py list shows the live set. Missing one means files under that config can’t be decrypted by the new key.

3. Re-encrypt data keys against the new recipient set

./scripts/rotate-age-key.py updatekeys

Each encrypted file’s metadata block (the per-file data-key envelope, encrypted to each recipient) gets re-written to include all three recipients. The actual encrypted secret values do not change.

This step needs SOPS_AGE_KEY_FILE pointing at a key that’s currently a recipient — it has to decrypt the existing data key in order to re-wrap it for the new recipient list.

4. Distribute the new private key

Out-of-band (encrypted email, secure file share, in-person USB stick — anything but git). The new operator saves it as their /configs/sops/age/keys.txt (the workspace-container default) or wherever their SOPS_AGE_KEY_FILE points.

5. Verify with the new key

SOPS_AGE_KEY_FILE=/path/to/new.key ./scripts/rotate-age-key.py verify

Use the new key. If all files report OK, the rotation worked. If any FAIL, the new recipient didn’t get added to that file’s .sops.yaml rule, or updatekeys was skipped — revisit step 2 and 3.

6. Remove the old recipient

Once everyone’s confirmed they have and can use the new key, edit each .sops.yaml to remove the old recipient line (both the anchor at the top and every *old reference under age: lists). Then re-wrap:

./scripts/rotate-age-key.py updatekeys
./scripts/rotate-age-key.py verify

7. Retire the old private key

If the rotation was for compromise reasons, securely destroy the old key (shred -u /configs/sops/age/keys.txt on every operator’s machine; coordinate). If it was just hygiene/staffing, archive it somewhere offline in case a future operator needs to read an encrypted-at-the-time backup of these files.

Procedure: recovery (no working private key for any recipient)

SOPS recovery is mostly “you can’t.” If no operator has a private key matching any current recipient in any .sops.yaml, the encrypted values in the matched files are unrecoverable from those files alone. The age encryption is doing its job.

The realistic recovery paths, in order of preference:

Restore .sops.yaml + encrypted files from a git commit when an old recipient was still trusted. Then decrypt with that old key (if anyone still has it), updatekeys against the new recipient set, and re-commit. This works only if at least one previously-trusted key still exists somewhere.
Restore the encrypted files from a backup that predates the lockout (borg/restic snapshot of /configs/sops/age/keys.txt, or an operator workstation backup). Same prerequisite — recover an old private key, then run the rotation flow.
Re-bootstrap the encrypted values from primary sources. Treat the encrypted contents as lost: regenerate every password / htpasswd / PEM / pubkey from scratch, write them into fresh SOPS files encrypted to a new recipient set, then redeploy every consumer (rotate the live admin creds for Forgejo/Zot/Headlamp/OpenBao, re-issue heimdall’s intermediate CA materials per ca-rotation, re-key borg backup clients per playbooks/setup-backup-servers.yaml --tags clients, regenerate per-site LAMP db_password values + roll the MariaDB user passwords to match, etc.).

The add-then-remove strategy structurally avoids ever needing this recovery path: at no point are the OLD private key holders cut off until the new key is verified working. The atomic swap risks landing here if it fails mid-step.

Verification commands

# What .sops.yaml configs exist, who's a recipient where, what files match each rule
./scripts/rotate-age-key.py list

# All currently-encrypted files decrypt with the configured SOPS_AGE_KEY_FILE
./scripts/rotate-age-key.py verify

# Verify with a SPECIFIC key file (the no-false-positive check —
# don't trust the ambient SOPS_AGE_KEY_FILE if it might point at the old key)
SOPS_AGE_KEY_FILE=/path/to/key ./scripts/rotate-age-key.py verify

# Re-encrypt data-key envelopes (after editing .sops.yaml's recipient list)
./scripts/rotate-age-key.py updatekeys

# Generate a new keypair (does not touch any existing files)
./scripts/rotate-age-key.py keygen --out NEW_KEY_FILE

# Inspect a single file's data-key envelope without decrypting the values:
sops -d --extract '["sops"]["age"]' <file>     # shows which recipients can decrypt

Don’t rely on the ambient SOPS_AGE_KEY_FILE during verify — if it points at the old key, verify will pass even after step 6 of the procedure has already cut the old recipient out of the data-key envelopes (wait — actually that’s the danger zone, see Rollback). Always test verify against the NEW key explicitly before declaring success.

Rollback

If a step fails mid-way and you need to revert:

Failure during step 2 (.sops.yaml edits): git checkout the file(s). The encrypted files’ data-key envelopes (set by the LAST successful updatekeys) still match whichever recipients were in .sops.yaml at that time, so reverting the config and not re-running updatekeys puts you back in the working state.
Failure during step 3 (first updatekeys): data-key envelopes now wrap for the EXPANDED recipient set (old + new). Both keys work. Either continue forward by distributing the new key, or roll back by reverting .sops.yaml AND re-running updatekeys to strip the new recipient out of the envelopes.
Failure during step 5 (verify fails): indicates step 2 or 3 was incomplete. Don’t proceed to step 6 — fix the incomplete state first. The fleet is in a safe dual-recipient state; nothing is broken.
Failure during step 6-7 (removed old recipient too early): as long as you still have at least one private key from the CURRENT recipient set, you can re-add the old recipient to .sops.yaml, re-run updatekeys, and you’re back in dual-recipient state.

The danger zone is “old recipient removed from .sops.yaml, updatekeys already re-wrapped envelopes to exclude the old, new private key not yet distributed to every operator who needs to decrypt, and the one operator with the new key is unavailable.” The add-then-remove strategy structurally avoids this by completing step 4 (distribute) and step 5 (verify) BEFORE step 6 (remove old). If you’re tempted to compress those steps “to save time,” don’t — they’re spread out specifically to keep the danger window closed.

Special cases

Rotating one `.sops.yaml`’s recipient set without touching the others

Edit only that .sops.yaml, then ./scripts/rotate-age-key.py updatekeys — it walks every encrypted file regardless of which .sops.yaml rule applies. Files outside the affected rule are no-ops (their data-key envelope already matches their config; SOPS detects no change needed). Useful for project-scoped operator access (e.g. an operator who needs ansible secrets but not k0s platform secrets).

MAC mismatch errors

If verify reports MAC mismatch, someone edited an encrypted file outside SOPS (e.g. hand-edited the YAML, broke the integrity check). This isn’t a rotation issue per se — fix it by:

Restore the file from git: git checkout <file>
Decrypt and re-edit through SOPS: sops <file> (opens editor, re-encrypts on save, refreshes MAC)

Don’t updatekeys a MAC-mismatched file — you’ll just propagate the broken state.

Rotating the AES data-key too, not just recipients

sops updatekeys re-wraps the existing AES data key for the new recipient set. If you want a fresh AES key as well (paranoid post-compromise — assume the old AES key may have leaked alongside the recipient key), use sops -r <file> per file. Forces full re-encryption of the values. Optional; usually not needed because the AES key never leaves the SOPS workflow.

Adding the second operator from scratch (multi-recipient steady state)

The multi-recipient state was reached via exactly this flow — and it has since been exercised twice more: the cicd recipient (2026-05-30) and the reserved agent anchor. If the repo has only ever had one age recipient and you want to start tracking who-has-access in a structured way, the add-then-remove procedure’s steps 1-5 are the entry point. Add the new operator’s public key alongside the existing one, run updatekeys, never run steps 6-7 (remove old). The result is a multi-recipient config that’s easy to grow further — same procedure, just always stop after step 5.

Rotating in lockstep with a downstream secret

Some encrypted fields don’t stand alone — they map to live state on a service that has to be updated in lockstep. Specifically:

intermediate_ca_key / intermediate_ca_key_password in group_vars/step_ca_servers.sops.yaml: changing these without redeploying step-ca on heimdall leaves the live signer using stale materials. Rotation of these fields specifically is documented in ca-rotation — SOPS recipient rotation alone doesn’t touch the values, just who can read them, so this is normally a non-issue. But if you’re rotating BOTH (compromise of both the SOPS key AND the intermediate key) — do the SOPS rotation first, then the CA bundle rotation, to avoid re-encrypting freshly-issued material under a soon-to-be-retired recipient.
(Historical: pubkey values were once SOPS-encrypted in per-backup-host files; pubkey is deliberately NOT in the encrypted_regex today, and those host_vars files no longer exist. Borg client SSH-key rotation lives in playbooks/setup-backup-servers.yaml --tags clients.)