SOPS handling

Operational SOP for editing, renaming, moving, and inspecting SOPS-encrypted files in this repo without leaking plaintext to disk, git history, or logs.

Companion to sops-key-rotation (covers age-recipient changes and full key replacement — orthogonal concern). This SOP is the day-to-day handling layer.

Status

Reference procedure. Distilled from the 2026-05-29 same-day SOPS reorg (5 commits — see the operator changelog) where the first commit accidentally exposed a GitHub PAT in plaintext on disk for ~1 second because the rename path went through /tmp (disk in this container, not tmpfs) and the file’s embedded encrypted_regex predated the regex term that would catch the renamed key. Subsequent commits used the tmpfs-based pattern documented below; nothing was committed plaintext.

Threat model

Surface	Risk	Why
Repo working tree	Plaintext briefly visible in an `.sops.yaml` file	`cp tmp/plain → ansible/dst.sops.yaml` writes plaintext before the follow-up `sops -e -i` overwrites it. Journaling FS (overlayfs in Docker, ext4 on host) may retain prior block contents in the journal until checkpoint.
`/tmp`	Plaintext written to disk via `sops -d > /tmp/foo`	`/tmp` in this container is a normal disk-backed mount, not tmpfs. `shred -u` on ext4/overlayfs cannot guarantee unrecoverability (journal blocks, COW history).
`ansible/ansible.log`	Decrypted hostvars dumped by ansible’s INFO callback	`ansible-inventory --host <h>` returns ALL hostvars including SOPS-decrypted ones; ansible.log captures them verbatim. Gitignored — never committed — but lives in the workspace FS.
Shell history	Plaintext value passed as cmdline arg	`sops --set '["foo"] "<value>"' file` puts the value into `~/.bash_history` unless `unset HISTFILE` first.
Git history	Plaintext committed by accident	`git add` of a half-encrypted file (e.g. value didn’t match `encrypted_regex` and SOPS left it plaintext).
Multi-tenant tmpfs	Other users on same host can read /dev/shm	Single-tenant Docker container per operator — not an issue here. Worth noting if this SOP travels to a shared host.

Out of scope: backup/snapshot pipelines reading the workspace FS, container image layers preserving deleted files, kernel memory dumps. If those are in your threat model, full-disk encryption + rotate-on-suspicion is the answer, not file-level hygiene.

Tools and conventions

Path	Purpose
`/dev/shm/sops-work/`	RAM-backed tmpfs. Use this for any intermediate plaintext file. Anything written here lives only in RAM; `rm` is sufficient (no `shred` needed). Default tmpfs size in Docker is 64 MB — enough for any single sops file in the tree. Create with `mkdir -p && chmod 700`.
`/tmp`	Disk-backed in this container. Never use for plaintext.
`~/.bash_history`	Mind it for any one-shot commands containing secret values. `unset HISTFILE` or use `HEREDOC` form for sensitive values.
`$SOPS_AGE_KEY_FILE`	Preset in Dockerfile to `/configs/sops/age/keys.txt`. Don’t override.
`ansible.cfg`	Has `community.sops` vars plugin enabled — `host_vars/<h>.sops.yaml` + `group_vars/<g>.sops.yaml` auto-decrypt at vars-load time. Implication: ANY ansible command (incl. `ansible-inventory --host`) that touches a host triggers decryption of its sops files.

Recipes

Pick the recipe that matches the operation. Each one closes the plaintext-on-disk window for that specific shape of edit.

1. Inspect a value (read-only)

sops -d ansible/group_vars/foo.sops.yaml | grep '^field:'
# or pipe into a viewer; the decrypted output is stdout, not a file.

No disk writes. Safe.

2. Edit a value in place (simple field)

The “EDITOR trick” — sops opens a temp file in $TMPDIR/$EDITOR, you mutate it, sops re-encrypts on save. Critical: point $TMPDIR at tmpfs.

TMPDIR=/dev/shm EDITOR='sed -i s/old-value/new-value/' \
  sops ansible/group_vars/foo.sops.yaml

For interactive edits with $EDITOR=vim:

TMPDIR=/dev/shm sops ansible/group_vars/foo.sops.yaml

vim also writes swap files under $TMPDIR, so the tmpfs redirect closes that gap too.

3. Rename a key in a single file

Same EDITOR trick — sed handles single-key renames cleanly:

TMPDIR=/dev/shm EDITOR='sed -i s/old_field_name:/new_field_name:/' \
  sops ansible/group_vars/foo.sops.yaml

Gotcha — embedded encrypted_regex. Each .sops.yaml-managed file carries its OWN encrypted_regex in its sops metadata block (snapshotted at last edit time). If the new field name doesn’t match the embedded regex but does match the rule’s regex in ansible/.sops.yaml, sops will re-encrypt using the EMBEDDED regex and leave the renamed field plaintext. Symptom: sops -d file | grep ^field: shows the value in cleartext.

Fix: decrypt and re-encrypt under the current .sops.yaml rule:

mkdir -p /dev/shm/sops-work && chmod 700 /dev/shm/sops-work
sops -d ansible/group_vars/foo.sops.yaml > /dev/shm/sops-work/foo.yaml
# (rename already happened above; this round-trip refreshes embedded metadata)
cp /dev/shm/sops-work/foo.yaml ansible/group_vars/foo.sops.yaml
cd ansible && sops -e -i group_vars/foo.sops.yaml && cd ..
shred -u /dev/shm/sops-work/foo.yaml && rmdir /dev/shm/sops-work

Always verify after rename: sops -d file | grep ^newfield: should show ciphertext-free plaintext content; raw head file should show newfield: ENC[...].

4. Move a value between files

mkdir -p /dev/shm/sops-work && chmod 700 /dev/shm/sops-work
sops -d ansible/host_vars/src.sops.yaml > /dev/shm/sops-work/src.yaml
sops -d ansible/group_vars/dst.sops.yaml > /dev/shm/sops-work/dst.yaml
# Compose new dst contents — header comments + merged keys
cat > /dev/shm/sops-work/new-dst.yaml <<'EOF'
# Header explaining the file's scope...
EOF
cat /dev/shm/sops-work/dst.yaml >> /dev/shm/sops-work/new-dst.yaml
cat /dev/shm/sops-work/src.yaml >> /dev/shm/sops-work/new-dst.yaml
# Encrypt into place
cp /dev/shm/sops-work/new-dst.yaml ansible/group_vars/dst.sops.yaml
cd ansible && sops -e -i group_vars/dst.sops.yaml && cd ..
# Delete source
git rm ansible/host_vars/src.sops.yaml
# Cleanup
shred -u /dev/shm/sops-work/*.yaml && rmdir /dev/shm/sops-work

Pre-flight: update .sops.yaml encrypted_regex FIRST if the destination rule needs to cover field names it doesn’t currently catch. Otherwise sops -e -i will write some values plaintext.

Verify before deleting source:

sops -d ansible/group_vars/dst.sops.yaml | grep '^<expected-field>:'

5. Add a new SOPS file from scratch

mkdir -p /dev/shm/sops-work && chmod 700 /dev/shm/sops-work
cat > /dev/shm/sops-work/new.yaml <<'EOF'
# Header docblock — what's in this file, who consumes it, rotation notes.
field_one: <value>
field_two: <value>
EOF
cp /dev/shm/sops-work/new.yaml ansible/group_vars/new.sops.yaml
cd ansible && sops -e -i group_vars/new.sops.yaml && cd ..
shred -u /dev/shm/sops-work/new.yaml && rmdir /dev/shm/sops-work

Confirm the rule fired: head ansible/group_vars/new.sops.yaml should show field_one: ENC[...]. If values are plaintext, the field names don’t match the rule’s encrypted_regex — fix the rule, delete the file, retry.

6. Edit only comments (no value change)

If you’re rewriting documentation comments inside a sops file without touching any encrypted value, the EDITOR trick is fine — sops doesn’t care that values are unchanged, it’ll round-trip them. Use the same TMPDIR=/dev/shm redirect.

For multi-line comment surgery (where sed gets ugly), the round-trip pattern (recipe 4 without the move step) is cleaner.

7. Generate + encrypt fresh secret material

For scripts that mint a new secret and need to land it encrypted (e.g. scripts/generate-knot-tsig-keys.sh):

Write the plaintext stub to a tmpfile under a trap rm -f $TMP EXIT guard so abnormal exit doesn’t leave plaintext behind.
mv into final path BEFORE sops -e -i — sops resolves .sops.yaml rules relative to the file path being encrypted, not the temp’s path.
For new key MATERIAL (random bytes), use openssl rand -base64 N piped into a shell var; the value transits memory only.
Avoid --set for the value — shell history risk.

The current generate-knot-tsig-keys.sh follows this pattern (writes to mktemp, moves to final path, sops --encrypt --in-place).

What NOT to do

vim ansible/group_vars/foo.sops.yaml — bypasses sops entirely; invalidates the MAC; any save corrupts the file. Use sops <file>.
sops -d file > /tmp/foo.yaml — /tmp is disk-backed; never use it for plaintext SOPS material in this container. Use /dev/shm/.
sops --set '["foo"] "<plaintext-secret>"' file without unset HISTFILE first — the secret lands in shell history.
cat plain.yaml | sops -e > file.sops.yaml from a non-tmpfs source — same disk-write problem as /tmp.
EDITOR='sed -i ...' sops file without TMPDIR=/dev/shm — sed’s in-place edit happens against sops’s temp file, which lands in /tmp by default.
ansible-inventory --host <h> when <h> has a sops file — dumps decrypted values into ansible.log via the INFO callback. If you must, shred -u ansible/ansible.log after.
git add without a quick git diff review — confirm encrypted fields show ENC[...], not raw values, before committing.

Cleanup + verification (always run after a sops operation)

# 1. The file you touched still round-trips:
sops -d <file> | grep '^<expected-field>:'   # should be the plaintext value

# 2. The encrypted form on disk really IS encrypted:
grep '^<expected-field>:' <file>             # should be `field: ENC[...]`

# 3. No plaintext leftover in the tree:
git diff <file>                              # values should be ENC[...] both sides
grep -rn '<known-plaintext-prefix>' ansible/ k0s/  # e.g. `ghp_` for GH PATs

# 4. tmpfs work area gone:
ls /dev/shm/sops-work/                       # should be empty / not exist

# 5. ansible.log didn't capture the value (if you ran ansible during the edit):
grep '<known-plaintext-prefix>' ansible/ansible.log   # should return nothing
# If found: shred -u ansible/ansible.log

Rotation triggers

Run the secret through sops-key-rotation (or its per-secret rotation runbook — e.g. CA bundle rotation, openbao re-init) if any of these happened:

A plaintext value was written to /tmp or any disk-backed path, even briefly. ext4/overlayfs journals can retain prior block contents.
A plaintext value appeared in ansible/ansible.log or any other log the workspace FS retains. Backup pipelines / Docker volume snapshots may have captured it.
A plaintext value was passed on the command line (shell history risk).
A SOPS file was committed with values in plaintext form (git log -p
- git revert is NOT a rotation — the value lives in object history).

The 2026-05-29 commit 099cf33 is an example: the oci_mirror_github_pat value was briefly plaintext on disk during the rename. Operator chose not to rotate (workspace FS treated as trusted; PAT scope was read-only public_repo); your call may differ depending on the secret + threat model.