CA rotation

This document describes how to rotate the internal CA chain — root and/or intermediate — across the fleet trust bundle, heimdall’s step-ca service, and the in-cluster cert-manager ClusterIssuer. The mechanical chain generation is automated by ansible/playbooks/generate-ca-bundle.yaml (the same playbook used at initial bootstrap); this SOP is the downstream rollout — what to do with the chain it produces.

Three flavors of “rotation” are covered:

Intermediate-only — routine expiry (2-year cadence, low-risk): the intermediate hits its notAfter date, or you want to rotate the online-key material proactively. The root is untouched, so fleet trust bundle and cert-manager caBundle need no edits. Leaf certs auto-renew through cert-manager within ~16h once heimdall serves the new chain.
Intermediate-only — compromise (rare, higher-risk): the intermediate key on heimdall may be exposed. Same scope as routine expiry, but no parallel-validity window — replace immediately and accept brief chain mismatch on connections in flight at cutover.
Root + intermediate (10-year cadence, algorithm change, or root-key compromise): the full chain is reissued. Fleet trust bundle gets a multi-PEM overlap window so every host trusts both the old and new root simultaneously; cert-manager’s caBundle follows the same overlap. The intermediate procedure runs as a sub-step inside this one.

generate-ca-bundle.yaml does the cryptographic work (root + intermediate, signed, verified, ECDSA P-384, 10y/2y validity) and writes a SUMMARY.txt next to the materials with an 8-step ordered downstream runbook. This SOP is that runbook in long form — same steps, with the rationale, verification gates, and rollback paths the SUMMARY.txt only references.

Design + decision-record for the chain itself: ca-bootstrap. Why heimdall is an IncusOS container and not a k0s pod: internal-ca “Why we moved out of k0s”.

What CA bundle rotation entails

The internal CA chain is consumed at four surfaces, and a rotation has to touch some or all of them depending on the flavor:

Surface	File	Form	Consumers	Touched by
Fleet trust bundle	`ansible/group_vars/all.yaml` → `ca_root_cert`	Plaintext PEM (multi-PEM-aware)	every Debian VM via `init` role’s `ca-trust` task (`update-ca-certificates`); also heimdall’s `step-ca` roots.pem (same value, single source)	root rotations only
Intermediate cert	`ansible/group_vars/step_ca_servers.yaml` → `intermediate_ca_cert`	Plaintext PEM	heimdall’s `step-ca` (signs all leaves)	intermediate rotations (both flavors)
Intermediate private key + password	`ansible/group_vars/step_ca_servers.sops.yaml` → `intermediate_ca_key` + `intermediate_ca_key_password`	SOPS-encrypted PEM + base64 password	heimdall’s `step-ca` (unlocks the signing key at boot)	intermediate rotations
cert-manager `caBundle`	`k0s/cert-manager/heimdall-clusterissuer.yaml` → `spec.acme.caBundle`	Base64 of the root PEM (single value)	cert-manager (verifies heimdall’s TLS leaf when calling its ACME directory)	root rotations only

Three properties of this layout determine how rotations actually play out:

Root is the trust anchor; intermediate is replaceable cheaply. An intermediate rotation only mutates heimdall’s served materials (cert + key + password). Every leaf cert is then re-issued via cert-manager’s auto-renewal loop within 2 × max-leaf-validity (24h in our config), automatically chaining through the new intermediate. Nothing else in the fleet has to be told anything.
Root rotation needs parallel validity. During a root swap the fleet must trust both the old and new roots simultaneously, because leaves signed by the old chain are still valid until they renew. ca_root_cert accepts multiple stacked PEM blocks; update-ca-certificates picks both up. The overlap window stays open until every leaf has been re-issued through the new chain at least once — then the old root comes out.
cert-manager’s caBundle is a manual lockstep edit. The init role pushes ca_root_cert automatically to every fleet VM, but the in-cluster ClusterIssuer reads its trust anchor from a base64-encoded copy that has to be updated by hand whenever ca_root_cert changes. Forget this and cert-manager will refuse to talk to heimdall after the rotation (TLS verify failure against the new chain).

A typical intermediate rotation therefore looks like:

generate fresh chain → swap intermediate cert+key on heimdall → restart step-ca → wait ~16h → done.
fleet trust bundle: unchanged.    cert-manager caBundle: unchanged.    leaf certs: auto-renew.

A typical root+intermediate rotation looks like:

generate fresh chain → multi-PEM the trust bundle (old + new) → push to fleet
                    → swap intermediate cert+key on heimdall → restart step-ca
                    → update caBundle to NEW root (or temporarily multi-PEM base64'd)
                    → wait for all leaves to renew through new chain (24-48h)
                    → strip old root from trust bundle → re-push to fleet → done.

The mechanical 8-step list in the playbook’s SUMMARY.txt (generate-ca-bundle.yaml:393-485) covers the root+intermediate case; intermediate-only is steps 2-3 and 6-7 of that list. The playbook also echoes the SUMMARY.txt body inline at the end of the run, so you don’t need to ssh to the target host to read it.

Status

Reference procedure. Current chain generated 2026-05-17; intermediate notAfter 2028-05-16 (rotate on or before that date), root notAfter 2036-05-14 (rotate well before). Both fingerprints live in /configs/incus/seeds/ per the bootstrap doc Step 4. No rotation has happened yet on this fleet.

What gets touched

See the table in What CA bundle rotation entails above. To enumerate the live state of each surface:

# Fleet trust bundle source-of-truth (multi-PEM if mid-rotation):
grep -A1000 '^ca_root_cert: |' ansible/group_vars/all.yaml | grep -c BEGIN

# Intermediate cert (PEM) — fingerprint:
yq '.intermediate_ca_cert' ansible/group_vars/step_ca_servers.yaml | step certificate fingerprint /dev/stdin

# Intermediate private key + password (SOPS — decryption uses /configs/sops/age/keys.txt):
sops -d ansible/group_vars/step_ca_servers.sops.yaml | yq '.intermediate_ca_key' >/dev/null && echo "ok"

# cert-manager caBundle (base64 of root PEM):
yq '.spec.acme.caBundle' k0s/cert-manager/heimdall-clusterissuer.yaml | base64 -d | step certificate fingerprint /dev/stdin

# Live on-fleet trust-store fingerprint (one host as sample):
ansible <one-vm> -m shell -a 'step certificate fingerprint /usr/local/share/ca-certificates/rrchnm-internal-ca.crt 2>/dev/null || openssl x509 -in /usr/local/share/ca-certificates/rrchnm-internal-ca.crt -fingerprint -sha256 -noout'

# Live on-heimdall served leaf chain:
echo | openssl s_client -showcerts -connect heimdall.rrchnm.internal:443 -servername heimdall.rrchnm.internal 2>/dev/null | openssl crl2pkcs7 -nocrl -certfile /dev/stdin | openssl pkcs7 -print_certs -noout

All five fingerprints must agree pairwise once a rotation has converged:

ca_root_cert (group_vars) ↔ caBundle (clusterissuer, base64-decoded) ↔ live fleet trust-store cert ↔ root in heimdall’s served chain.
intermediate_ca_cert (group_vars) ↔ intermediate in heimdall’s served chain.

Strategy: in-place reprovision (recommended)

Default path for ECDSA → ECDSA rotations (no algorithm change). Re-run the bundle playbook against heimdall, swap the materials in group_vars + host_vars, redeploy step-ca, let cert-manager’s renewal loop heal the fleet within ~16h (= 24h max-leaf-validity / 2 + some slack).

1. ansible-playbook playbooks/generate-ca-bundle.yaml          (or `make generate-ca-bundle`)
                                          ← prompts for ca_host, rotation_context,
                                            root_password, intermediate_password;
                                            pauses for review showing the current
                                            chain's fingerprints; writes
                                            ~/<TS>-rrchnm-ca-bundle/ with chain
                                            + SUMMARY.txt; echoes SUMMARY.txt
                                            inline at the end of the run
2. (root rotations only) escrow new root materials to USBs
3. SOPS-edit group_vars/step_ca_servers.sops.yaml with intermediate key + password
4. Plain-edit group_vars/step_ca_servers.yaml with intermediate cert PEM
5. (root rotations only) multi-PEM prepend new root to ca_root_cert
   in group_vars/all.yaml — KEEP old root until step 8
6. (root rotations only) push trust bundle to fleet:
        ansible-playbook playbooks/pre-flight.yaml --tags ca-trust -e pre_flight_hosts=all
7. Redeploy step-ca:
        make step-ca
8. (root rotations only) update caBundle in heimdall-clusterissuer.yaml
   and kubectl apply
9. Wait for cert-manager auto-renewals (24-48h); spot-check renewals validate
10. (root rotations only) strip old root from ca_root_cert; re-push trust bundle
11. Securely destroy working-dir materials per playbook's SUMMARY.txt

For intermediate-only flavors, steps 5/6/8/10 are skipped. For compromise flavor (intermediate-only or root-only), step 9’s overlap window is collapsed to “as fast as possible” with the understanding that some in-flight TLS handshakes will fail and need to retry.

Rollback at any step before 10 is “paste the previous SOPS values back, re-run step 7” — see Rollback.

Strategy: parallel-overlap via `forseti` (PQ migration, risk-averse)

Use when you can’t tolerate chain-validity mismatch even briefly — primarily future post-quantum algorithm migrations where old clients literally cannot verify new-algorithm signatures and the in-place self-heal window doesn’t apply.

The infrastructure for this is pre-staged (inventory entry forseti at .100 in ansible/inventory.yaml, per-host SAN templating via step_ca_dnsnames in setup-step-ca-servers.yaml, multi-PEM ca_root_cert distribution already in place — see ca-bootstrap “Two rotation patterns”). The procedural shape:

1. activate forseti as a second step-ca instance, chaining to the SAME root
   (or a parallel new-algorithm root)
2. create a parallel forseti-clusterissuer.yaml
3. migrate Certificates one at a time via spec.issuerRef.name flip
4. once every Certificate has issuerRef: forseti, retire heimdall
5. rename forseti → heimdall (or leave; it's just a name)

Per-service rollback is a one-field flip on the Certificate’s issuerRef. Full design: plan-step-ca-pq-migration.md (operator note).

Don’t pick this strategy for routine ECDSA → ECDSA intermediate rotations — the operational overhead is significant and the in-place strategy’s 16h self-heal handles those cases without it.

Pre-rotation checklist

All cert-manager Certificates are healthy. Don’t rotate from a broken state.
Terminal window
```
kubectl get certificate -A -o wide   # expect Ready=True everywhere
```

Heimdall is healthy. Verify step-ca’s ACME directory + the served chain validates against the current root:

curl -fsS --cacert <(yq '.ca_root_cert' ansible/group_vars/all.yaml) \
  https://heimdall.rrchnm.internal/acme/acme/directory | jq .

All four surfaces’ fingerprints currently agree (per What gets touched).

step CLI works on heimdall.

ansible heimdall -m command -a 'step version'

For root rotations: identify the two USBs for the new root-key escrow (per ca-bootstrap Step 4) — physically present before generation.
Decide flavor: intermediate-only / root+intermediate / compromise (intermediate-or-root). Document why.
Decide strategy: in-place reprovision (almost always) or parallel-overlap via forseti (only if PQ / extreme risk-aversion).
Verify SOPS access: SOPS_AGE_KEY_FILE is set, and you can decrypt ansible/group_vars/step_ca_servers.sops.yaml. Without this, step 3 is impossible.
Schedule the rotation window. Step 9’s renewal-loop watch is 24-48h of low-touch monitoring; pick a window that doesn’t collide with other planned changes that would mask renewal failures.

Procedure A: intermediate-only rotation (in-place)

The common path. Runs in ~30 minutes plus the 16-24h renewal window.

1. Generate the new chain

# Either invocation works; the `make` form passes ca_host via -e so the
# ca_host prompt is skipped:
make generate-ca-bundle                            # uses CA_HOST=heimdall (Makefile default)
make generate-ca-bundle CA_HOST=forseti            # parallel-overlap pattern
ansible-playbook playbooks/generate-ca-bundle.yaml    # all 4 prompts interactive

Prompts, in order:

ca_host — target host for chain generation (default heimdall; override for forseti / parallel-overlap). Skipped when -e ca_host=… is set, e.g. via make generate-ca-bundle CA_HOST=….
rotation_context — free-form audit trail line, written into SUMMARY.txt’s “Generated by:” header. Default pre-fills USER@HOSTNAME, YYYY-MM-DD; edit to capture the why (e.g. “routine 2y intermediate expiry”, “compromise response 2026-05-23”, “PQ migration”).
root_password — long passphrase; will be escrowed per ca-bootstrap Step 4. Prompted twice (confirm: true) to catch typos.
intermediate_password — distinct from root; lands in group_vars/step_ca_servers.sops.yaml as intermediate_ca_key_password. Prompted twice.

After the four prompts, the playbook reads the current root + intermediate fingerprints from group_vars/all.yaml and group_vars/step_ca_servers.yaml on the controller, then pauses for a review screen showing all the resolved inputs (target host, output dir, DN, validity, key algorithm) alongside the current chain’s fingerprints. Press Enter to proceed; Ctrl-C + A to abort. The final summary echoes the SUMMARY.txt content inline so you don’t need to ssh to the target host to read the downstream runbook.

This generates root + intermediate even though you’re only rotating the intermediate. The new root is discarded for this flavor; only the intermediate cert + intermediate key + intermediate password are used downstream. (Re-signing the existing intermediate with the existing root via just step certificate sign would also work but isn’t what the playbook exposes — the playbook always generates a fresh root in the bundle dir, and that’s fine because you don’t propagate it.)

Why generate a fresh root when only rotating the intermediate? The playbook’s purpose is one-shot bundle generation; specializing it for intermediate-only would duplicate the templating logic. The discarded root is harmless as long as you don’t accidentally promote it into group_vars/all.yaml. The playbook writes a ca_root_cert.pem in the output dir — for intermediate-only rotations, do not copy that file into group_vars/all.yaml.

Note the new intermediate fingerprint from the playbook output for verification later.

2. SOPS-edit heimdall’s intermediate key + password

sops ansible/group_vars/step_ca_servers.sops.yaml

Replace:

intermediate_ca_key ← contents of ~/<TS>-rrchnm-ca-bundle/rrchnm-intermediate.key
intermediate_ca_key_password ← contents of ~/<TS>-rrchnm-ca-bundle/intermediate.pass.b64 (single line, base64 of the passphrase)

Save. The community.sops vars plugin re-encrypts on save and refreshes the MAC.

3. Plain-edit the intermediate cert in group_vars

$EDITOR ansible/group_vars/step_ca_servers.yaml

Replace intermediate_ca_cert with the contents of ~/<TS>-rrchnm-ca-bundle/rrchnm-intermediate.crt (full PEM, including BEGIN/END markers, indented to match the block scalar).

4. Redeploy step-ca

make step-ca
# == ansible-playbook playbooks/setup-step-ca-servers.yaml

The play’s step-ca tag re-templates step-ca’s ca.json + leaf serving cert and restarts the systemd unit. Restart should take <5s.

5. Verify heimdall serves the new chain

# Served leaf chain — intermediate should match new fingerprint:
echo | openssl s_client -showcerts -connect heimdall.rrchnm.internal:443 \
  -servername heimdall.rrchnm.internal 2>/dev/null \
  | openssl crl2pkcs7 -nocrl -certfile /dev/stdin \
  | openssl pkcs7 -print_certs \
  | step certificate fingerprint /dev/stdin

Cross-check against the playbook’s output int_fp line.

# ACME directory still validates with EXISTING root (root didn't change):
curl -fsS --cacert <(yq '.ca_root_cert' ansible/group_vars/all.yaml) \
  https://heimdall.rrchnm.internal/acme/acme/directory >/dev/null && echo "directory OK"

6. Force-renew a non-critical Certificate; confirm new chain

Pick something non-load-bearing to fail safely. Annotate to force renewal:

kubectl -n <ns> annotate certificate <name> \
  cert-manager.io/issue-temporary-certificate=true --overwrite
# Or: kubectl cert-manager renew <name> -n <ns>   (if the krew plugin is available)

Watch its issuance:

kubectl -n <ns> describe certificate <name> | tail -30

A Issued event with notAfter ≈ now + 24h means cert-manager successfully roundtripped the new chain. Inspect the resulting Secret’s chain to confirm the intermediate fingerprint matches what the playbook produced.

7. Watch cert-manager renewals over the next 16-24h

kubectl get certificate -A -o wide
# Re-run periodically. As each Certificate approaches its renewal threshold
# (cert-manager default: 2/3 of validity), it auto-renews. Watch for
# any "Renewing" stuck at error.

All Certificates should auto-renew once and validate against the new intermediate (the existing root, which they already trust). Investigate any stuck Renewing — usually a transient ACME glitch; retry the annotate.

8. Securely destroy the working-dir materials

shred -u ~/<TS>-rrchnm-ca-bundle/intermediate.pass ~/<TS>-rrchnm-ca-bundle/intermediate.pass.b64
shred -u ~/<TS>-rrchnm-ca-bundle/rrchnm-intermediate.key
shred -u ~/<TS>-rrchnm-ca-bundle/root.pass ~/<TS>-rrchnm-ca-bundle/rrchnm-root.key  # discarded root materials
# Public certs (rrchnm-root.crt, rrchnm-intermediate.crt) and SUMMARY.txt
# can stay archived if you want a paper trail.

Done. The fleet trust bundle was never touched; cert-manager’s caBundle was never touched; root is unchanged.

Procedure B: root + intermediate rotation (in-place)

Steps 1-4 of Procedure A apply identically, plus the additional steps below. Total wall-clock ~30 minutes of operator work + 24-48h of unattended renewal-loop watching.

1-4. (Same as Procedure A above)

But this time you DO propagate the new root.

5. Escrow the new root key materials

Per ca-bootstrap Step 4: two encrypted USBs with rrchnm-root.{crt,key} + root.pass, in physically separate locations. Verify both USBs round-trip-readable before continuing.

Without this, you have no recovery path if the working heimdall step-ca state gets corrupted before the new chain is fully bedded in.

6. Multi-PEM prepend new root to `ca_root_cert`

$EDITOR ansible/group_vars/all.yaml

In ca_root_cert, place the new root PEM block above the existing one:

ca_root_cert: |
  -----BEGIN CERTIFICATE-----
  <NEW root cert content from rrchnm-root.crt — paste here>
  -----END CERTIFICATE-----
  -----BEGIN CERTIFICATE-----
  <EXISTING root cert content — leave in place until step 11>
  -----END CERTIFICATE-----

Both certs are public-by-design. No SOPS needed.

7. Push the multi-PEM trust bundle to the entire fleet

ansible-playbook playbooks/pre-flight.yaml --tags ca-trust -e pre_flight_hosts=all

The init role’s ca-trust task writes the multi-PEM content as a single file in /usr/local/share/ca-certificates/rrchnm-internal-ca.crt; update-ca-certificates splits it into two trust-store entries and refreshes the hash bundle. Idempotent — re-runs are safe.

Verify on a representative sample of hosts:

ansible <one-host-per-group> -m shell -a \
  'awk "/BEGIN/,/END/" /usr/local/share/ca-certificates/rrchnm-internal-ca.crt | grep -c BEGIN'
# expect: 2 on every host

8. Redeploy step-ca

make step-ca

step-ca now serves a leaf signed by the new intermediate, which chains to the new root. Existing fleet leaves are still valid (signed by the OLD intermediate-OLD root chain), and clients validate them against the OLD root in the multi-PEM bundle.

Verify chain on the wire:

echo | openssl s_client -showcerts -connect heimdall.rrchnm.internal:443 \
  -servername heimdall.rrchnm.internal 2>/dev/null \
  | openssl crl2pkcs7 -nocrl -certfile /dev/stdin \
  | openssl pkcs7 -print_certs
# Should show: leaf -> NEW intermediate -> NEW root

9. Update cert-manager’s `caBundle` in lockstep

NEW_ROOT_B64="$(base64 -w0 < ~/<TS>-rrchnm-ca-bundle/rrchnm-root.crt)"
$EDITOR k0s/cert-manager/heimdall-clusterissuer.yaml
# Replace spec.acme.caBundle with $NEW_ROOT_B64
kubectl apply -f k0s/cert-manager/heimdall-clusterissuer.yaml

cert-manager re-reads the ClusterIssuer on apply; the next ACME directory call uses the new caBundle. Confirm:

kubectl describe clusterissuer heimdall | grep -i ready
# expect: Ready True (ACME registration valid)

Why not multi-PEM the caBundle? It can be done — concatenate both PEMs before base64-encoding — but it’s not necessary. cert-manager only talks to heimdall, and heimdall now serves only the new chain (since step 8). The fleet trust bundle is multi-PEM to cover the leaves that are still signed by the old chain on fleet hosts; cert-manager has no equivalent population of “leaves it needs to trust against the old root.”

10. Watch cert-manager auto-renewals (24-48h)

Same as Procedure A step 7. Every Certificate should renew once and re-issue a leaf signed by the new chain. Spot-check:

kubectl get certificate -A -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}: {.status.notAfter}{"\n"}{end}'
# Every notAfter should be > now + 23h within ~24h of step 8.

A leaf whose notAfter doesn’t advance after 24h is stuck — usually a Gateway/HTTPRoute/DNS issue blocking the HTTP-01 challenge, not a CA issue. Investigate per cert.

11. Strip the OLD root from the trust bundle

Only after step 10 confirms every Certificate has renewed via the new chain. Edit group_vars/all.yaml:

ca_root_cert: |
  -----BEGIN CERTIFICATE-----
  <NEW root cert content — only this block remains>
  -----END CERTIFICATE-----

Re-push:

ansible-playbook playbooks/pre-flight.yaml --tags ca-trust -e pre_flight_hosts=all

Verify the fleet trust store now has exactly one rrchnm root:

ansible <one-host-per-group> -m shell -a \
  'awk "/BEGIN/,/END/" /usr/local/share/ca-certificates/rrchnm-internal-ca.crt | grep -c BEGIN'
# expect: 1 on every host

12. Destroy working-dir materials

Same as Procedure A step 8, but the root materials are real now — keep the escrow USBs, destroy the on-disk working copies (rrchnm-root.key, root.pass).

Procedure: recovery

Heimdall step-ca won’t start after redeploy

Likely causes:

Intermediate key + password mismatch (sops yaml has new key but old password, or vice versa).
Intermediate cert PEM in group_vars doesn’t match the key on disk.
step-ca’s roots.pem (rendered from ca_root_cert) doesn’t contain the root that signs the intermediate.

Read step-ca’s systemd journal first:

ansible heimdall -m command -a 'journalctl -u step-ca -n 100 --no-pager'

The fix is almost always to re-run Procedure A steps 2-4 with the matching set of materials from the same ~/<TS>-rrchnm-ca-bundle/ output dir.

cert-manager Certificate stuck at “Renewing” with TLS verify errors

The caBundle in the ClusterIssuer doesn’t match the root in heimdall’s served chain. Re-run step 9 of Procedure B; or, if you forgot to update it at all and you’re still in the overlap window, the EXISTING caBundle (old root) won’t verify the NEW heimdall leaf — you must update.

Sanity check:

kubectl logs -n cert-manager deploy/cert-manager | tail -50
# Look for "x509: certificate signed by unknown authority" against heimdall's URL.

Fleet host can’t validate a leaf served by another fleet host

The trust bundle on the validating host is missing the root that signed the leaf. Either step 7 of Procedure B didn’t run on that host, or step 11 ran prematurely (before the leaf had renewed).

# On the validating host:
awk "/BEGIN/,/END/" /usr/local/share/ca-certificates/rrchnm-internal-ca.crt | grep -c BEGIN
# Should be 2 during overlap, 1 after step 11.

# Force a single host's trust bundle to re-sync:
ansible <stuck-host> -m import_role -a name=init --tags ca-trust

Lost the new intermediate key before SOPS-staging it

The working-dir materials in ~/<TS>-rrchnm-ca-bundle/ are the only copy of the new intermediate key. If shredded prematurely (or the workspace container restarted before staging), the rotation is unrecoverable — re-run generate-ca-bundle.yaml from scratch to produce a different chain.

Mitigation: don’t shred until after Procedure A step 7 or Procedure B step 11.

Lost access to the SOPS age key during rotation

The host_vars edit at step 3 of Procedure A requires SOPS decrypt + re-encrypt. Without the age key, you can’t write the new intermediate key into step_ca_servers.sops.yaml. Recovery is sops-key-rotation — restore SOPS access first, then resume the CA rotation. The CA rotation can pause indefinitely at this gate; nothing else is in flight.

Verification commands

# All four surfaces' fingerprints agree:
echo "ca_root_cert (group_vars):   $(yq '.ca_root_cert' ansible/group_vars/all.yaml | step certificate fingerprint /dev/stdin)"
echo "caBundle (clusterissuer):    $(yq '.spec.acme.caBundle' k0s/cert-manager/heimdall-clusterissuer.yaml | base64 -d | step certificate fingerprint /dev/stdin)"
echo "intermediate (group_vars):   $(yq '.intermediate_ca_cert' ansible/group_vars/step_ca_servers.yaml | step certificate fingerprint /dev/stdin)"

# Served chain on heimdall:
echo | openssl s_client -showcerts -connect heimdall.rrchnm.internal:443 \
  -servername heimdall.rrchnm.internal 2>/dev/null \
  | openssl crl2pkcs7 -nocrl -certfile /dev/stdin \
  | openssl pkcs7 -print_certs -noout

# Fleet-wide trust-store fingerprint sample:
ansible all:!incus_servers -m shell -a \
  'step certificate fingerprint /usr/local/share/ca-certificates/rrchnm-internal-ca.crt 2>/dev/null \
   || openssl x509 -in /usr/local/share/ca-certificates/rrchnm-internal-ca.crt -fingerprint -sha256 -noout'

# cert-manager renewal health:
kubectl get certificate -A -o wide
kubectl get clusterissuer heimdall -o yaml | grep -A5 conditions

# step-ca service health:
ansible heimdall -m command -a 'systemctl status step-ca'
curl -fsS --cacert <(yq '.ca_root_cert' ansible/group_vars/all.yaml) \
  https://heimdall.rrchnm.internal/acme/acme/directory | jq .

# How many roots are in the fleet trust bundle right now (1 = converged, 2 = mid-rotation):
ansible <one-host> -m shell -a \
  'awk "/BEGIN/,/END/" /usr/local/share/ca-certificates/rrchnm-internal-ca.crt | grep -c BEGIN'

Don’t use curl --cacert /etc/ssl/certs/ca-certificates.crt to verify heimdall’s chain — that file is built by update-ca-certificates from the multi-PEM trust bundle, so it’ll happily validate against either root during overlap. Use --cacert <(yq '.ca_root_cert' …) against the source-of-truth file to know which root specifically validated the leaf.

Rollback

The CA rotation has clear rollback points up to step 10 of Procedure B (= step 7 of Procedure A); past that, the world has already converged on the new chain and “rollback” means rotating again, not reverting.

Failure during	Rollback path
Step 1 (generate)	`rm -rf ~/<TS>-rrchnm-ca-bundle/`. No state mutated yet.
Step 2-3 (sops/group_vars edit)	`git checkout` the files; SOPS file edits are auto-reverted as long as `git checkout` precedes `make step-ca`.
Step 4 (`make step-ca`), step-ca won’t start	Re-apply the PREVIOUS SOPS values from `git log -1 ansible/group_vars/step_ca_servers.sops.yaml` (decrypt + re-encrypt) AND the previous `intermediate_ca_cert`, then `make step-ca` again.
Step 6-7 (Procedure B; multi-PEM trust bundle pushed)	The bundle is harmlessly multi-PEM. Strip the new root (revert `group_vars/all.yaml`), re-push with `--tags ca-trust`.
Step 9 (Procedure B; caBundle updated, heimdall still on new chain)	Revert the caBundle to the OLD root, kubectl apply — but cert-manager will then fail to talk to heimdall because heimdall is now serving the new chain. Don’t roll back here unless you ALSO step-ca-rollback (steps 2-4). It’s almost always better to fix the caBundle forward.
Step 11 (Procedure B; old root stripped)	`git checkout ansible/group_vars/all.yaml` to restore the multi-PEM, `--tags ca-trust` push. Only do this if you discover a leaf still chains to the old root (= step 10 verification missed something).

The danger zone is “intermediate rotated on heimdall + caBundle not yet updated + cert-manager attempts a renewal.” cert-manager will fail TLS verification against heimdall; existing leaves keep working until their notAfter, but no new leaves can be issued until you complete step 9. The recommended sequencing puts caBundle update right after step-ca redeploy specifically to minimize this window.

Special cases

Intermediate-only rotation triggered by approaching expiry

Run Procedure A. The bootstrap doc’s “When to rotate” table notes the 2-year intermediate validity; ideally rotate at the 18-month mark to leave a 6-month safety margin. Calendar reminder + cert-manager Prometheus alert (certmanager_certificate_expiration_timestamp_seconds) cover this.

Compromise of the intermediate key

Run Procedure A with the in-place strategy and don’t wait for the renewal-loop overlap window — accept that some in-flight TLS handshakes will fail during the cutover, and force-renew all cert-manager Certificates immediately after step 4 instead of waiting for the auto-renewal threshold:

# Force-renew every certificate cluster-wide:
kubectl get certificate -A -o jsonpath='{range .items[*]}{.metadata.namespace} {.metadata.name}{"\n"}{end}' \
  | while read ns name; do
      kubectl -n "$ns" annotate certificate "$name" \
        cert-manager.io/issue-temporary-certificate=true --overwrite
    done

This makes the renewal window minutes instead of 16h.

Compromise of the root key

Run Procedure B with the in-place strategy and similarly force-renew. The multi-PEM overlap step still happens (steps 6-7) — fleet hosts still need to trust the old root long enough for in-flight leaves to be replaced — but step 11 (strip old root) follows immediately after force-renewal of every Certificate, instead of waiting 24-48h.

Note that “compromise of the root” is itself a higher-order incident than this SOP covers — also rotate the SOPS keys (might have been on the same machine), rotate the Incus client cert if it was on the same machine, audit recent git history for anything else that traveled with the root, and reissue from a known-clean host.

Algorithm change (e.g. ECDSA → post-quantum)

Don’t use Procedure A or B as-is. The in-place strategy assumes clients can validate both old and new algorithms simultaneously, which doesn’t hold across an algorithm boundary. Use the parallel-overlap via forseti strategy and follow plan-step-ca-pq-migration.md (operator note).

Rotating only the intermediate password (not the key)

Not exposed by generate-ca-bundle.yaml. Do it by hand: pick a new passphrase, re-encrypt the existing rrchnm-intermediate.key with step crypto change-pass, base64 the new passphrase, sops-edit both fields in step_ca_servers.sops.yaml, make step-ca. No fleet-wide changes; no caBundle change.

Rotating the root’s DN or SAN fields (CN, O, OU, …)

This is structurally a new root even if the keypair is reused — clients identify roots by issuer DN, not just by key. Run Procedure B (root + intermediate rotation) regardless of which materials are technically “fresh.” Edit the DN fields in generate-ca-bundle.yaml’s vars: block before running (commit the change via PR, since it bakes in).

A subset of the fleet was unreachable during step 7 / step 11 trust-bundle push

ansible-playbook ... --tags ca-trust reports which hosts failed in its summary. Rerun against the failed hosts specifically:

ansible-playbook playbooks/pre-flight.yaml --tags ca-trust -e pre_flight_hosts=<comma,separated,failed,hosts>

Don’t proceed to step 11 until step 7 covered every host — a host stuck on a single-root trust bundle that’s missing the new root will TLS-fail against every new-chain leaf served by the rest of the fleet.