CA bootstrap & rotation

Executed 2026-05-17; now the rotation runbook

The bring-up ran once (all-online flow, trade-off recorded below); this page is the procedure for the 2028 intermediate expiry or any unscheduled rotation. The SOPS staging location moved in May 2026 — references below say group_vars/step_ca_servers.sops.yaml, the current home (originally host_vars/heimdall.sops.yaml; the retired k0s-era Helm-values shape in the Status paragraph is historical).

One-time procedure to generate the root + intermediate CA materials that back Phase 1 of internal-ca: a two-tier CA (offline root + online intermediate), with the intermediate deployed as step-ca inside k0s.

This document is the manual half of Phase 1. The IaC half — Helm values file, HTTPRoute, deploy-platform.yaml wiring, DNS records, ca_root_cert distribution — is committed separately and consumes the materials produced here.

Status

Executed 2026-05-17 (all-online flow). Root + intermediate generated on the workspace container, materials populated into k0s/platform/heimdall-values.sops.yaml (inject.certificates.{root_ca,intermediate_ca}, inject.secrets.{x509.intermediate_ca_key,intermediate_ca_key_password}); root cert also distributed via ca_root_cert in ansible/group_vars/all.yaml. heimdall deployed; chain verifies end-to-end.

Cert timestamps (sanity reference): root valid 2026-05-17 17:07:59 → 2036-05-14 (ECDSA P-384, 10y); intermediate valid 2026-05-17 17:08:54 → 2028-05-16 (ECDSA P-384, 2y). The ~47 s gap is signature of the all-online flow — both keys lived on the same machine for the duration.

Trade-off acknowledged: per Where to generate, the root key existed on an online machine at generation time. Compromise of the workspace container in that window would have compromised the root. If that risk profile ever becomes unacceptable, recovery is to reissue (new root, new intermediate, fleet-wide trust update via ca_root_cert rotation).

Escrow complete (Step 4):

• Primary: encrypted USB with rrchnm-root.{crt,key} + root.pass.
• Backup: second encrypted USB, offsite (different physical building).
• Recovery passphrase: written on paper, separate physical location from both USBs.

Working copies of rrchnm-root.key, rrchnm-intermediate.key, intermediate.pass{,.b64}, and root.pass shredded from the workspace container per Steps 4 + 5. Public certs (rrchnm-root.crt, rrchnm-intermediate.crt) retained on the workspace for chain-verification use.

Phase 1 of internal-ca is complete; Phase 2 (cert-manager) landed in the same 2026-05-17 session — see the operator changelog’s 2026-05-17 entry. This runbook now serves as the procedure for rotation (when the intermediate hits its 2-year mark in 2028, or any unscheduled rotation) rather than first-time bring-up.

Decisions baked into this runbook

Decision	Choice	Why
Software	step CLI (smallstep)	Matches the chart we deploy (smallstep/step-certificates); same on-disk formats end-to-end
Architecture	Two-tier: offline root + online intermediate	Per design doc — compromise containment
Root key algorithm	ECDSA P-384 (RSA-4096 also valid)	Long-lived (10y); P-384 is ~equivalent to RSA-7680 in classical strength, faster signing, smaller certs, universal modern-stack compat. Neither is post-quantum resistant — see plan-step-ca-pq-migration.md (operator note).
Intermediate key algorithm	ECDSA P-384	Must be the same family as the root so the chain verifies cleanly across every client; lives encrypted on disk in step-ca’s PVC
Root validity	10 years (87600h)	Plenty for our horizon; intermediate rotation handles the typical “rotate the signer” need
Intermediate validity	2 years (17520h)	Long enough to avoid constant rotation; short enough that compromise blast radius is bounded
Leaf validity	90 days (issued by step-ca later)	Standard public-CA cadence; matches cert-manager defaults
Root password and intermediate password	Different	Root password never leaves the air-gapped machine; intermediate password lands in SOPS and is used at every step-ca boot
Root storage	Encrypted USB (primary) + offsite backup	Per design doc; cataloged outside this repo

If any of those don’t fit your environment, adjust before generating — the values bake into the certs and changing them later means reissuing.

Prerequisites

Software

The step CLI from smallstep. The workspace container ships it via the Dockerfile’s smallstep apt repo — no manual install needed for the all-online flow. Verify:

step version
# Expected: Smallstep CLI/0.30.x or newer

For the air-gapped machine: grab step-cli_amd64.deb from https://dl.smallstep.com/cli/ onto a USB, install offline. Any machine that can run a Debian package works; a fresh laptop or one-time live USB is ideal.

Note: Steps 1-3 below are fully automated by ansible/playbooks/generate-ca-bundle.yaml (prompts for two passwords, bakes DN from the playbook’s vars: block, runs the same step commands documented below, outputs the chain + a SUMMARY.txt with the downstream procedure). Use the playbook for the all-online flow. The manual steps below remain authoritative for the split air-gapped/online flow and as a reference for what the playbook does under the hood.

Working directories

# On the air-gapped machine (or jump box if you accept the trade-off):
mkdir -p ~/rrchnm-ca-root
chmod 700 ~/rrchnm-ca-root
cd ~/rrchnm-ca-root

# On the online machine (jump box / workspace container) for intermediate gen:
mkdir -p ~/rrchnm-ca-intermediate
chmod 700 ~/rrchnm-ca-intermediate
cd ~/rrchnm-ca-intermediate

These directories must NOT be inside /workspace/vibes-systems/vibes/ (the git repo) — accidental git add would commit private keys. Pick paths under $HOME or /tmp/ (latter wiped on container restart, which is actually convenient for the intermediate workflow but bad for the root).

Where to generate

Material	Machine	Reason
Root cert + root key	Air-gapped (preferred) — no network, no shared filesystem with online infra	Root key compromise is catastrophic. Keep it physically isolated.
Intermediate CSR + intermediate key + password	Online (this jump box / workspace container is fine)	Intermediate key ships into the cluster anyway; generating it on the online machine matches its eventual exposure surface.
Signing the intermediate CSR (produces intermediate cert)	Air-gapped (where the root key lives)	The root key never leaves the air-gapped machine. The CSR is brought to the root, signed, intermediate cert is brought back out.

The CSR / cert hand-off is via removable media (USB stick, SD card). Plain text — neither CSR nor cert is sensitive — but treat the medium like the root key during transit (don’t lose it).

If a true air-gapped machine isn’t available and you accept the weaker trust story (compromise of this jump box = compromise of root, since root key existed online once), generate everything on the workspace container. Document this choice in changelog.md so a future reviewer can decide whether to reissue.

Step 1 — Generate the root CA (air-gapped machine)

cd ~/rrchnm-ca-root

# Write the root password to a file (so step doesn't need stdin and you
# don't typo it). Pick a long passphrase; this file gets shredded later.
# `stty -echo` for silent input is POSIX-portable (bash's `read -s` is a
# bashism that errors with "Illegal option -s" in dash/sh).
stty -echo; printf 'Root password: '; IFS= read -r ROOT_PW; stty echo; printf '\n'
printf '%s' "$ROOT_PW" > root.pass
chmod 600 root.pass
unset ROOT_PW

# DN fields beyond CN are set via a step-ca certificate template. The
# positional `step certificate create <subject>` arg only ever populates
# CN — the template fills in O/OU/L/ST/C and the CA extensions
# (keyUsage, basicConstraints) that would otherwise come from
# `--profile root-ca`. Single-quoted EOF so the shell passes the
# `{{ ... }}` Go-template directives through to step-ca unmodified
# (they're interpolated at cert-create time, not by the shell).
# Edit the JSON values here BEFORE the create call to change org/locality.
cat > root-ca.tpl <<'EOF'
{
  "subject": {
    "commonName": {{ toJson .Subject.CommonName }},
    "country": ["US"],
    "province": ["Virginia"],
    "locality": ["Fairfax"],
    "organization": ["Roy Rosenzweig Center for History and New Media"],
    "organizationalUnit": ["Infrastructure"]
  },
  "issuer": {{ toJson .Subject }},
  "keyUsage": ["certSign", "crlSign"],
  "basicConstraints": {
    "isCA": true,
    "maxPathLen": 1
  }
}
EOF

step certificate create \
  "RRCHNM Internal Root CA" \
  rrchnm-root.crt rrchnm-root.key \
  --template root-ca.tpl \
  --kty EC --crv P-384 \
  --not-after 87600h \
  --password-file root.pass

# Verify (use long-form inspect to see the full DN — --short only prints CN)
step certificate inspect rrchnm-root.crt --short
# Expected: Subject: RRCHNM Internal Root CA
#           Issuer: RRCHNM Internal Root CA  (self-signed)
#           Validity: 10y
#           Key: ECDSA P-384

rrchnm-root.crt is the public root certificate. rrchnm-root.key is the password-encrypted private key. Both stay on this machine for now; the cert leaves with the CSR-signing step later.

Step 2 — Generate the intermediate key + CSR (online machine)

cd ~/rrchnm-ca-intermediate

# Intermediate password — used by step-ca at runtime to decrypt this key.
# It lands in the SOPS-encrypted values file; pick a fresh passphrase
# (NOT the root password). We write TWO files atomically:
#   intermediate.pass     — plaintext, consumed by `step --password-file`
#   intermediate.pass.b64 — base64 form, ready-to-paste into intermediate_ca_key_password
# Doing both at write time avoids a class of bug where the operator
# generates one and re-derives the other later, with the two falling
# out of sync (or skipping the base64 step entirely).
stty -echo; printf 'Intermediate password: '; IFS= read -r INT_PW; stty echo; printf '\n'
printf '%s' "$INT_PW" > intermediate.pass
printf '%s' "$INT_PW" | base64 -w0 > intermediate.pass.b64
chmod 600 intermediate.pass intermediate.pass.b64
unset INT_PW

# Subject template for the CSR. Subject-only — issuer / keyUsage /
# basicConstraints come from `--profile intermediate-ca` at sign time
# (Step 3) and don't belong in the CSR. Same shell-quoting rule as the
# root-ca template.
cat > intermediate-csr.tpl <<'EOF'
{
  "subject": {
    "commonName": {{ toJson .Subject.CommonName }},
    "country": ["US"],
    "province": ["Virginia"],
    "locality": ["Fairfax"],
    "organization": ["Roy Rosenzweig Center for History and New Media"],
    "organizationalUnit": ["Infrastructure"]
  }
}
EOF

step certificate create \
  "RRCHNM Internal Intermediate CA" \
  rrchnm-intermediate.csr rrchnm-intermediate.key \
  --template intermediate-csr.tpl \
  --csr \
  --kty EC --crv P-384 \
  --password-file intermediate.pass

# Verify CSR before shipping it to the air-gapped machine
step certificate inspect rrchnm-intermediate.csr --format json | jq '.subject, .public_key.algorithm, .public_key.curve'

Outputs at this point:

• rrchnm-intermediate.csr — public, carry to the air-gapped machine on USB.
• rrchnm-intermediate.key — private, encrypted with intermediate.pass. Stays on this machine; will be consumed by the IaC step.
• intermediate.pass — the passphrase as plaintext. Stays on this machine; will be consumed by the IaC step.
• intermediate.pass.b64 — base64-encoded form of the passphrase. This is the value that goes into intermediate_ca_key_password in the SOPS file.

Step 3 — Sign the intermediate CSR (air-gapped machine)

Move rrchnm-intermediate.csr to the air-gapped machine via USB.

cd ~/rrchnm-ca-root

# rrchnm-intermediate.csr is in the working dir (copied off USB)
step certificate sign \
  rrchnm-intermediate.csr \
  rrchnm-root.crt rrchnm-root.key \
  --profile intermediate-ca \
  --not-after 17520h \
  --password-file root.pass \
  > rrchnm-intermediate.crt

# Verify the chain
step certificate verify rrchnm-intermediate.crt --roots rrchnm-root.crt
# Expected: (no output, exit 0)

step certificate inspect rrchnm-intermediate.crt --short
# Expected: Subject: RRCHNM Internal Intermediate CA
#           Issuer: RRCHNM Internal Root CA
#           Validity: 2y
#           Key Usage: Certificate Sign, CRL Sign
#           Basic Constraints: CA:TRUE, pathlen:0

Now move TWO files back to the online machine via USB:

• rrchnm-root.crt — the public root cert
• rrchnm-intermediate.crt — the public intermediate cert (just issued)

Leave behind on the air-gapped machine:

• rrchnm-root.key — proceed to step 4 for storage
• root.pass — proceed to step 4

Step 4 — Escrow the root key (air-gapped machine)

The root key must survive but never go online. Three storage layers:

1. Primary: encrypted USB stick. Recommended: LUKS-encrypted USB with a passphrase different from root.pass. Files on the USB: rrchnm-root.crt, rrchnm-root.key, root.pass. Label the USB physically with a date and the words “RRCHNM Root CA — do not discard.”

2. Backup: a second encrypted USB or sealed envelope in an offsite location (different physical building / safe). Same contents.

3. Recovery passphrase escrow: write the LUKS passphrase down on paper. Store the paper in a physically secure location, separate from both USBs. Whoever can read the paper can decrypt the USB; this is intentional and is your disaster-recovery story.

After both USBs are written and verified (mount, read back, compare SHA-256), shred the working copies on the air-gapped machine:

cd ~/rrchnm-ca-root
shred -u rrchnm-root.key root.pass rrchnm-intermediate.csr rrchnm-intermediate.crt
# rrchnm-root.crt and rrchnm-intermediate.crt are public; they exit with
# you on the carry-out USB. The .key and .pass are gone from disk.

If the air-gapped machine is a live USB / VM, simply power it off (disk image is throwaway) — but only after confirming the encrypted USBs are valid.

Step 5 — Stage materials for the IaC half (online machine)

You now have, on the online machine:

File	Sensitivity	Goes into
rrchnm-root.crt	Public	ansible/group_vars/all.yaml as ca_root_cert (plain text — root certs are by-design public)
rrchnm-intermediate.crt	Public	ansible/group_vars/step_ca_servers.yaml as ca_intermediate_cert (plain text — intermediate certs are public; only the key is secret)
rrchnm-intermediate.key	Secret (encrypted)	ansible/group_vars/step_ca_servers.sops.yaml under intermediate_ca_key (SOPS-encrypted via field-level encrypted_regex). Whole PEM including the -----BEGIN ENCRYPTED PRIVATE KEY----- / END lines.
intermediate.pass.b64	Secret (base64 of the passphrase)	ansible/group_vars/step_ca_servers.sops.yaml under intermediate_ca_key_password (SOPS-encrypted). Paste the file content verbatim — the setup-step-ca-servers.yaml playbook base64-decodes it before writing /etc/step-ca/secrets/intermediate_ca_password. Do NOT paste the plaintext intermediate.pass here; do NOT double-encode.

This layout reflects the 2026-05-21 move of step-ca out of k0s onto the heimdall IncusOS container. The earlier Helm-chart shape (k0s/platform/heimdall-values.sops.yaml with inject.certificates.* / inject.secrets.x509.* fields) is retired — see internal-ca “Why we moved out of k0s”.

The IaC commit that consumes these adds the setup-step-ca-servers.yaml playbook and the step-ca Make target. Don’t paste anything into the repo yet — wait until the IaC commit is up so you can sops edit the already-templated group_vars/step_ca_servers.sops.yaml. Pasting into a non-SOPS file (then adding SOPS later) leaves the secrets in git reflog essentially forever.

What you can do now to keep momentum: confirm both certs verify against the root before sending them anywhere.

# In the staging directory:
step certificate verify rrchnm-intermediate.crt --roots rrchnm-root.crt
step certificate inspect rrchnm-root.crt --short
step certificate inspect rrchnm-intermediate.crt --short

# Concatenate into the chain step-ca will serve as its `roots.pem`
# (step-ca needs root + intermediate as separate files; the chain
# concatenation is mainly a sanity convenience):
cat rrchnm-intermediate.crt rrchnm-root.crt > rrchnm-chain.pem
step certificate verify rrchnm-chain.pem --roots rrchnm-root.crt

When the IaC commit is in, sops ansible/group_vars/step_ca_servers.sops.yaml and populate the two fields, plus add the intermediate cert PEM to ansible/group_vars/step_ca_servers.yaml’s ca_intermediate_cert. After populating, shred the working copies of the intermediate key + password:

shred -u rrchnm-intermediate.key intermediate.pass intermediate.pass.b64
# Keep the .crt files (public); you'll likely want them again for
# debugging or to re-verify chains.

Step 6 — Smoke test (after the IaC half is applied)

Once make step-ca is up and step-ca is running on the heimdall container, verify the chain end-to-end:

# From inside the workspace container (or any host with the root cert
# trusted, which is everywhere the `init` role has run with the
# updated ca_root_cert):
curl -fsS https://heimdall.rrchnm.internal/health

# Without trust (using the staged chain):
curl --cacert ~/rrchnm-ca-intermediate/rrchnm-chain.pem \
  -fsS https://heimdall.rrchnm.internal/health
# Both should return: {"status":"ok"}

# step-ca presents its server cert; verify it's signed by our chain:
step ca health --ca-url https://heimdall.rrchnm.internal --root rrchnm-root.crt

If health checks pass and the chain verifies, Phase 1 is done. Proceed to Phase 2 (cert-manager) in a separate PR.

Rotation

Re-issuing the chain (root + intermediate, or intermediate only) uses the same playbook as the initial bootstrap — generate-ca-bundle.yaml generates a fresh chain into a timestamped ~/<TS>-rrchnm-ca-bundle/ dir each run. The mechanical work (Steps 1-3 above) is identical to initial bootstrap. What’s different is the downstream rollout — fleet trust bundle, SOPS staging, step-ca redeploy, cert-manager verification — which is the procedure documented in the playbook’s generated SUMMARY.txt and summarized below.

When to rotate

Trigger	Scope	Why
Routine 2-year intermediate expiry (same algorithm)	Intermediate only	Forced by validity. Re-run playbook, skip the “new root” steps.
Intermediate key suspected compromised	Intermediate only, immediate	Speed > caution. Tolerate brief chain mismatch.
Algorithm change (e.g. ECDSA → PQ)	Root + intermediate, parallel-overlap	Old clients can’t validate new-algorithm chains. See plan-step-ca-pq-migration.md (operator note).
10-year root expiry	Root + intermediate	Same as initial bootstrap — schedule pre-2036.
Defense-in-depth / opportunistic root rotation	Root + intermediate	Cheapest while the fleet trust footprint is small.

Two rotation patterns

Path A — in-place reprovision (default for ECDSA → ECDSA): re-run the playbook on the existing host (or via -e ca_host=heimdall), SOPS-edit group_vars/step_ca_servers.sops.yaml with the new intermediate materials, make step-ca. With 24h max cert validity, all leaf certs auto-renew through the new chain within ~16h. Chain bundle gap is short and self-healing. Rollback: paste the previous SOPS values back, re-run.

Path B — parallel overlap (for PQ migration, risk-averse rollouts, or any future >24h cert validity): activate the pre-staged forseti inventory entry as a second step-ca instance, both chaining to the same root; create a parallel forseti-clusterissuer.yaml; migrate Certificates one at a time via issuerRef.name; retire heimdall after all consumers move. Per-service rollback is one-field. Inventory reservation + step_ca_dnsnames per-host templating + multi-PEM ca_root_cert distribution are all already in place.

Multi-PEM trust bundle overlap (root rotation only)

The trust anchor — ca_root_cert in ansible/group_vars/all.yaml — is multi-PEM-aware by design. During root rotation, prepend the new root above the old root in ca_root_cert, push the bundle fleet-wide (ansible-playbook playbooks/pre-flight.yaml --tags ca-trust), wait for the overlap window (every cert needs to renew at least once through the new chain), then remove the old root and re-push. Without this, fleet hosts validating an existing leaf signed by the old chain against a trust bundle containing only the new root would fail.

Downstream procedure (after running the playbook)

The playbook’s SUMMARY.txt writes an 8-step ordered checklist:

1. Escrow new root materials (USBs) — root rotations only
2. SOPS-edit group_vars/step_ca_servers.sops.yaml (intermediate key + intermediate_ca_key_password)
3. Plain-edit group_vars/step_ca_servers.yaml (intermediate cert PEM)
4. Prepend new root to ca_root_cert in group_vars/all.yaml — root rotations only, multi-PEM
5. ansible-playbook playbooks/pre-flight.yaml --tags ca-trust to push trust bundle
6. make step-ca to redeploy step-ca with new intermediate
7. Watch cert-manager auto-renewals over 24h
8. Remove old root from ca_root_cert, re-push — root rotations only

Verify gates at each step are in the SUMMARY.txt itself. Stop-and-recover at any failure — see Failure modes below.

Operator ceremony that doesn’t automate

• Root password + intermediate password: prompted by the playbook, never written to env / history / process listing.
• Root key escrow: physical-world (encrypted USBs, offsite copy). Per Step 4 above.
• Cutover decision (which path, when): operator judgment based on the decision table above.

Appendix — scripted variants of Steps 1–3

The two scripts below are exact copy-pastes of the step invocations in Steps 1–3, wrapped for one-shot execution. They produce files in the current working directory (cd ~/rrchnm-ca-… per Prerequisites first). Passwords are read at runtime via stty -echo, never written to env or history; no set -x.

provision-certificates.sh covers Steps 1 + 2 in a single run — the “all-online” trade-off documented in Where to generate. For the split air-gapped/online flow, copy the script into two halves and run each on the appropriate machine.

#!/bin/sh
# provision-certificates.sh — Steps 1+2 combined.
# Generates rrchnm-root.{crt,key} + root.pass AND
# rrchnm-intermediate.{csr,key} + intermediate.pass{,.b64}
# in the current working directory.

set -eu

# ---- Step 1: Root CA ----
stty -echo; printf 'Root password: '; IFS= read -r ROOT_PW; stty echo; printf '\n'
printf '%s' "$ROOT_PW" > root.pass
chmod 600 root.pass
unset ROOT_PW

cat > root-ca.tpl <<'EOF'
{
  "subject": {
    "commonName": {{ toJson .Subject.CommonName }},
    "country": ["US"],
    "province": ["Virginia"],
    "locality": ["Fairfax"],
    "organization": ["Roy Rosenzweig Center for History and New Media"],
    "organizationalUnit": ["Infrastructure"]
  },
  "issuer": {{ toJson .Subject }},
  "keyUsage": ["certSign", "crlSign"],
  "basicConstraints": {
    "isCA": true,
    "maxPathLen": 1
  }
}
EOF

step certificate create \
  "RRCHNM Internal Root CA" \
  rrchnm-root.crt rrchnm-root.key \
  --template root-ca.tpl \
  --kty EC --crv P-384 \
  --not-after 87600h \
  --password-file root.pass

step certificate inspect rrchnm-root.crt --short

# ---- Step 2: Intermediate key + CSR ----
stty -echo; printf 'Intermediate password: '; IFS= read -r INT_PW; stty echo; printf '\n'
printf '%s' "$INT_PW" > intermediate.pass
printf '%s' "$INT_PW" | base64 -w0 > intermediate.pass.b64
chmod 600 intermediate.pass intermediate.pass.b64
unset INT_PW

cat > intermediate-csr.tpl <<'EOF'
{
  "subject": {
    "commonName": {{ toJson .Subject.CommonName }},
    "country": ["US"],
    "province": ["Virginia"],
    "locality": ["Fairfax"],
    "organization": ["Roy Rosenzweig Center for History and New Media"],
    "organizationalUnit": ["Infrastructure"]
  }
}
EOF

step certificate create \
  "RRCHNM Internal Intermediate CA" \
  rrchnm-intermediate.csr rrchnm-intermediate.key \
  --template intermediate-csr.tpl \
  --csr \
  --kty EC --crv P-384 \
  --password-file intermediate.pass

step certificate inspect rrchnm-intermediate.csr --format json \
  | jq '.subject, .public_key.algorithm, .public_key.curve'

sign-intermediate-csr.sh covers Step 3. Run it on the machine that holds the root key (air-gapped machine in the split flow; same dir as provision-certificates.sh in all-online mode), with the CSR present in the working dir.

#!/bin/sh
# sign-intermediate-csr.sh — Step 3.
# Signs rrchnm-intermediate.csr with the root CA, producing
# rrchnm-intermediate.crt. Expects in cwd:
#   rrchnm-root.crt, rrchnm-root.key, root.pass, rrchnm-intermediate.csr

set -eu

step certificate sign \
  rrchnm-intermediate.csr \
  rrchnm-root.crt rrchnm-root.key \
  --profile intermediate-ca \
  --not-after 17520h \
  --password-file root.pass \
  > rrchnm-intermediate.crt

step certificate verify rrchnm-intermediate.crt --roots rrchnm-root.crt
step certificate inspect rrchnm-intermediate.crt --short

Steps 4–6 (escrow, IaC staging, smoke test) remain manual — they involve physical media, the SOPS edit loop, and post-deploy verification, none of which compose cleanly into a script.

Failure modes and recovery

Symptom	Likely cause	Recovery
step certificate verify fails after signing	Wrong root.key used (multiple roots in working dir) or --profile intermediate-ca omitted (treated as leaf)	Re-run step 3 with the correct root and profile. The CSR is reusable.
Helm install fails or step-ca pod CrashLoopBackOff with “incorrect password”	intermediate_ca_key_password in values doesn’t match what the intermediate key was encrypted with. Common variants: (a) pasted intermediate.pass plaintext instead of intermediate.pass.b64; (b) regenerated the intermediate but reused the old intermediate_ca_key_password; (c) intermediate_ca_cert cert and key are from different signing rounds.	Verify three things in lockstep: the cert under intermediate_ca_cert (group_vars/step_ca_servers.yaml), the key under intermediate_ca_key and the base64-passphrase under intermediate_ca_key_password (sops edit group_vars/step_ca_servers.sops.yaml), all come from the SAME run of steps 2-3. Re-run make step-ca.
Browsers/curl on a fleet host don’t trust heimdall.rrchnm.internal	ca_root_cert not yet pushed to that host	Run ansible-playbook playbooks/setup-<group>-servers.yaml --tags ca-trust for the relevant group
Want to rotate the intermediate (e.g. annual cadence)	Normal operations	Repeat step 2-5 with a new intermediate. The root key escrow is reused. step-ca handles overlap if you stage the new intermediate before retiring the old.
Root key compromise (or suspected)	Catastrophic	Bring out the offsite backup root key (or generate a new one), sign a new intermediate, update ca_root_cert across the fleet, redistribute trust. Every existing leaf cert is suspect — force cert-manager to renew them all against the new intermediate. Procedure to be drafted as internal-ca-root-rotation.md before the situation arises.

Decision log

• Root key offline, intermediate online. Standard two-tier. Per design doc.
• Different passwords for root and intermediate. Root password never leaves the air-gapped machine; intermediate password lives in SOPS. Same password would mean compromise of either leaks the other.
• ECDSA P-384 for both root and intermediate. ~equivalent to RSA-7680 in classical strength, ~10× faster signing, ~half the cert size, universally supported on every stack we run (Linux, k0s, browsers, curl, Go, Python). RSA-4096 is an equally valid fallback if a specific client surfaces ECDSA-compat issues — just keep root and intermediate in the same family so the chain verifies cleanly. Neither algorithm is post-quantum resistant; the rotation path to a PQ root is documented separately in plan-step-ca-pq-migration.md (operator note) and the current design (multi-PEM ca_root_cert, additive Helm releases) is built to support that transition without rework.
• Scripts embedded in the doc, not committed under scripts/. The appendix provides provision-certificates.sh (Steps 1+2) and sign-intermediate-csr.sh (Step 3) as copy-paste shortcuts; they sit next to the prose so each command stays auditable in context and can’t drift from the runbook. Passwords are read at runtime via stty -echo, never written to env or history; no set -x. A standalone script in scripts/ would invite env-var-credential wrappers and risk silent divergence between code and runbook.
• Materials staged on the online machine before the SOPS file exists. Trade-off: lets the operator pre-generate while the IaC commit is still being written. Mitigation: shred working files immediately after the SOPS file is populated; keep them out of shell history (HISTCONTROL=ignorespace + leading-space invocations).