OpenBao ops

Operational SOP for the in-cluster OpenBao deployment (k0s/platform/openbao-values.yaml). Covers the 6-playbook ops chain under ansible/playbooks/k0s/ops/ plus the ESO consumer side.

The chart deploys OpenBao in standalone + file-storage mode (server.standalone.enabled: true, storage "file", tls_disable = 1 on the internal listener — TLS is terminated at the internal Cilium Gateway, not at the pod). There is no Raft, no KMS auto-unseal, no Transit auto-unseal. Every pod restart is a manual unseal. Auto-unseal is a separate planning track (TODO; not written yet).

Background on why standalone+file rather than Raft+integrated-storage: it’s the simplest mode and our scale doesn’t justify Raft yet. Switching to Raft later is a data-migration exercise (bao operator migrate), not an in-place toggle.

Concepts: master key, shares, root token, rekey token, ESO

The procedures below touch five distinct pieces of cryptographic and authentication material. They’re constantly conflated; the SOP only makes sense once they’re separated.

Master key. A single 256-bit AES key. Encrypts every secret on the PVC; nothing else decrypts those bytes. Generated once at bao operator init, never written to disk in plaintext, never seen by humans. Lives only in RAM, only while the vault is unsealed. The whole seal/unseal design exists to preserve “master key on disk = never” — if the PVC is stolen, the data is unreadable without the key, which is somewhere else entirely.

Unseal shares. 5 base64 strings (~88 chars each), each a polynomial point in Shamir’s Secret Sharing scheme over the master key. Any 3 of the 5 can mathematically reconstruct the master key; 2 or fewer reveal nothing usable. Produced at init time, printed once, never regenerated by OpenBao on its own. Humans paste them into bao operator unseal at every pod start; after 3 have been received OpenBao reconstructs the master key into RAM and the vault transitions to unsealed. The shares themselves are discarded from RAM after reconstruction.

In our setup all 5 shares live together in k0s/platform/secrets/openbao-keys.sops.yaml, SOPS-encrypted (whole-file) to both age recipients. That’s a deliberate trade — the math defends against partial-leak (“attacker gets 2 of 5 shares”) but in practice all 5 live in one file, so the threat model collapses to “attacker has the age key, or doesn’t.” The seal still protects against “attacker has the PVC but not the age key” — the realistic theft scenario for an on-prem cluster.

Root token. Issued once at init, printed once, revoked by configure-openbao.yaml after it mints the rekey_token. We deliberately don’t keep the root token around: OpenBao 2.5.4 disables sys/generate-root/attempt, so the post-revoke recovery path for a lost rekey token is full re-init (delete Helm release + PVC + re-run init/unseal/configure). That cost is acceptable in exchange for not having a god-mode credential sitting in the SOPS file long-term.

Rekey token. A scoped, periodic (720h), orphan token whose policy includes only path "sys/rotate/root/*" { capabilities = [..., "sudo"] }. Minted by configure-openbao.yaml immediately before revoking the root token. Replaces root_token in openbao-keys.sops.yaml. Sufficient for Shamir rotation (rekey-openbao.yaml), NOT for any other admin path. The reason this is an orphan token: parent revocation cascades, and we want the rekey token’s lifetime decoupled from any auth-method tree.

Kubernetes auth + KV v2 + ESO. configure-openbao.yaml also enables the kubernetes/ auth method (validates SA JWTs against the cluster’s TokenReview API), mounts a kv/ v2 secrets engine, writes four policies (admin, rekey, external-secrets-operator-eso, kv-writer), and creates two roles bound to specific ServiceAccounts:

external-secrets-operator — SA external-secrets.external-secrets, ttl 24h, policy external-secrets-operator-eso (read-only on kv/data/eso/*). Used by ESO to materialize Kubernetes Secrets from KV.
kv-writer — SA openbao-operator.platform, ttl 1h, policy kv-writer (write anywhere under kv/). Used by kv-patch.yaml to bump a single field without touching siblings, and kv-put.yaml for create-only seeding.

Lifecycle:

make k0s-openbao-init           (one time per PVC, ever)
    ├─▶ generates master key                (RAM only during init)
    ├─▶ splits into 5 shares                ─▶ openbao-keys.sops.yaml
    └─▶ generates root token                ─▶ openbao-keys.sops.yaml

make k0s-openbao-unseal         (every pod start)
    └─▶ reads 3 shares from SOPS, posts to /sys/unseal until unsealed

make k0s-openbao-configure      (one time, immediately after first init+unseal)
    ├─▶ enables kubernetes/ auth + kv/ v2
    ├─▶ writes 4 policies + 2 k8s auth roles + initial KV seeds
    ├─▶ mints rekey_token (orphan, periodic 720h, policy=rekey)
    ├─▶ atomically swaps root_token → rekey_token in openbao-keys.sops.yaml
    └─▶ revokes root_token

make k0s-eso                    (after configure)
    └─▶ deploys ESO + ClusterSecretStore + 3 ExternalSecrets

make k0s-openbao-rekey          (periodic; rotates Shamir shares)
    ├─▶ phase 1: /sys/rotate/root/init (new shares; old still valid)
    ├─▶ phase 2: atomic SOPS rewrite (tempfile + verify + mv -f)
    ├─▶ phase 3: /sys/rotate/root/verify (commits new; invalidates old)
    └─▶ token field unchanged (rekey_token preserved)

make k0s-openbao-kv-patch SVC=… FIELD=…   (any time, single field)
    └─▶ SA JWT (1h) → auth/kubernetes/login → bao kv patch
make k0s-openbao-kv-put                   (create-only seeding, idempotent)

Status

Reference procedure — in production use since 2026-05. All 5 playbooks under ansible/playbooks/k0s/ops/ are operational. The make k0s chain calls k0s-openbao-bootstrap (a non-interactive wrapper) between k0s-platform and k0s-eso so first-time bring-up + per-restart unseal happen as part of the normal pipeline. Operator-driven re-runs use the interactive Make targets below.

What gets touched

k0s/platform/secrets/openbao-keys.sops.yaml — in-repo, SOPS-encrypted whole-file (k0s/.sops.yaml rule 2). Both age recipients (init + operator).
- Pre-configure fields: unseal_keys_b64: [5], unseal_keys_hex: [5], unseal_shares: 5, unseal_threshold: 3, recovery_*: null, root_token: s.xxx.
- Post-configure fields: same as above except root_token is replaced by rekey_token: s.xxx.
- Created by init-openbao.yaml; mutated by configure-openbao.yaml and rekey-openbao.yaml (atomic temp-write + sops-encrypt + round-trip verify + mv -f).
k0s/platform/secrets/initial-kv-seeds.sops.yaml — in-repo, SOPS-encrypted whole-file. Top-level keys = service names (forgejo, grafana); inner dicts = { username, password } etc. Read at configure-openbao.yaml time to populate kv/eso/<svc> for any missing path (existing paths are not modified — that’s kv-patch.yaml’s job).
openbao PVC (openbao-data-openbao-0 in the platform namespace, 10 Gi on local-path) — bao operator init writes the encrypted master key + initial keyring to /openbao/data inside the pod. After this, the PVC is the source of truth for what the shares can decrypt; if the PVC is destroyed, the shares are useless.
openbao-0 pod’s in-memory seal state — toggled by bao operator unseal. Resets to sealed on every pod restart. Not persisted.
k8s API objects created by configure-openbao.yaml:
- Auth method: kubernetes/
- Secrets engine: kv/ (KV v2)
- 4 policies: admin / rekey / external-secrets-operator-eso / kv-writer
- 2 k8s auth roles: external-secrets-operator / kv-writer
ESO-materialized Secrets (created by deploy-eso.yaml, ongoing refresh interval 1m) — originally three (forgejo-admin, eso-test, grafana-admin); the roster has since grown to ~20. Enumerate live: kubectl get externalsecret -A.

Pre-flight checklist (interactive ops)

KUBECONFIG=/configs/k0s/config set in the workspace; kubectl get nodes returns Ready. (SOPS_AGE_KEY_FILE=/configs/sops/age/keys.txt is preset in the workspace Dockerfile.)
Helm release installed: helm -n platform list | grep openbao shows deployed.
For init: pod sealed + uninitialized — kubectl -n platform exec openbao-0 -- bao status returns exit code 2 and shows Initialized false, Sealed true.
For configure / rekey: working tree clean for k0s/platform/secrets/openbao-keys.sops.yaml (git status reports no uncommitted changes). Both playbooks abort otherwise to prevent overwriting in-progress edits. The bootstrap wrapper passes -e skip_git_check=true for the configure step only (CI/non-interactive use).
DNS resolves: dig +short bao.rrchnm.internal returns the internal Gateway IP (per ansible/vars/k0s-gateways.yaml). Not strictly required for the CLI-via-exec path, but confirms ingress is intact for UI use.

Procedures

All Make targets below run from the workspace container.

Bootstrap chain (non-interactive)

make k0s-openbao-bootstrap

The operative path during make k0s. Idempotent end-to-end:

Waits for openbao-0 to be Running.
Runs init-openbao.yaml with confirm=init only if bao status reports initialized=false.
Always runs unseal-openbao.yaml (skips already-unsealed pods).
Runs configure-openbao.yaml with confirm=configure -e skip_git_check=true only if the SOPS file’s token field is root_token (i.e., configure hasn’t run yet). After successful configure the file has rekey_token and re-runs short-circuit.

Leaves the SOPS file dirty for the operator to commit manually (intentional — the operator should review the diff before pushing).

Initial init (one-shot)

make k0s-openbao-init

Equivalent to ansible-playbook playbooks/k0s/ops/init-openbao.yaml -e confirm=init. Prompts for literal init confirmation.

What happens:

Pre-flight: asserts openbao-0 is Running; asserts bao status returns initialized=false; asserts openbao-keys.sops.yaml does not exist.
bao operator init -key-shares=5 -key-threshold=3 -format=json against openbao-0.
Parses JSON output → renders YAML → sops --encrypt --in-place with chdir: k0s/ so SOPS walks up to k0s/.sops.yaml and resolves rule 2.

Never re-run against an existing PVC — the playbook’s pre-flight prevents this, but if you bypass the gate manually you’ll get Vault is already initialized from bao and a dangling file write.

Initial unseal (recurring)

make k0s-openbao-unseal

Equivalent to ansible-playbook playbooks/k0s/ops/unseal-openbao.yaml. No confirmation prompt — fully idempotent and safe to re-run at any time.

What happens:

Reads unseal shares from openbao-keys.sops.yaml via community.sops.sops lookup.
Discovers every openbao server pod (label=app.kubernetes.io/name=openbao,component=server).
For each pod (sorted), checks bao status; if sealed=true, posts unseal_threshold (=3) shares to /sys/unseal via kubectl exec.

Run this after any pod restart — make k0s reapply, node drain/uncordon, OOMKill, manual kubectl delete pod openbao-0, etc. The bootstrap wrapper invokes it on every make k0s run.

Configure (one-shot, immediately after first init+unseal)

make k0s-openbao-configure

Equivalent to ansible-playbook playbooks/k0s/ops/configure-openbao.yaml -e confirm=configure. Prompts for literal configure confirmation; aborts if openbao-keys.sops.yaml is dirty in git (unless -e skip_git_check=true).

What happens (in order):

Authenticates as the current root_token.
Enables kubernetes/ auth method + kv/ v2 secrets engine.
Writes 4 policies (admin, rekey, external-secrets-operator-eso, kv-writer).
Creates 2 k8s auth roles (external-secrets-operator, kv-writer) bound to specific SAs.
Seeds kv/eso/eso-test with {foo, timestamp} for ESO smoke-testing.
Loops initial-kv-seeds.sops.yaml: for each service whose kv/eso/<svc> does NOT exist, writes the seed values. Existing paths are not modified (this is by design — once a service is up, use kv-patch.yaml to change a field).
Mints rekey_token: bao token create -policy=rekey -orphan -period=720h -no-default-policy.
Atomic SOPS file mutation: write openbao-keys.new.sops.yaml (token field renamed root_token → rekey_token) → sops --encrypt --in-place (chdir k0s/) → SOPS round-trip verify → mv -f over the original.
bao token revoke -self using the captured root_token.

Side effect: the root token is now invalid. Re-running this playbook fails (no valid root token to authenticate with). For policy/auth changes after configure has run, the path is full re-init (delete Helm release + PVC + re-run init/unseal/configure). This is why operator changes to policy/role definitions should be made in the playbook BEFORE first configure on a given PVC.

Rekey (periodic; rotates Shamir shares)

make k0s-openbao-rekey

Equivalent to ansible-playbook playbooks/k0s/ops/rekey-openbao.yaml -e confirm=rekey. Prompts for literal rekey confirmation; aborts on uncommitted openbao-keys.sops.yaml (no skip-git-check escape hatch — rekey must be a deliberate operator action).

What happens (3-phase via OpenBao 2.5.0+ sys/rotate/root/{init,update,verify} API with require_verification=true):

Pre-tasks: cancels any pending rotation (bao delete sys/rotate/root/init). Authenticates as the current token from the SOPS file.
Phase 1 (init): posts {secret_shares: 5, secret_threshold: 3, require_verification: true} to /sys/rotate/root/init. OpenBao returns a nonce + new shares; old shares remain valid for unseal at this point.
Phase 2 (atomic persist): writes openbao-keys.new.sops.yaml with new shares + same token → sops --encrypt --in-place → SOPS round-trip verify (decrypt the new file and re-parse) → mv -f over original. Failure before mv -f leaves the original untouched.
Phase 3 (verify): posts new shares to /sys/rotate/root/verify. OpenBao commits the new shares and invalidates the old. Past this point, only the new shares can unseal the pod.
Confirms bao status still shows sealed=false.

The token field (rekey_token or root_token) is preserved. Rekey rotates only the Shamir shares — it doesn’t touch the master key, the root/rekey token, or anything in kv/.

Default cadence: ~annually, or after any operator with share-knowledge leaves. The 3-phase verification mode means a network blip mid-rekey doesn’t strand the cluster (old shares stay valid until phase 3 commits).

Rotate or seed a single KV value

make k0s-openbao-kv-patch SVC=<service> FIELD=<field>   # change ONE field on an existing path
make k0s-openbao-kv-put                                  # create-only seeding from initial-kv-seeds (idempotent)
make k0s-openbao-kv-reseed                               # re-assert seeds
make k0s-openbao-kv-shell                                # interactive kv-writer shell

kv-patch (playbook ops/kv-patch.yaml) prompts for the new value interactively — no value= Make var by design, keeping the secret out of make argv and shell history. What happens:

Mints a 1h SA token for openbao-operator.platform via kubectl create token.
bao login -method=kubernetes role=kv-writer jwt=<the SA token> — gets a kv-writer-policy token.
bao kv list kv/eso/ to discover existing services (printed for operator reference).
Prompts for value (single field; one value at a time).
bao kv patch kv/eso/<svc> <field>=<value> — patches the single field, preserves siblings.
Reads back bao kv metadata get kv/eso/<svc> to confirm current_version bumped by 1.

kv-put (playbook ops/kv-put.yaml) is the create-only sibling: it writes any path in initial-kv-seeds.sops.yaml that does not yet exist, and never touches existing paths. The playbook headers are authoritative for flags.

ESO refresh. Services consuming their ExternalSecret via env-var injection get the materialized Secret within the refresh interval, but the pod needs kubectl rollout restart to pick up the new env var.

Token scope: the kv-writer policy can write under kv/ but cannot read sys/*, other auth methods, etc.

Recovery scenarios

Lost `openbao-keys.sops.yaml`, vault is initialized

Catastrophic — the master key can’t be reconstructed without 3 of the 5 shares. Options:

Decrypt from a recent borg snapshot of the operator-side /configs mount (if SOPS recipients are in the same age key as the snapshot). If recoverable, git checkout or copy the file back, then make k0s-openbao-unseal should work.
If the file is genuinely gone: tear down the PVC and re-init. Run helm -n platform uninstall openbao, kubectl delete pvc -n platform openbao-data-openbao-0, then re-run make k0s-platform → make k0s-openbao-bootstrap. You lose every secret in kv/ — they have to be re-seeded from operator memory / external sources. ESO materializations re-sync on next refresh once configure-openbao.yaml re-seeds via initial-kv-seeds.sops.yaml.

`openbao-keys.sops.yaml` exists but a share is corrupted

Less catastrophic — you have 5 shares and only need 3 for unseal. If one is malformed (truncated, base64-invalid), edit the file via sops openbao-keys.sops.yaml and verify the others by manually attempting bao operator unseal <share> with shares 2/3/4 (or whatever subset avoids the suspect one). If you can unseal with a 3-share subset, the spare shares can stay broken until next rekey, which regenerates all 5.

If 3 or more shares are corrupted, treat as “lost SOPS file” above.

Lost `rekey_token`, configure has already run

Per the configure rationale: full re-init. OpenBao 2.5.4 disables sys/generate-root/attempt, so there’s no way to mint a new root token from the shares. Tear down + recreate the PVC as above.

This is a deliberate cost — we prefer the brutality of “tear down + reseed” over a permanent god-mode credential. Rotate keep-around tokens by re-running configure-openbao.yaml… except we can’t, because configure assumes the root token. So in practice the rekey-token’s lifespan is “until the PVC dies” and that’s fine.

Verification commands

# Status overall
kubectl -n platform exec openbao-0 -- bao status
# Expect: Initialized true, Sealed false, Version 2.5.x, HA Enabled false

# Authenticate with the rekey_token (post-configure)
sops -d k0s/platform/secrets/openbao-keys.sops.yaml | yq -r .rekey_token | \
  xargs -I% kubectl -n platform exec openbao-0 -- env BAO_TOKEN=% bao token lookup -self
# Expect: policies=[rekey], orphan=true, period=720h

# List KV paths (using the kv-writer SA token)
TOKEN=$(kubectl -n platform create token openbao-operator --duration=1h)
kubectl -n platform exec openbao-0 -- env BAO_TOKEN= bao login -method=kubernetes role=kv-writer jwt=$TOKEN
# Expect a token with policies=[kv-writer]; then:
kubectl -n platform exec openbao-0 -- bao kv list kv/eso/
# Expect: eso-test, forgejo, grafana (at minimum)

# ESO materialization
kubectl -n platform get externalsecret
# Expect: forgejo-admin, eso-test — both SecretSynced=True
kubectl -n observability get externalsecret
# Expect: grafana-admin SecretSynced=True

# Forgejo admin secret content (proves the kv → ESO path end-to-end)
kubectl -n platform get secret forgejo-admin -o jsonpath='{.data.username}' | base64 -d
# Expect the value in initial-kv-seeds.sops.yaml's forgejo.username field

Rollback

There is no “undo” for these operations. Specific paths:

init-openbao.yaml failed mid-write: if the SOPS file was created with partial content, delete it manually + tear down the PVC + re-init. Don’t try to repair a partial file.
configure-openbao.yaml failed after step 7 (rekey_token minted) but before step 9 (root token revoke): both tokens are valid; the SOPS file may or may not have been mutated. Inspect the file; if it has rekey_token, manually revoke root_token via kubectl exec openbao-0 -- bao token revoke <root>. If it has root_token, the rekey token is orphaned and harmless (it’ll expire in 720h).
rekey-openbao.yaml failed between phase 2 (file mutated) and phase 3 (verify): the SOPS file now has new shares, but the vault still uses the old shares for unseal until phase 3 commits. Re-run rekey-openbao.yaml — its pre-tasks cancel pending rotation; the next run starts a fresh 3-phase cycle. If shares are completely confused, restore the SOPS file from git (it was clean before the run thanks to the dirty-tree check) and re-run.