Internal CA (heimdall)

Design record — as-built May 2026, with drift

Key-custody specifics (key storage locations, escrow, backup paths) are deliberately operator-side; the bootstrap/rotation ceremony is published at ca-bootstrap. Known drift since writing: IPs have moved (heimdall, arceus, azelf per fleet.yaml; uxie now on atomsk); the letsencrypt ClusterIssuer is now production with a DNS-01/RFC2136 solver (not staging/Phase-D-gated) and the public edge sits behind the k0s HAProxy tier; heimdall has since gained a JWK provisioner (Kanidm HA cert issuance). The architecture and decision log remain operative.

This document captures the design for an internal CA that issues TLS certificates for services inside the infrastructure: IncusOS hosts (currently presenting self-signed certs), k0s-hosted services with internal-only hostnames, eventual mTLS between services. Pairs tightly with internal-dns — DNS resolves the names, the CA signs the certs that prove identity.

Status (2026-05-29)

Internal DNS is live — prerequisite for everything below. Knot DNS 3.5.x on arceus (hyperion) is authoritative for rrchnm.internal + the dynamic acme.rrchnm.internal subzone (RFC2136 + TSIG). CoreDNS 1.14.3 on the three Lake Guardian resolvers (mesprit kyojin, azelf theia, uxie atomsk) stub-forwards to arceus and forwards everything else to campus DNS. Fleet VMs resolve *.rrchnm.internal correctly. See internal-dns for the as-built picture.

The CA moved out of k0s on 2026-05-21. step-ca runs on the heimdall IncusOS container on hyperion. cert-manager v1.20.2 in k0s talks ACME to it at the external URL https://heimdall.rrchnm.internal/acme/acme/directory. See the operator changelog’s 2026-05-21 entry for the move + landmines.

Phase status:

Phase	State	Notes
1: stand up step-ca with offline root	Done 2026-05-17; re-architected to VM 2026-05-21	Root + intermediate (10y / 2y validity, ECDSA P-384, root 2026-05-17 → 2036-05-14, intermediate 2026-05-17 → 2028-05-16). Encrypted intermediate key + passphrase live SOPS-encrypted in the systems repo (exact location operator-side). Plaintext intermediate cert PEM in ansible/group_vars/step_ca_servers.yaml ca_intermediate_cert. Plaintext root PEM in ansible/group_vars/all.yaml ca_root_cert (renamed init_ca_cert → init_ca_root_cert 2026-05-19, then → ca_root_cert 2026-05-20 for clarity vs intermediate). Bring-up runbook: ca-bootstrap. Run by ansible/playbooks/setup-step-ca-servers.yaml (make step-ca); installs step-ca + step-cli from the smallstep apt repo, drops a self-managed systemd unit with AmbientCapabilities=CAP_NET_BIND_SERVICE so the step user binds :443. Single ACME provisioner named acme (24h max/default leaf TTL).
2: cert-manager integration	Done 2026-05-17; manifests moved to k0s/cert-manager/ 2026-05-23	cert-manager v1.20.2 deployed via ansible/playbooks/k0s/deploy-cert-manager.yaml. Admission webhook intentionally disabled (webhook.replicaCount: 0 + VWC/MWC patched to failurePolicy: Ignore); controller still issues + renews normally; admission-time validation moves to CI-side kubeconform. Background: cilium-gateway-l7 § Retiring the cert-manager hostNetwork pin. Two ClusterIssuers in k0s/cert-manager/: heimdall (ACME → step-ca on the VM) and letsencrypt (ACME → LE staging today; flip to prod after Phase D coordination). The heimdall ClusterIssuer carries three solvers: DNS-01 RFC2136 against knot.rrchnm.internal:53 (selector-gated to test.rrchnm.internal during incremental migration), HTTP-01 default → internal Gateway, HTTP-01 → devl-vdi Gateway.
3: roll out to services	Done	Cert internal-tls (k0s/cert-manager/internal-tls-cert.yaml) issued by heimdall covers forge / headlamp / bao / grafana / hubble (5 SANs, 90d duration / 15d renewBefore, ECDSA P-256). The internal Gateway listener on :443 references this Secret. Cert devl-tls (k0s/vdi/devl-tls-cert.yaml) covers juice / alpine / arch / debian / kali for the devl-vdi Gateway. heimdall.rrchnm.internal + ca.rrchnm.internal are deliberately NOT in the internal-tls SAN — step-ca-on-VM serves its own cert directly.
4: distribution to non-cluster consumers	Fleet: done 2026-05-17. Off-fleet: pattern documented 2026-05-19.	init Ansible role’s ca-trust task drops ca_root_cert at init_ca_cert_path = /usr/local/share/ca-certificates/rrchnm-internal-ca.crt on every fleet host and runs update-ca-certificates. As of the operator changelog’s 2026-05-17 “(latest)” entry, the role also runs at pre-flight, so k0s VMs receive the cert too. Verified across both sites + both roles: jean (RH461 controller), sasha (RH461 worker), odin (AQ114 controller). Off-fleet ACME consumers (XCP-ng-side services, third-party hosts) follow acme-off-fleet-clients — bind-mounted root cert + Caddy trusted_roots-style directive + dns_zones A record + HTTP-01. Canonical worked example: kuvasz cert, 2026-05-19.
5: DNS-01 wildcards / public certs / mTLS	DNS-01: plumbed, selector-gated. Public LE: applied, Phase-D gated. mTLS: future scope.	DNS-01 RFC2136 against arceus (TSIG acme-update.rrchnm.internal. HMACSHA256) is wired into the heimdall ClusterIssuer; the rfc2136-tsig Secret in cert-manager ns is materialized at deploy time from group_vars/dns_authoritatives.sops.yaml’s knot_acme_update_tsig_secret. Solver selector currently scopes to dnsNames: [test.rrchnm.internal] only — flip per-Certificate as migration proceeds. Public TLS (k0s/cert-manager/public-tls-cert.yaml) is live — the LE cert issues via DNS-01 and the public Gateway serves HTTPS through the HAProxy tier (see k0s public edge). mTLS uses the per-Deployment Certificate pattern (decided 2026-05-17, not csi-driver-spiffe and not service mesh) — adjacent install: trust-manager (Jetstack sibling).

Why this matters

Three current pain points:

1. IncusOS hosts present self-signed certs. kyojin and theia regenerate their TLS server certs on factory-reset; the SAN list includes only the hostname and loopback (not the LAN IP), and nothing trusts those certs without manual --accept-certificate. See incus-factory-reset. An internal CA whose root is pre-trusted on operator machines and inside pods would close this — IncusOS host certs become trustworthy, no fingerprint dance.
2. Internal services have no proper TLS story. Forgejo, Grafana, Headlamp, OpenBao, and other services exposed via Gateway API at *.rrchnm.internal need certs. Today they likely have self-signed or no TLS at all. Public CAs (Let’s Encrypt) won’t issue for .internal names. We need our own.
3. Future mTLS between services. Once we want service identity (services authenticate as themselves to each other, not via shared API keys), we need a CA that can issue short-lived workload certs.

Architecture: two-tier CA

Standard layout:

┌─────────────────────────────────────────────────────────┐
│  Root CA   (offline, long-lived, signs intermediate)     │
│  - 10-20 year validity                                   │
│  - Private key never online                              │
│  - Used only when rotating intermediate                  │
└─────────────┬───────────────────────────────────────────┘
              │ signs
              ▼
┌─────────────────────────────────────────────────────────┐
│  Intermediate CA   (online, signs leaf certs)            │
│  - 1-3 year validity                                     │
│  - Runs as a service in our infra                        │
│  - Issues leaf certs via ACME                            │
└─────────────┬───────────────────────────────────────────┘
              │ signs
              ▼
┌─────────────────────────────────────────────────────────┐
│  Leaf certs   (services, hosts, eventually mTLS clients) │
│  - Short-lived (1-90 days)                               │
│  - Auto-renewed via ACME / cert-manager                  │
│  - Per-service, per-host                                 │
└─────────────────────────────────────────────────────────┘

The two-tier model exists because:

• Compromise containment. If the intermediate is compromised, you revoke it and reissue from the offline root. If the root were online and compromised, every cert it ever signed is suspect and you start over.
• Convenience. Root signs once (the intermediate’s cert). Intermediate signs many times (every leaf). Root key stays offline; intermediate key is online but lower-stakes.

For a small prototype, you could use a single-tier CA and move to two-tier later. But the migration is painful (every leaf needs to chain through a different parent), so it’s better to start two-tier even if the root isn’t perfectly offline yet.

Software choice: smallstep step-ca

Software	Strengths	Weaknesses
smallstep step-ca	ACME native, small, modern, scriptable, good docs, supports passwords/JWT/X5C provisioners	Smaller community than older alternatives
HashiCorp Vault PKI engine	Powerful, integrates with OpenBao (we already plan to deploy Bao)	Heavyweight if PKI is the only Bao use case; ACME via vault-pki-monitor-issuer is third-party
BIND-style cfssl (Cloudflare)	Lightweight, Go-native	No ACME server (cfssl issues, doesn’t speak ACME)
OpenSSL + scripts	Universal, no dependencies	High operational burden — scripts for rotation, no ACME, no audit
Easy-RSA	Simple, well-known (OpenVPN ecosystem)	Manual; not designed for high-volume issuance

Recommendation: step-ca. Native ACME server is the deciding factor. cert-manager in k0s talks ACME → step-ca → certs issued, completely automated. step-ca is also fine for issuing IncusOS host certs via direct API (not just ACME).

If we end up deploying OpenBao for secrets anyway, the Vault PKI engine is a respectable alternative — Bao becomes the CA in addition to its other roles. Lower service count overall. But the Bao PKI ergonomics are a step worse than step-ca for ACME, and the failure-domain coupling (Bao seal = CA outage) is real. Defer this question; for now, plan step-ca and reconsider if Bao PKI matures.

Where to host (as-built since 2026-05-21)

Component	Where	Why
Root CA private key	Offline escrow (specifics operator-side)	Compromise = catastrophic; never online
Root CA cert	Distributed everywhere (operator machines, in-cluster trust bundles)	Public-by-design
Intermediate CA	heimdall IncusOS container on hyperion	See “Why moved out of k0s” below
Leaf certs	Issued to services on demand via ACME (cert-manager in k0s for in-cluster; off-fleet clients per acme-off-fleet-clients.md)	Short-lived, auto-renewed

The original plan placed the intermediate CA in k0s as a Helm release. Moved to a dedicated IncusOS container on 2026-05-21. See changelog for the migration sequence.

Why we moved out of k0s

Three forcing functions, in increasing weight:

1. GEP-1897 (BackendTLSPolicy) was unimplemented in Cilium at the time of the move. Confirmed empirically on 1.16.5, 1.17.16, 1.18.10, 1.19.4 against the in-cluster heimdall. Without it, the Cilium Gateway couldn’t re-encrypt to an HTTPS-only backend — step-ca is HTTPS-only, so a Gateway-fronted https://heimdall.rrchnm.internal/ returned HTTP 400 — Client sent an HTTP request to an HTTPS server from Envoy. This blocked every external ACME client (XCP-ng Caddies, future fleet members) from using the in-k0s CA. The single biggest driver. The canonical CFP cilium/cilium#31352 has since closed (presumably implemented in a post-1.19 release; not re-tested), but the heimdall-on-VM decision remains correct on the merits of points 2 and 3 below.
2. CA outage coupled to cluster outage. Single-replica Deployment + k0s control-plane stalls = fleet-wide cert-renewal halt. The 24h heimdall ACME provisioner cap makes the practical window tight.
3. Architectural mismatch. A single binary serving HTTPS doesn’t need k0s + Helm + Service + Gateway routing.

VM specs (current)

Property	Value	Rationale
Hostname	heimdall	Norse Aesir gatekeeper — on-theme for the hyperion Aesir trio (odin/thor/loki)
Site	AQ114	Reliability bias — matches the cluster controller bias
Incus host	hyperion	Theme fit + plenty of pool headroom (zfs-raidz1 ~5.4TB)
IP	static AQ114 slot, cloud-init network-config (current address: fleet.yaml)	Free slot near other special-purpose containers
Type	Container, not VM	step-ca is one Go binary; matches the azelf/mesprit/ho-oh/lugia pattern; sub-second startup
Resources	1 vCPU, 512MiB RAM, 5GiB disk	step-ca is tiny
Pool	hyperion_instances	Standard for hyperion-hosted instances
Bind	:443 via AmbientCapabilities=CAP_NET_BIND_SERVICE on the systemd unit	Step-ca runs as unprivileged step user but binds the privileged port — matches the unprivileged-CoreDNS-binding-53 pattern

Single-replica IncusOS container is fine here

The whole CA service is one running container. If hyperion reboots, the container restarts. If hyperion dies, recreate via TF + Ansible against the same SOPS-encrypted intermediate key + passphrase — same intermediate identity, no fleet-trust disruption.

Failure modes:

Failure	Impact	TTR	Mitigation
heimdall briefly down (reboot, snapshot restore)	New certs queue; cert-manager retries with backoff. Existing leaves valid until expiry.	minutes	None — ride out
hyperion hardware loss	Fleet certs expire over hours-to-days (renewBefore window)	hours	Recreate VM from TF + Ansible against same SOPS material; same intermediate identity → fleet trust unchanged
Intermediate key compromise	Reissue every cert under a new intermediate	hours	Root key in offline escrow signs a fresh intermediate (procedure: ca-bootstrap); restart step-ca with new materials; fleet root trust unchanged
Root key compromise	Full re-trust required across fleet	days	Out of scope — root key offline by design
step-ca BoltDB corruption	ACME account + issued-cert ledger lost; existing certs still valid (no revocation list)	minutes	Restore from snapshot or accept fresh state

You get 30-60 days of warning before the first cert actually expires. Plenty of time to restore from backup or rebuild from the offline root.

True multi-replica HA (active-active step-ca) requires a shared DB backend + HSM/KMS for the intermediate key + leader election. Don’t pursue until renewal failures during planned maintenance become unacceptable.

Backup discipline: heimdall is in the fleet backup flow; the CA’s encrypted key material + state directory go off-host nightly (paths operator-side).

What this didn’t solve

• mTLS for all Gateway-fronted services — still needs BackendTLSPolicy in Cilium (#30443) and Gateway listener tls.mode: Mutual. Independent effort.
• HA for the CA itself — single-replica, by choice.
• Public-tier cert path — Let’s Encrypt remains the issuer for *.rrchnm.org; heimdall is internal-tier only.
• Public-vs-internal Gateway split — internal Gateway stays in place for *.rrchnm.internal services like forge/grafana that DO terminate at Envoy. Only the heimdall hostname (HTTPS-upstream-only) sits outside the Gateway path.

Service hostname: heimdall.rrchnm.internal

The CA service itself runs at heimdall.rrchnm.internal — Heimdall, the Aesir gatekeeper of Bifrost, the god whose job is to see and verify everyone crossing between worlds. Semantically the tightest match in the fleet for a credential-issuing authority, and it extends the Norse trust/wisdom subtheme already forming with odin/thor/loki (Aesir trinity on hyperion) and the planned mimir (ops center, the keeper of wisdom whose well sits at Yggdrasil’s roots).

A ca CNAME pointing at heimdall is also worth adding to dns_service_aliases — short, memorable, and lets configuration that wants a generic name (https://ca.rrchnm.internal/acme/...) work without baking the Norse name into every ClusterIssuer manifest. Both names resolve to the same Gateway LB IP.

Naming and certificate scope

The CA issues for two main namespaces:

Scope	Examples	Issued by
*.rrchnm.internal	kyojin.rrchnm.internal, theia.rrchnm.internal, forgejo.rrchnm.internal, grafana.rrchnm.internal	Internal CA (this CA)
*.rrchnm.org (publicly-resolvable)	forgejo.rrchnm.org (if exposed externally)	Let’s Encrypt (public CA) — separate path, cert-manager ClusterIssuer

Don’t have the internal CA issue for public hostnames; it can’t be trusted by browsers outside our infra. Likewise, Let’s Encrypt won’t issue for .internal. The two CAs serve different namespaces.

Both can coexist in cert-manager — separate ClusterIssuers, services pick which to use based on their hostname.

Distribution of root CA cert

Anyone who needs to verify internal certs needs the root cert in their trust store:

Place	How
Operator machines (this jump box, laptops)	Manual install into the system trust store; document in onboarding
k0s nodes	Ansible role copies root cert to /usr/local/share/ca-certificates/ and runs update-ca-certificates
k0s pods	ConfigMap with the root cert, mounted at /etc/ssl/certs/ (or merged into the trust bundle); pods opting in via volumeMounts
IncusOS hosts	TBD — IncusOS may not allow custom CA installation. Workaround: distribute via cloud-init for VMs but accept that the IncusOS daemon on bare-metal validates against its own trust store. May require IncusOS image rebuild to bake in.
Operator browsers	Manual install to validate Headlamp / Grafana / Forgejo internal hostnames

The IncusOS bare-metal trust store is the awkward case. For our use of those hosts, they don’t typically verify outbound TLS to internal services — they’re servers, not clients. So this may be a non-issue in practice. If it becomes one, custom IncusOS image build is the path.

ACME automation flow

Once step-ca is up:

┌────────────────────────────────────────────────┐
│  cert-manager (k0s)                            │
│  - ClusterIssuer "internal-ca": ACME → step-ca │
│  - ClusterIssuer "letsencrypt": ACME → LE      │
└────────────┬───────────────────────────────────┘
             │ when a Certificate resource exists
             ▼
┌────────────────────────────────────────────────┐
│  cert-manager requests cert from step-ca       │
│  - Solves http-01 or dns-01 challenge          │
│  - Receives signed cert                        │
│  - Stores in Secret                            │
│  - Auto-renews before expiry                   │
└────────────┬───────────────────────────────────┘
             │
             ▼
┌────────────────────────────────────────────────┐
│  Service (Forgejo, Grafana, etc.)              │
│  - References Secret in its Gateway / Ingress  │
│  - Gets internal CA-signed cert                │
└────────────────────────────────────────────────┘

Per-service definition is concise — a single Certificate CR per workload that wants TLS:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: forgejo-tls
  namespace: platform
spec:
  secretName: forgejo-tls
  issuerRef: { name: internal-ca, kind: ClusterIssuer }
  dnsNames: [forgejo.rrchnm.internal]
  duration: 90d         # 3 months
  renewBefore: 720h     # renew 30 days before expiry

cert-manager handles the rest — solving challenges, receiving certs, auto-renewing, updating the Secret.

Challenge type: dns-01 vs http-01

Challenge	How it works	Best for
http-01	cert-manager serves a token over HTTP on the requesting hostname; CA fetches it	Services with HTTP exposed; standard Gateway API setup
dns-01	cert-manager creates a TXT record in DNS; CA queries DNS	Wildcard certs; non-HTTP services; works behind NAT

For our services exposed via Gateway API (Forgejo, Grafana, etc.), http-01 is simpler. For *.rrchnm.internal wildcard or non-HTTP services (e.g. databases needing TLS), use dns-01 — requires the internal DNS to support dynamic updates (CoreDNS+etcd backend, or a tsig-key authenticated update endpoint). As of 2026-05-14 the internal DNS pair runs CoreDNS with the file plugin (static zones rendered by Ansible from inventory.yaml), which doesn’t support dynamic updates. dns-01 would therefore require the switch to etcd-backed CoreDNS described in internal-dns Phase 3 — currently deferred until ExternalDNS demand justifies it.

Plan: start with http-01, add dns-01 when wildcard or non-HTTP demands it (and pair the migration with internal-dns.md Phase 3 since the two share an etcd backend).

IncusOS hosts: less awkward than we thought

IncusOS regenerates its server cert on factory-reset with a SAN list of <hostname>.<domain> + loopback only — the LAN IP is not included. In practice this only matters for the very first incus remote add call (Go’s TLS stack enforces SAN match before the cert can be pinned). After remote-add succeeds, the cert is pinned in /configs/incus/servercerts/<remote>.crt and Incus client validates by fingerprint, not SAN — so subsequent connections work via IP-literal URLs without any /etc/hosts entries. See incus-factory-reset.

So the migration story for IncusOS hosts under an internal CA is only about cleaning up that one-time remote add friction. Two paths if we ever want to:

1. Have IncusOS use a cert from our internal CA at install time. The Incus seed template (ansible/playbooks/incus/templates/incus.yaml.j2) supports preseed.certificates for trusted client certs. Whether it supports providing a server cert is unclear from the docs — needs research. If it does, we’d use step-ca to issue a host cert at install time and feed it into the seed via group_vars. Solves the SAN problem cleanly, makes incus remote add work without any DNS/hosts dance.

2. Live with self-signed + fingerprint pinning (current state, works fine). The “Easiest” workflow in the factory-reset doc — add the remote once via hostname URL (resolve via campus DNS or a transient /etc/hosts line), then switch the remote’s addr to the IP literal. After that, no DNS or /etc/hosts is needed for ongoing operation.

Option 2 is what this codebase actually does in practice. Option 1 would be a nice-to-have if step-ca is standing already, but it’s not a blocker — fingerprint pinning is a perfectly fine trust model for the IncusOS host count we have.

Phased plan

Phase 1: stand up step-ca with offline root

Unblocked as of 2026-05-14 — internal DNS is live, so the step-ca service hostname (heimdall.rrchnm.internal) will resolve from the moment it’s deployed.

1. Generate root CA on an air-gapped machine (or carefully on this jump box, then move the private key into offline escrow)
2. Issue intermediate CA cert from root
3. Deploy step-ca in k0s with the intermediate (root key NOT present); add a heimdall A record (pinned to the Gateway LB IP) plus a ca CNAME → heimdall for the generic alias, both via the dns_service_aliases map in group_vars/dns_resolvers.yaml (rendered into playbooks/dns/templates/coredns.zone.j2)
4. Distribute root cert to operator trust stores (k0s-node + non-cluster-host distribution is already handled by the init Ansible role — see Phase 4)

Phase 2: cert-manager integration

1. Install cert-manager in k0s (Helm chart)
2. Configure two ClusterIssuers: internal-ca (ACME → step-ca) and letsencrypt (ACME → LE; if/when we expose anything publicly)
3. Issue first leaf cert as a smoke test (e.g. for a test pod)

Phase 3: roll out to services

For each Gateway API-exposed service: add a Certificate resource referencing internal-ca, point Gateway/HTTPRoute at the resulting Secret. Order:

1. Forgejo (high-traffic)
2. Grafana
3. Headlamp
4. OpenBao
5. Zot

Test each service’s TLS before moving on. Verify auto-renewal works (force a short-validity test cert and watch renewal trigger).

Phase 4: distribution to non-cluster consumers

Mostly already wired. The init Ansible role already drops a CA cert into every non-IncusOS host’s trust store as part of the universal baseline (runs as Play I / Play 0 of every setup-*.yaml wrapper). Concrete steps:

1. Replace the placeholder CA cert currently shipped by the init role with the root cert produced in Phase 1
2. Decide on IncusOS host cert path (option 1 vs 2 above)
3. Document operator-machine root cert install (onboarding)

The hard part — actually running update-ca-certificates across the fleet on a tagged Ansible run — is already there.

Phase 5: future scope

• mTLS between services — when needed, the pattern is cert-manager Certificate per Deployment (issuer = heimdall), Secret mounted into the pod, app configures TLS against the file paths and the ca_root_cert trust bundle. Decided 2026-05-17: not csi-driver-spiffe and not a service mesh. The fleet is ~15 workloads of mostly off-the-shelf software that doesn’t speak SPIFFE natively; the per-Deployment Certificate pattern scales fine at this size and avoids a CSI driver / mesh as new infrastructure. Adjacent install worth doing soon regardless: trust-manager (Jetstack sibling to cert-manager) — distributes the root CA as a Bundle resource across namespaces, makes mTLS rollout (and any future namespace-spanning trust work) cleaner. Triggers that would flip this decision: more than ~5 structurally-identical per-Deployment Certificates accumulating in the repo, service-mesh adoption, or external SPIFFE federation with another organization. Cilium-native mTLS is worth tracking separately as it matures past experimental in 1.16 — same outcome, eBPF-not-sidecar, but tied to Cilium identities rather than SPIFFE IDs.
• dns-01 challenges + wildcard certs — once internal DNS supports dynamic updates
• Code signing / image signing — separate intermediate, used by cosign/sigstore

Decision log

• Service hostname heimdall.rrchnm.internal + ca CNAME alias. Heimdall — Aesir gatekeeper, verifies all who cross Bifrost — is the cleanest semantic match for a CA in the fleet’s mythology vocabulary and extends the Norse trust subtheme (odin/thor/loki on hyperion, mimir on ops center). The generic ca CNAME keeps ClusterIssuer manifests theme-agnostic.
• Two-tier CA (offline root + online intermediate). Standard architecture; migration from one-tier later is painful. Start right.
• step-ca over Vault PKI. Native ACME, lighter ops, decouples CA from secrets vault. Reconsider if OpenBao becomes very central to ops.
• In-k0s for intermediate. CA outage is bounded (existing certs keep working); k8s is fine. Pair with cert-manager.
• mTLS via per-Deployment Certificate resources, not csi-driver-spiffe and not a service mesh. Fleet is ~15 off-the-shelf workloads — none speak SPIFFE natively, and the per-Deployment Certificate pattern scales fine at that size. csi-driver-spiffe is the right answer at dozens-to-hundreds of services with workload-identity-aware apps; we’re not there. trust-manager is the lightweight piece worth installing soon (CA bundle distribution across namespaces). Re-evaluate at the triggers listed in Phase 5.
• Single-replica Deployment, not active-active HA. State on one PVC + nightly snapshots into the backup_servers flow. CA pod reschedules across k0s nodes on node loss; renewal failures take 30-60 days to become user-visible, so single-replica + good backups beats the operational tax of shared-DB + HSM/KMS HA. Re-evaluate if compliance requires HA SLOs.
• Internal CA only issues for .internal namespace. Keep separation from public-CA (Let’s Encrypt) for public hostnames.
• http-01 first, dns-01 later. Simpler bootstrap; wildcard / non-HTTP needs are deferred.
• IncusOS host cert: research path 1 (use our CA) before committing. May not be feasible; if not, accept current state.
• Root key offline in escrow. Compromise of root = catastrophic. Treat the storage and rotation procedure as a runbook item, not an afterthought.

Re-evaluation triggers

• Root key compromise (or suspected) → emergency rotation. Document the procedure now so it’s not invented under pressure.
• Intermediate CA outage > a few days → renewals fail, services start losing TLS. Watch for it; have a runbook for emergency intermediate replacement.
• Renewal failures during planned maintenance windows become unacceptable (regulated-environment SLO, frequent k0s upgrades touching CA’s node) → migrate from single-replica to active-active HA: shared DB backend + HSM/KMS for the signing key + leader-election config. ~10x operational complexity, justified only when single-replica restart windows are intolerable.
• Compliance regime requires HSM-backed CA, audit logging, formal CP/CPS → bigger operational investment; possibly switch to Vault PKI with HSM or a managed PKI service.
• We adopt service mesh with automatic mTLS → may displace some of cert-manager’s role (mesh issues workload identity certs); revisit scope.
• IncusOS gains the ability to use external CA-signed server certs via seed → migrate; removes the one-time remote add SAN dance (currently mitigated by fingerprint pinning, but a CA-issued cert would let incus remote add <ip> succeed on first try).
• Cross-trust with another organization (federation, B2B integration) → may need cross-signed certs or a different namespace strategy.