Operator access model

Decision record — 2026-08-29 status, since largely EXECUTED

The “nothing built yet” status below is overtaken: person accounts, the infra-admins group, and Grafana, Argo CD, and Forgejo OIDC are live (recommended-order steps 1–5), and the in-cluster reach detail is superseded — Kanidm now sits behind the internal HAProxy VIP rather than being dialed directly. The three-paths model, the ansi/moby boundary, and the SSH-CA target remain the operative thinking. Two risk statements are stated more generally here than in the operator original.

How an operator’s credentials are divided across three trust roots — Kanidm (who you are), OpenBao (what secret you may have), and a dependency-free static key (break-glass) — and, concretely, where the cheap wins are for finally leveraging Kanidm instead of just running it.

Companion to kanidm, which is the IdM platform design + phased plan. This doc is the operator-facing synthesis: it does not re-derive the platform design, it records the decisions and reasoning from the 2026-08-28/29 design conversation about how to start using Kanidm and how it relates to ansi/moby and the automation path. Where the two docs overlap, identity-access-mgmt.md is authoritative for platform/topology; this one is authoritative for the operator credential model and the adoption order. See “Reconciling with identity-access-mgmt.md” at the end for the small edits that keep them consistent.

Status (2026-08-29 — reasoning captured, nothing built yet)

Kanidm (huginn, hyperion, kanidmd 1.11.1) is running, bootstrapped, and backed up, and doing zero identity work: no human accounts, no groups beyond idm_admin, no OIDC client registered anywhere, LDAP gateway off, services_idm: []. Every recommendation below is net-new value from an asset already being paid for and backed up.

The core model: three paths, three trust roots, no cycle

The single most useful frame that came out of the conversation. An operator’s access decomposes into three distinct activities, and each has a different right credential. Conflating them is what makes the ansi key look un-fixable.

Activity	Example	Credential	Trust root
Run the fleet (automation, human-triggered)	make up, make k0s, make plan	shared machine creds: ansible_ed25519 (SSH), Incus client cert, kubeconfig	OpenBao (target state: SSH-CA-signed short-lived certs)
Touch one thing by hand (interactive, as yourself)	incus exec hyperion:huginn, Grafana/Argo UI, poking a box	your per-person identity + MFA	Kanidm
Emergency (the stores are down)	SSH as ansi to run kanidmd recover-account	static ansi key	itself — depends on neither k0s nor Kanidm

The whole point: Kanidm owns only the middle row. The top row is automation (service-to-service auth — IAM is the wrong shape, see identity-access-mgmt.md “Service-to-service auth ≠ IAM”), and the bottom row must stay dependency-free by construction. No cycle: humans → Kanidm; automation → OpenBao; break-glass → a static string that neither can gate.

Cheap wins: where Kanidm earns its keep, ranked

The IdM design doc’s Phase-1/2 order is right; this is the operator-eye view of which are actually cheap, with the operational gotchas verified against the live manifests.

Win #0 — persons + one group (prerequisite, ~30 min, no deploy). Nothing authenticates against an IdM with no users. This is enumeration, not design: one kanidm person create per real human (today: a handful of operators, not a roster — see the federation open question), set a credential, enrol WebAuthn. Plus exactly one group, infra-admins. This blocks every integration below and is the cheapest thing here.

Win #1 — Grafana (the proof-of-concept). Lowest blast radius in the fleet, and verified cheap: local admin comes from OpenBao (kv/eso/grafana → GF_SECURITY_ADMIN_*, k0s/observability/grafana.yaml) and keeps working alongside OIDC, so break-glass is intact; grafana.ini is already a mounted ConfigMap (grafana-config), so enabling OIDC is a ConfigMap edit + rollout, nothing structural. Register one OAuth2 client in Kanidm, map infra-admins → Admin, everyone-else → Viewer. Proves the web/OIDC path end-to-end; everything transfers to Argo.

Win #2 — Argo CD (highest value). The control plane for every vibes workload, and its own manifests flag the local admin as interim (k0s/external-secrets/argocd-admin-externalsecret.yaml, k0s/gitops/kustomization.yaml). Dex is already scaled to 0 with a note saying “direct OIDC, not Dex” — so this is oidc.config in argocd-cm, keep local admin as break-glass, map infra-admins → role:admin. Slightly more involved than Grafana (RBAC matters more), biggest attribution payoff.

Then Forgejo → Headlamp → OpenBao, same shape, in the design doc’s order.

The gotcha every in-cluster consumer shares

Grafana, Argo, Headlamp, OpenBao, Forgejo all run in k0s and reach huginn at huginn’s :443 server-side (the token + userinfo legs — the browser handles only the redirect). Two things bite that no OIDC tutorial mentions:

1. Server-side CA trust. The pod must trust rrchnm-root or the token exchange fails the TLS handshake even though the browser’s redirect leg works. The root PEM is already inlined as caBundle in k0s/cert-manager/heimdall-clusterissuer.yaml; mount it and point the consumer’s root-CA option at it. (Grafana is Go — SSL_CERT_DIR/a combined bundle works; it has no per-provider server-CA option.)
2. Egress netpol. These namespaces are default-deny egress; add a CiliumNetworkPolicy allow to huginn’s IP (/32) on :443. Well-worn pattern — k0s/vibes/netpol.yaml already reaches the Garage VMs this way, popquiz reaches SMTP, turnstone has its own toCIDR.

Plus: pods must resolve idm.rrchnm.internal (near-certain — everything internal is *.rrchnm.internal), and the OAuth2 client’s redirect URI must be the public https://<svc>.rrchnm.internal/... path, registered in Kanidm.

How a Kanidm group maps to a service role (Grafana worked example)

There is no “binding” object. Kanidm emits group membership as a claim; the service runs a rule whose return value is the role.

• Kanidm side: registering the client isn’t enough — grant the groups scope via a scope-map so the token carries a groups claim. Values are group SPNs, name@domain (e.g. infra-admins@idm.rrchnm.internal), not bare names or UUIDs.

  kanidm system oauth2 create grafana "Grafana" https://grafana.rrchnm.internal
  kanidm system oauth2 update-scope-map grafana infra-admins openid profile email groups
  kanidm system oauth2 enable-pkce grafana

• Grafana side: a JMESPath expression whose result is the role string:

  role_attribute_path = contains(groups[*], 'infra-admins@idm.rrchnm.internal') && 'Admin' || 'Viewer'

Semantics worth knowing: it returns one role (multi-tier = ordered || chain, highest first); valid strings are GrafanaAdmin/Admin/Editor/ Viewer/None (GrafanaAdmin needs allow_assign_grafana_admin = true, and is server-wide vs. the org-scoped rest); role_attribute_strict = true fails loud on a typo’d SPN instead of silently under-privileging; skip_org_role_sync = false lets OIDC drive the role — which also means you can no longer hand-edit roles in the UI (overwritten each login).

Brittleness + the fix. The SPN string is hardcoded in every service’s config, so a group rename silently breaks all of them. Kanidm claim-maps move the mapping to the IdP side — emit a purpose-built grafana_role claim computed from membership, and Grafana collapses to role_attribute_path = grafana_role. Trade: logic lives in Kanidm (one place, survives renames) vs. scattered in service configs. For this fleet, where the value is centralizing identity decisions, prefer claim-maps — and they’re what makes “defer the group taxonomy” (below) safe, since you can restructure groups without touching any service.

Groups and persons: scope persons, defer the taxonomy

infra-admins / developers / viewers in the design doc were not a decision — they appear only in that doc, are wired to nothing, have no git attribution, and are the generic admin/user/read-only triad every SSO tutorial ships. Treat them accordingly:

• The mechanism is the prize, not the tiers. Services mapping group → role gives you onboard-once / offboard-once / one-subject-to-audit — and that value is fully present with a single group. You get 100% of it with just infra-admins.
• The lower tiers are already thin. developers-as-a-Linux-tier is dead (Phase 3 dropped — nobody SSHes as a Kanidm user, devs must be moby). On the web side the design’s own mappings only ever positively privilege infra-admins; the rest is the JMESPath default || 'Viewer', which needs no group at all.
• Empty tiers are drift surface, not structure. For a team the design doc itself calls small (re-eval trigger at ~10 operators), developers/viewers would be provisioned with zero members to satisfy a diagram.

Decision: create infra-admins now; defer developers/viewers/per-project until a real person needs exactly that reduced access. Adding a group + mapping later is cheap and changes no OIDC contract (cheaper still with claim-maps).

Sequencing — persons before Grafana, taxonomy not. Person accounts are a prerequisite (you can’t test an OIDC login with no users) and a fixed enumeration (no design to get wrong), so do them first, credential + WebAuthn in one motion. The one thing to lock before any service depends on it is the naming convention — person SPN shape (roy vs roy.trinh), kebab group names — because those strings get embedded in service configs and audit trails, and renames are disruptive. That’s a 5-minute convention call, not a taxonomy project. (The domain idm.rrchnm.internal is already fixed and expensive to change — WebAuthn-invalidating.)

Why ansi and moby stay out of Kanidm

Asked directly: should the existing shared accounts be Kanidm identities? No — across every meaning (person account, kanidm-unixd POSIX user, Kanidm service account). Not an oversight; it’s the boundary the design rests on.

• They’re roles many humans assume, not identities. An IdM’s value depends on the account being 1:1 with a person; ansi/moby are deliberately 1:many. “In Kanidm” gains nothing (nothing to attribute) and blurs the human/machine line. Same category as service user backup (uid 34).
• ansi must survive a Kanidm outage — it’s the break-glass path to Kanidm (SSH to the IdM host → kanidmd recover-account). Gating it behind Kanidm creates a bootstrap cycle: you’d need the IdM up to fix the IdM.
• moby is forced by rootless docker — it owns the compose projects under its uid, reachable only via its DOCKER_HOST socket, so any named account is bypassed with sudo -iu moby immediately. Kanidm can’t change the ownership model. (This is why Phase 3 was dropped.)
• A Kanidm service account is for workloads that call Kanidm’s API (e.g. a user-provisioning script). ansi/moby never call Kanidm, so they aren’t even that.

Attribution — the real thing under the question — is not obtained by absorbing these accounts but by putting per-person auth in front of the entry point: operators authenticate to the Incus API via Kanidm OIDC, so “who ran incus exec and became moby” is attributable even though the in-instance identity stays shared. The humans absolutely get Kanidm person accounts — but for the web UIs and Incus, and that doesn’t attribute their moby actions. Honest ceiling: actions inside a shared account are attributable only to the entry event, never line-by-line; that’s inherent to shared accounts.

The shared accounts do touch the access story — on the OpenBao side: ansi’s SSH private key is a secret (custody/rotation), not an identity.

The ansi key broad-compromise problem

One long-lived shared key with fleet-wide, root-equivalent reach — worst-case blast radius on both axes. Can Kanidm minimize it? Not the key itself (it’s outside Kanidm by design) — but yes, the underlying risk, indirectly and substantially.

Two axes: time (static → persists) and space (root on all hosts).

• Kanidm’s lever is a third axis — how much the key is used. ansi today carries three jobs: break-glass, automation, and day-to-day human access (each operator’s personal pubkey is pushed into ansi’s authorized_keys by the init role). Move the human traffic onto per-person Kanidm identities at the Incus API, then pull the personal keys out of authorized_keys. ansi shrinks to break-glass + automation: rarely used, tightly held. Buys attribution + MFA + one-place revocation. (Blast-radius reduction depends on the open Incus/OpenFGA authz question — may be attribution-only. Be honest.)
• The direct attack on the time axis is OpenBao’s SSH-CA, not Kanidm — see next section.
• Cheapest concrete win, no Kanidm required: split the break-glass key from the automation key, so the everyday automation credential and the emergency credential have independent blast radii. Give automation its own OpenBao-managed, rotatable, source-restricted key; keep a separate break-glass key in offline/hardware custody, rarely mounted. Independent blast radii; losing the busy one doesn’t hand over the emergency one. Highest-leverage single move against the named scenario.

The automation path: OpenBao SSH-CA (how it works)

For the “run the fleet” row — including human-triggered make — the target credential is an OpenBao SSH secrets engine in CA mode: Bao signs short-lived certs; nothing long-lived sits in authorized_keys.

• One-time: enable the engine, generate_signing_key, distribute the CA pubkey to every host’s sshd as TrustedUserCAKeys (bootstrapped by the existing static key via the init role — no host-side chicken-and-egg), define a role (allowed_users=ansi default_user=ansi ttl=30m).
• Per run: toolbox authenticates to Bao (AppRole or TLS cert — not Kanidm; this is machine auth), generates an ephemeral keypair, asks Bao to sign it with principal ansi + a TTL covering the whole playbook (sign once at run-start, let ControlPersist reuse connections), Ansible connects with the ephemeral key + cert. Key + cert discarded at run end.

Buys: time axis collapses (minutes, not forever); nothing persistent between runs; per-signing audit trail in Bao (automation attribution you lack today); central rotation (rotate the CA, rare).

Two honest catches. (1) Secret zero didn’t vanish, it changed shape — you now mount an AppRole secret_id/cert instead of an SSH key. But it’s TTL-bound, use-count-limited, CIDR-bound, and only yields short-lived scoped certs — a much smaller, more controllable blast radius. A trade, not an elimination. (2) Availability dependency — OpenBao runs in k0s, so routine Ansible would need k0s + Bao up, but Ansible is often how you repair k0s. Same bootstrap-cycle shape as gating ansi behind Kanidm. Resolution (why the break-glass/ automation split matters): make the CA path the everyday path and keep the static key as the dependency-free fallback for “Bao/k0s is down, fix it.”

make targets use the ansi key — what changes for the operator?

Nothing, day one — and the reason clarifies the whole model. Running make is operating the automation harness; it uses ansible_ed25519 (the shared automation key), the Incus client cert (tofu), and the kubeconfig (kubectl) — all machine credentials. Putting Kanidm/OIDC in front of any of them would be the “SSO is the wrong shape for non-interactive auth” mistake.

The “pull personal keys out of ansi” advice targets a different credential: your personal pubkey in ansi’s authorized_keys, used only when you hand-ssh ansi@host outside the toolbox. make never uses your personal key, so pulling it breaks no make target. Two keys, same account, different jobs.

So: keep running make exactly as now (the automation SSH key later graduates to an SSH-CA cert, transparently). Kanidm’s job is the hand-driven things you do as yourself alongside make.

Attributing who ran make (shared creds → downstream logs show the automation identity, not you), ascending effort: (a) record who launched the run — cheap, often enough; (b) have interactive make authenticate you (Kanidm → OpenBao OIDC) so the signed SSH cert’s key-id carries your identity and Bao’s audit records “operator X’s run” — cron falls back to AppRole; (c) honest ceiling: full per-line downstream attribution fights the shared-harness design — “attributed at launch” is the realistic target.

Recommended order

1. Naming convention (person SPN shape, kebab groups) — 5 min, write it down.
2. Person accounts for the real operators + credential + WebAuthn. ← blocks Grafana
3. infra-admins, members added. Defer developers/viewers.
4. Grafana OIDC — client + scope-map (prefer a claim-map), CA mount, egress netpol. Learn the pattern here.
5. Argo CD OIDC — the real payoff; disable local admin once SSO works, keep it in OpenBao as break-glass.
6. Split ansible_ed25519 into break-glass + automation keys (independent of the above; do whenever).
7. Later: OpenBao SSH-CA for the automation path; Incus-API OIDC for interactive operator access (pull personal keys out of ansi).

Open questions (gating how much of this is worth building)

• Federation (design doc §1, still unanswered): does RRCHNM have an org IdM to federate with? Decides whether these person accounts are the source of identity or eventually a cache. Doesn’t block starting (Kanidm stays the broker, group→role mappings survive federation), but caps account investment — create the real operators, not a roster.
• Incus authz (OpenFGA): does per-operator scoping actually reduce privilege, or does everyone land with full access (attribution-only)? Determines whether moving humans to Incus OIDC shrinks the ansi blast radius or just attributes it.
• Automation availability coupling: how much of make should depend on OpenBao being up, given Ansible is the repair tool? The break-glass/automation split is the hedge; the exact boundary is unsettled.

Reconciling with identity-access-mgmt.md

Small edits that keep the platform-design doc consistent with the decisions here (not yet applied):

• Phase 1, group step: change “create infra-admins, developers, viewers” → “create infra-admins; defer developers/viewers/per-project until a real member exists,” with the reasoning (empty tiers = drift; the indirection, not the tier count, is the value).
• Phase 1 sequencing: make explicit that person accounts precede the first integration (they’re a prerequisite, not a parallel task) and that the naming convention is locked before any service embeds an SPN.
• Prefer claim-maps over raw-SPN role_attribute_path where a service supports them, to decouple service config from group names (enables the deferral above).

Cross-references

• kanidm — IdM platform design, topology (Phase 0: huginn direct), phased plan, OpenBao division-of-labor, break-glass, Phase-3 drop.
• internal-ca — heimdall/internal CA; the caBundle in-cluster consumers must trust for the server-side OIDC legs.
• internal-dns — *.rrchnm.internal resolution, incl. from pods.
• sop-ansible-ssh-key-rotation.md (operator note) — the ansi/ansible_ed25519 key as a secret (custody/rotation), the OpenBao half of the shared-account story.