Kanidm (IdM)

Design record — status as of 2026-08-22, since largely EXECUTED

The Status section below is overtaken: real persons + groups now exist, Grafana, Argo CD, and Forgejo OIDC clients are live, replication is configured (muninn is a served read-only replica — HA phase B), and the HAProxy VIP fronting this doc calls impossible-until-JWK happened — heimdall gained its JWK provisioner and Kanidm now sits behind the internal VIP (phase C). Credential storage paths and break-glass invocations are operator-side. The architecture, division-of-labor, and decision log remain operative; group_vars/idm_servers.yaml is the as-built authority.

This document captures the design for unified identity across the infrastructure: a single Identity Management platform (IdM) that all human-facing services trust, supporting both OIDC for web/UI services and PAM/NSS for Linux system users, with a clear separation between human and machine credentials.

The chosen platform is Kanidm, with Authentik as a fallback if requirements outgrow Kanidm.

Status (deployed 2026-08-21, bootstrapped 2026-08-22 — running, no integrations yet)

Kanidm is running and serving. huginn (hyperion) answers https://idm.rrchnm.internal — kanidmd 1.11.1, 303 → /ui, holding a heimdall cert (24h, rotated on an 8h timer with SIGHUP reload). Its database is on the protected kanidm-db block volume, and [online_backup] dumps ship to backup1/backup2 by restic. make kanidm is idempotent and green.

Bootstrapped 2026-08-22. kanidmd recover-account idm_admin has been run once against this database; the credential is stored in OpenBao (path operator-side). The restic client is authorized on backup1 and backup2, and the shipper posts a Slack heartbeat through the fleet’s shared restic-notify driver.

Still not doing any identity work, which is the honest summary: no OIDC client is registered anywhere, so nothing authenticates against it yet. Outstanding before Phase 1 is done:

• Base groups (infra-admins, developers, viewers) and one account per real human, then WebAuthn for infra-admins.
• A first integration — Grafana is the recommended proof-of-concept: lowest blast radius, its local admin keeps working alongside OIDC, and everything learned transfers to Argo CD.

There is no kanidm-client role and there will not be one — Phase 3 (Linux PAM) was dropped 2026-08-22, because operators reach hosts via incus exec and developers must be moby. See Phase 3.

muninn (theia) is provisioned and baselined but does not run the role — see “ACME dictates the topology” in the decision log. Two instances are not HA today regardless: replication is unconfigured, so they would be two servers over one database’s worth of state.

The 2026-08-20 review corrected drifted facts without changing any decision. The 2026-08-21 build reversed decision §3: Kanidm moves out of k0s onto dedicated VMs. See the decision log for why; the short version is that k0s bought no availability here (local-path RWO pins a single-replica StatefulSet to one node) while imposing a circular dependency and a bespoke ingress exception.

What exists today:

• roles/init/ runs fleet-wide: internal CA trust, pam_mkhomedir, and — added 2026-08-21 — capping /etc/login.defs UID_MAX/GID_MAX at 9999 (init_login_defs_caps, tag uid-range). That cap is worth keeping to hold local allocation clear of the pinned 9000–9999 daemon block, but its original stated purpose (“reserve 10000+ for Kanidm”) was wrong on two counts: Kanidm allocates above 65536, and with Phase 3 dropped it allocates no POSIX ids here at all. See the corrected range entry in the decision log.
• Argo CD went live during the interval and is running on a local admin account that its own manifests flag as interim, pending this work. It is now the highest-value Phase-2 target.
• Multi-operator SSH already works via ansi’s aggregated authorized_keys, which lowers the urgency: the gap is per-user attribution and clean offboarding, not access.

The open question from §1 (does RRCHNM have an org IdM to federate with?) remains unanswered, and still gates how much of this is worth building.

Why this matters

The current state has each service maintaining its own user database:

• Forgejo (git forge)
• Grafana (observability)
• Headlamp (k8s dashboard)
• Zot (OCI registry — basic auth via htpasswd)
• OpenBao (secrets vault)
• Argo CD (GitOps) — live since this document was first written, running on a local admin account that both k0s/gitops/kustomization.yaml and k0s/external-secrets/argocd-admin-externalsecret.yaml explicitly flag as interim, pending the OIDC work described here
• Carrying their own logins, not yet triaged for SSO: Kuvasz (uptime), Matomo (analytics), turnstone

…plus every Linux host maintaining its own /etc/passwd for operator SSH and sudo access. That is 5 bare-metal IncusOS hosts (kyojin, theia, hyperion, atomsk, ibuypwr) plus 43 active instances — 26 VMs and 17 Incus containers, including both backup1 and backup2. See fleet.yaml for the live roster; do not trust a host list hardcoded in this document.

The init role now pushes every /configs/ansible/*.pub into ansi’s authorized_keys (roles/init/tasks/main.yaml:168), so multi-operator SSH already works. What is missing is per-user identity: everyone lands on the host as ansi, so /var/log/auth.log cannot attribute an action to a person, and removing an operator means editing a key list rather than deactivating an account.

Reassessed 2026-08-22. The host half of this argument does not survive contact with how people actually work, and Phase 3 was dropped as a result: infra operators do not SSH at all (they use incus exec, which is Incus-API auth and root in-instance), and developers on the docker hosts must be moby because rootless docker owns the compose projects under that uid. So the per-host /etc/passwd sprawl described above is real but inert — nobody is logging into those accounts. The unattributed-privilege problem is real and simply lives one layer up, at the shared Incus client cert. The web half of the argument stands unchanged and is where the value is.

This compounds in three ways:

1. Onboarding/offboarding cost scales with the number of services AND the number of hosts. Adding a new operator means 6+ web accounts AND a useradd + key push on every host; removing one is a checklist that’s easy to get wrong (forgotten accounts = lingering access).
2. Audit fragmentation. “Who accessed what when” requires reading 6+ different log streams plus per-host /var/log/auth.log. SIEM-grade correlation is impossible without unified subject identity.
3. Credential hygiene. Each service’s users pick their own passwords (or reuse). MFA enforcement is per-service. SSH keys live forever and rotate inconsistently. Compromise of one credential store doesn’t necessarily revoke access elsewhere.

For a production prototype replacing XCP-ng, this matters. XCP-ng admins typically have 1–3 logins (XO, the host CLI, maybe a wiki); a fragmented Kubernetes-era stack has 6–10 web logins plus N hosts × M operators of SSH state unless you actively unify.

Decisions to make first

Three orthogonal decisions, listed in dependency order:

1. Federate with an existing org IdM, or stand up our own?

If RRCHNM has an existing IdM (Active Directory, Entra ID, Google Workspace, Okta, internal Keycloak), federate with it. Our IdM becomes a relying party / OIDC client / SAML SP, and user identity stays with the org’s existing system.

Benefits: single account for org-wide tools and our infra; org-managed offboarding (deactivate at the source = lose access here); lower ops burden for us.

If no existing IdM, or we want infrastructure-team isolation, stand up our own. Slightly more work but full control.

Action item before continuing this design: confirm what RRCHNM already runs and whether we can federate. If yes, the rest of this doc is “which platform do we use as a federation client” — much simpler. If no, it’s a full IdM deployment.

This document assumes the worst case (no existing IdM) and designs for it. Adjust if otherwise.

2. Which IdM software?

Software	Stack	OIDC	LDAP	Linux PAM/NSS	SAML	MFA	Footprint
Kanidm	Rust	✓ native	✓ native gateway	✓ first-class (kanidm-unixd)	limited	WebAuthn, TOTP	Single binary + sqlite/postgres
Authentik	Python + Postgres + Redis	✓ native	LDAP outpost (extra component)	sssd or pam_oauth2 hacks	✓	TOTP, WebAuthn	Heavier (3+ services)
Keycloak	Java + Postgres	✓	✓ federation	sssd via LDAP	✓	many	JVM heavyweight
Authelia	Go	✓	LDAP backend (BYO)	no	no	TOTP, WebAuthn	Lightweight
Dex	Go	✓ broker only	upstream	no	limited	depends on upstream	Tiny

Recommendation: Kanidm as the primary identity management platform.

Why Kanidm specifically for this stack:

• Linux PAM/NSS is first-class. Operator SSH and sudo access on every host become Kanidm-managed via kanidm-unixd (a small client daemon installed on each host). No LDAP outpost, no sssd plumbing, no pam_oauth2 hacks. This collapses what was originally a Phase-4 advanced item (“real but advanced”) into the basic deployment.
• OIDC for web services. Forgejo, Grafana, Headlamp, OpenBao etc. integrate the same way they would with any other OIDC provider.
• Built-in LDAP gateway for the rare service that doesn’t speak OIDC (Zot, perhaps).
• WebAuthn-first MFA, phishing-resistant by default. TOTP fallback.
• Lightweight footprint. Single Rust binary + a sqlite or postgres backend, vs Authentik’s Python + Postgres + Redis stack.
• Strong security defaults. No NIS-style cleartext passwords, no LDAP simple bind by default, opinionated about credential hygiene.

When Authentik would win (and is therefore our fallback): if requirements grow to need rich SAML federation, SCIM-driven provisioning, polished admin UX for less-technical admins, or integration with a Microsoft-heavy upstream IdM. None of these are current needs. If they become so, the OIDC contracts with our services don’t change — we’d swap the IdM behind them.

3. Where to host?

Option	Pros	Cons
k0s Deployment	Use existing infra; HA via replication; backup via PVC	k0s outage = SSO outage; recoverable but disruptive
Dedicated Incus VM	Isolated; survives k8s issues	More VMs to maintain
External / SaaS	Zero ops	Cost, vendor lock-in, data residency

k0s Deployment is fine here. Superseded — see the decision log. The reasoning below is kept because the tradeoff framing is still right; the conclusion is not. Same reasoning as before: SSO outage is bounded — services can still run, you just can’t log into their UIs. Service-to-service auth doesn’t go through SSO (uses API tokens, mTLS, etc.). And k0s loss = bigger problems than SSO anyway.

One Kanidm-specific wrinkle: kanidm-unixd on each host has an offline cache, so a brief Kanidm/k0s outage doesn’t immediately lock operators out of SSH (cached credentials keep working for a configurable window). This makes the “SSO outage” tolerance better than a pure-OIDC story.

Concretely: deploy Kanidm via Helm chart (or raw manifest if no chart fits) in k0s/platform/kanidm-values.yaml alongside Forgejo, Zot, etc.

Architecture

                    ┌──────────────────────────────────────┐
                    │           Kanidm server              │
                    │  - User & group store                │
                    │  - OIDC provider                     │
                    │  - LDAP gateway (for legacy svcs)    │
                    │  - Auth + MFA (WebAuthn/TOTP)        │
                    └──┬───────────────┬───────────────────┘
                       │               │
       OIDC redirect   │               │   kanidm-unixd
       (web flow)      │               │   (NSS + PAM + cache)
                       │               │
        ┌──────────────▼───┐    ┌──────▼───────────────────┐
        │  Web services    │    │  Linux hosts (ENROLLED   │
        │  - Forgejo       │    │  SUBSET — see Phase 3)   │
        │  - Grafana       │    │  - 5 IncusOS bare metal  │
        │  - Headlamp      │    │  - k0s ctrl + workers    │
        │  - OpenBao       │    │  - docker, backup, cicd  │
        │  - Argo CD       │    │  - NOT single-purpose    │
        │  - Zot (LDAP)    │    │    Incus containers      │
        │                  │    │  Operators SSH/sudo via  │
        │                  │    │  Kanidm group membership │
        └──────────────────┘    └──────────────────────────┘

Web flow (OIDC, same as any IdP):

1. User visits forgejo.rrchnm.internal
2. Forgejo detects no session → redirects to Kanidm
3. Kanidm authenticates user (password + MFA)
4. Kanidm issues OIDC ID token + access token, redirects back
5. Forgejo validates token, extracts identity + groups, creates session

Linux flow (kanidm-unixd):

1. Operator SSHes to a host as their own Kanidm username (e.g. roy@kyojin)
2. SSHd hands off to PAM
3. PAM consults kanidm-unixd (running locally on the host)
4. kanidm-unixd checks the cache, falls back to the Kanidm server if cold/expired
5. Auth succeeds → user gets shell; group membership determines sudo policy

Hosts only need outbound HTTPS to the Kanidm server. The cache means operators stay logged in for a configurable window even if Kanidm/k0s is down.

Relationship to OpenBao

Kanidm and OpenBao are both about access control and look superficially similar — both authenticate, both have “service accounts,” both gate access to something — but they own different concerns and complement each other. The boundary matters because confusion here leads to either putting credentials in the wrong store, or building circular dependencies.

Division of labor

Concern	Kanidm	OpenBao
Who is a person (user accounts, MFA, group membership)	✓ owns	—
Linux PAM/NSS	✓ owns	—
OIDC tokens for web services	✓ owns (issuer)	trusts (consumer)
API keys, DB passwords, TLS keys, etc.	—	✓ owns
Encryption-as-a-service (transit)	—	✓ owns
Dynamic secret generation (DB creds, AWS IAM, PKI on demand)	—	✓ owns
Service-to-service auth	service accounts (limited)	✓ AppRole, K8s auth, etc.
Audit trail of secret access	—	✓ owns
Audit trail of human logins	✓ owns	(Bao’s own UI logins only)

Rule of thumb:

• “Who are you?” → Kanidm
• “What secret can you have?” → OpenBao (which checks who you are via Kanidm if you’re a human)

Integration point

Humans authenticate to OpenBao via Kanidm OIDC. OpenBao trusts the Kanidm OIDC issuer and maps Kanidm groups → Vault policies:

Roy → SSO login at Kanidm (with WebAuthn)
   → Kanidm issues OIDC token
   → vault login -method=oidc → OpenBao validates token, applies infra-admins policy
   → Roy reads/writes secrets per that policy

Services do NOT route through Kanidm to reach OpenBao. They use OpenBao’s native auth methods (AppRole, Kubernetes ServiceAccount, AWS IAM, etc.) — see Service-to-service auth ≠ IAM.

Dependency direction (no cycle)

OpenBao depends on no IAM service (it bootstraps from its own root token + unseal keys). Kanidm depends on OpenBao (its bootstrap admin password lives there).

OpenBao  ───── bootstraps from ─────►  external/manual (printed unseal keys, hardware token)
Kanidm   ───── bootstraps from ─────►  OpenBao
Services ───── auth to OpenBao via ───►  AppRole / K8s SA
Humans   ───── auth to OpenBao via ───►  Kanidm OIDC

Inverting this (e.g. putting OpenBao’s unseal key in Kanidm-protected storage) creates a cycle: Kanidm needs OpenBao to fetch its bootstrap secret, but OpenBao needs Kanidm to authenticate the operator who can fetch the unseal key. Don’t.

This is why the bootstrap account section calls out OpenBao’s root token / unseal keys as “the one credential that can’t be IdM-managed” — it’s the bottom turtle.

Naming overlap to avoid confusing

Both products have the term “service account” and they mean different things:

• Kanidm service account — a non-human Kanidm identity that uses Kanidm’s APIs (e.g. a script that provisions user accounts via kanidm CLI).
• OpenBao AppRole / Kubernetes auth — how a service authenticates to OpenBao to fetch its secrets. OpenBao doesn’t call these “service accounts” (it uses “auth methods”), but in conversation people often do.

Don’t conflate. They solve orthogonal problems: Kanidm’s are for using Kanidm; OpenBao’s are for fetching secrets from OpenBao.

Service integration

Each service that participates is registered with Kanidm as either an OIDC client (web services) or via kanidm-unixd (Linux hosts).

Target	Mechanism	Group → role mapping	Notes
Linux hosts	kanidm-unixd (PAM/NSS)	—	DROPPED 2026-08-22. Nobody logs in this way: infra operators use incus exec (Incus API, root in-instance), and developers must be moby. See Phase 3.
Incus API (candidate)	OIDC (incus remote add --auth-type oidc)	TBD — Incus authz is OpenFGA, separate from OIDC	The operator path that actually exists, and the most privileged one in the fleet. Every operator shares one client cert today, so incus exec — root in any instance — has no attribution. Keep the cert as break-glass. See Phase 3.
Forgejo	OIDC	infra-admins → admin; developers → user	Native OIDC support since v1.18
Grafana	OIDC	infra-admins → Admin; everyone → Viewer	auth.generic_oauth config
Headlamp	OIDC	infra-admins → cluster-admin RBAC	Headlamp has OIDC integration; map to k8s ClusterRoleBinding
OpenBao	OIDC	infra-admins → admin policy	For human auth; services use AppRole
Zot	LDAP (Kanidm gateway)	infra-admins → push/pull; developers → pull	Zot’s OIDC story is rough; LDAP is well-supported. Use Kanidm’s built-in LDAP gateway.
Argo CD	OIDC	infra-admins → role:admin; developers → role:readonly	Live today on a local admin account that is documented as interim (k0s/gitops/kustomization.yaml, argocd-admin-externalsecret.yaml). Argo has native OIDC (oidc.config in the ConfigMap) and the admin account can be disabled once SSO works. Highest-value Phase-2 target: it is the control plane for every vibes workload.
k8s API server	OIDC	infra-admins → cluster-admin via RoleBinding	Configures --oidc-issuer-url etc. on the API server. Defer until Headlamp + UI flows aren’t enough.

Not yet triaged. These have their own logins and are deliberately absent from the table until someone decides whether they warrant SSO: Kuvasz (uptime UI), Matomo (analytics — has its own user model and a large external-user surface), turnstone. The vibes Slack/IRC agents are not human-facing and are out of scope by the service-to-service rule below. Keep this list current: an untriaged service is a silent gap in “who has access to what”.

Host preparation for Kanidm enrollment

Two Ansible roles handle the host side, kept separate so VMs can be provisioned before Kanidm exists and joined later without re-provisioning:

1. init role — runs against every host (all in inventory). Applies the universal baseline that isn’t Kanidm-specific but is needed for Kanidm (and other internal services) to work. Lives in ansible/roles/init/.
2. kanidm-client role — runs once per host once Kanidm is deployed. Installs kanidm-unixd, configures NSS/PAM/sshd to consult Kanidm, applies sudo policy.

Cloud-init does not participate in Kanidm prep. It does the bare minimum needed to let Ansible take over (sshd up, bootstrap user with SSH key, hardened sshd config) and nothing else. The broader cloud-init slimming this is part of was completed in 2026-05 (three fat templates collapsed to cloud_init_vm_debian + _k0s + _ctr_alpine in opentofu/system_cloud-init.tf); the prior plan-opentofu-cloud-init-refactor.md was retired 2026-05-29.

init role (universal baseline)

Three tasks, idempotent, applied to every host. The role exists and runs fleet-wide today — two of the three are already in place:

1. ✅ Internal CA cert in trust store — implemented, roles/init/tasks/main.yaml:45-57. kanidm-unixd (and any other client of an internal-CA-issued service: Forgejo, Grafana, OpenBao, Kanidm itself) needs to trust the internal CA. The role drops the CA cert at /usr/local/share/ca-certificates/rrchnm-internal-ca.crt and runs update-ca-certificates.

2. ✅ Enable pam_mkhomedir — implemented, roles/init/tasks/main.yaml:67-68. Kanidm users don’t have entries in /etc/passwd — they appear via NSS from kanidm-unixd. When such a user SSHes for the first time, no home directory exists. pam-auth-update --enable mkhomedir --force configures PAM to create it on demand. Idempotent and inert until a Kanidm-NSS user actually logs in.

3. ❌ UID/GID range commitment — NOT implemented. Originally specified here as “a comment in /etc/login.defs or a README in /etc/skel/ — documentation, not config.” That was a mistake: /etc/login.defs is live configuration, not a notes file, and using it as one throws away the only mechanism that can actually enforce the boundary.

   Do this instead:

   UID_MAX  9999
   GID_MAX  9999

   Debian ships UID_MIN 1000 / UID_MAX 60000, so the default local-user range spans Kanidm’s 10000–19999 entirely. Rewritten 2026-08-22. Debian’s UID_MIN 1000 / UID_MAX 60000 does not reach Kanidm’s real range, which is 65536–524287 — so the cap was never protecting Kanidm from anything. Keep it anyway, for the reason below: it holds useradd clear of the pinned 9000–9999 daemon block (system_user_ids), where a collision would cost a recursive chown of a service’s data on a protected volume.

   The cap is invisible to Kanidm either way: Kanidm users arrive through NSS, never useradd. And with Phase 3 dropped there are no Kanidm POSIX users on these hosts at all.

   Note the practical risk is low either way — useradd allocates the lowest free id at or above UID_MIN, so a new local user lands at 1001, not 10001; you would need thousands of local users before auto-allocation reached Kanidm’s range. The realistic failure is an explicit useradd -u, and the cap closes that. Treat this as cheap insurance, not an urgent fix.

   Ranges, including what practice has already settled on:

   Range	Owner
   < 9000	Distro / package-daemon users (pinned by init before the package installs)
   9000–9999	RRCHNM service users — de-facto convention, highest currently pinned is 9130
   10000–60000	Debian local users (default UID_MAX; capped to 9999 fleet-wide by init_login_defs_caps)
   61184–65519	systemd DynamicUser=yes — kanidmd’s own transient uid comes from here
   65536–524287	Kanidm POSIX accounts/groups, auto-allocated. Not used today: Phase 3 is dropped.

   The 9000–9999 block is not an accident and does not need fixing — it is a deliberate, documented allocation in ansible/group_vars/all.yaml:6-13 (system_user_ids), stepping by 10, adopted after garage and alloy both auto-allocated to 988 across a vishnu rebuild and garage lost access to its own LMDB. That file records the rationale and the change cost (a recursive chown of the service’s data). It is cross-referenced here only because this document previously described the sub-10000 space as undifferentiated “local users”, which understated how much of it is already spoken for.

   So the remaining gap is small and is about enforcement, not correctness: the ranges are already right and authoritatively documented in the repo, but on the host itself useradd is still free to allocate anywhere up to 60000. Baseline item 3 closes that — cheap insurance, not a correction.

The init role does NOT install kanidm-unixd itself — that belongs in the kanidm-client role so the baseline is Kanidm-agnostic. A host can have init applied without ever joining Kanidm.

What stays the same

• ansi user with the static SSH key stays as the break-glass path. If Kanidm is unreachable past the offline-cache window, you can still SSH as ansi from any machine that has the ansible_ed25519 key. Don’t remove this even after full Kanidm rollout.
• Service users (e.g. backup on backup2 with UID 34) stay; they’re not “human users” and don’t go through Kanidm.
• Hardened sshd config (PubkeyAuthentication only, etc.) unchanged. The Kanidm SSH flow uses keys, not passwords (see SSH auth flow below).

kanidm-client role (active enrollment)

When Kanidm is deployed and the role runs against a host, it:

1. Depends on the init role (so prereqs are guaranteed)
2. Installs kanidm-unixd (apt package or upstream binary)
3. Writes /etc/kanidm/config with the Kanidm server URL + token
4. Modifies /etc/nsswitch.conf to add kanidm to passwd/group lines
5. Adds kanidm to /etc/pam.d/common-* via pam-auth-update
6. Drops a sudoers file: %infra-admins ALL=(ALL) NOPASSWD:ALL (or with-password variant) — keyed on Kanidm groups
7. Adds an AuthorizedKeysCommand directive to sshd_config pointing at /usr/bin/kanidm_ssh_authorizedkeys, so Kanidm users can SSH using SSH keys stored in their Kanidm account
8. Restarts sshd
9. Verifies enrollment via id <some-kanidm-user>

Idempotent. Doesn’t touch ansi or any local users.

SSH auth flow once enrolled

Identity	SSH key source	MFA
ansi (break-glass)	Static /home/ansi/.ssh/authorized_keys (set by cloud-init)	None
Kanidm operator (e.g. roy)	Looked up dynamically from Kanidm via AuthorizedKeysCommand	Required for sudo (TOTP/WebAuthn via PAM)

Operators upload their SSH public key to their Kanidm account once. Every Kanidm-enrolled host accepts it without per-host distribution. Removing the key in Kanidm revokes SSH access everywhere on next sshd lookup.

Order of operations

For a new VM:

1. Tofu provisions with the minimal cloud-init template (sshd + ansi user). VM boots; ansi works.
2. init role runs. Universal baseline applied: CA trust, mkhomedir, UID-range docs.
3. Existing service-specific playbooks run as needed (setup-kyojin-style for IncusOS hosts, backup/setup.yaml for backup hosts, etc.).
4. kanidm-client role runs (depends on init). VM joins the Kanidm fleet.

For an existing host (anything already in fleet.yaml, plus the bare-metal IncusOS hosts): same order, and init has already run against all of them — so in practice only step 4 is outstanding, and only for the enrolled subset. No reflashing required.

Critical design choices

Bootstrap / break-glass account

Kanidm runs in k0s. If k0s is broken, you can’t log in via Kanidm to fix it. Corrected 2026-08-22 — this predates the §3 reversal and had the exposure backwards. Kanidm runs on its own VM (huginn), so it survives a k0s outage; that was the point of moving it. What does not survive is the stored credential: idm_admin’s password lives in OpenBao, which runs in k0s.

That is an availability coupling, not a dependency cycle — OpenBao authenticates via Shamir shares and Kubernetes SAs and never consults Kanidm, so it remains the bottom turtle (see Dependency direction). And it is survivable, because the stored password is not the actual break-glass path. See “Kanidm itself” below.

Need a way out at every layer.

Per-service (web): each integrated service must keep at least one local admin account independent of Kanidm:

• Forgejo’s initial admin user (created at first run, password in OpenBao)
• Grafana’s admin user (still works alongside OIDC)
• Headlamp’s admin token
• OpenBao’s root token / unseal keys (the one credential that can’t be IdM-managed; OpenBao is itself the credential store)

Per-host (Linux): every host keeps the existing ansi user with its hardcoded SSH key as the break-glass account. kanidm-unixd adds a parallel set of operator users — it doesn’t replace local users. If Kanidm/k0s is down past the offline cache window, you can still SSH as ansi from a machine that has the ansible_ed25519 key.

Kanidm itself: the idm_admin account. Day-to-day admin happens via regular accounts in idm_admin_grp; idm_admin is for initial bootstrap and emergency recovery only.

The real break-glass here is root on the kanidm host, not a stored string: kanidmd recover-account idm_admin, run as root on the kanidm host via the per-host break-glass path. It runs against the live server over its local admin socket (/var/run/kanidmd/sock, no downtime) and mints a NEW password every time. So being unable to read the stored copy — during a k0s outage, say — does not lock you out of Kanidm: you SSH to huginn as ansi and re-run it. The chain that must hold is ansi SSH key → root on huginn, which is the same per-host break-glass as every other machine and deliberately involves neither k0s nor Kanidm.

Run it ONCE per database (not per host — a replica shares the directory).

Where the password goes — OpenBao, via the fleet’s KV seed/patch flow (exact path + commands operator-side).

These accounts are emergency-only, documented in a runbook, and rotated quarterly. Credentials go in OpenBao and off-site secure storage — “or” was fine when everything shared a failure domain, but a credential for a VM-hosted service stored only inside k0s is unreadable in exactly the incident where you reach for it. The off-site copy is the cheap half of that; the recover-account path above is the real one.

Service-to-service auth ≠ IAM

IAM is for human auth. Services authenticate to each other via:

• API tokens (Forgejo deploy keys, Zot service accounts) — long-lived, scoped, stored in OpenBao
• mTLS (services with cert-based identity, paired with the internal CA — see internal-ca)
• OpenBao AppRole (services request short-lived tokens from Bao based on role+secret-id)
• Kubernetes ServiceAccount tokens (in-cluster, automatic)
• Kanidm service accounts for the few cases where a non-interactive workload needs to use Kanidm (e.g. a script that creates user accounts) — these are first-class in Kanidm but separate from the human user model.

Don’t make services log in through OIDC. SSO is the wrong shape for non-interactive auth, and routing a service through Kanidm adds latency and a cascading failure path.

Group-based authorization, not role-based

Kanidm’s group memberships drive every service’s role:

• infra-admins (full admin everywhere; sudo on all hosts; cluster-admin in k8s)
• developers (read+write to dev resources; SSH but no sudo)
• viewers (read-only)
• Per-namespace/per-project groups as needed

Each service maps these to its own RBAC. Adding a person to a Kanidm group propagates everywhere on next login (or next cache refresh on Linux hosts). Removing them locks them out.

This means groups are managed in Kanidm only. No service should manage its own role lists for IAM users (only for the bootstrap accounts). Drift between services becomes invisible otherwise.

POSIX attributes and UID/GID range allocation

Kanidm’s Linux integration requires that users and groups have POSIX attributes (uidNumber, gidNumber, login shell, home directory). Kanidm allocates these from a configurable range; pick a range that doesn’t collide with system-default UIDs/GIDs (Debian uses 0–999 for system, 1000+ for regular users).

Suggested: Kanidm-managed users in 10000–19999 range; groups in 20000–29999. CORRECTED 2026-08-22 — Kanidm allocates 65536–524287, and everything below that is systemd-reserved (notably 61184–65519 for DynamicUser=yes). This is not configurable-to-taste: kanidmd 1.11.1 has a gidnumber range validity check that FAILs entries outside it. Let it auto-allocate. Moot in practice while Phase 3 is dropped — POSIX attributes exist for kanidm-unixd, which is not deployed. Original text: doesn’t collide with anything we currently use, leaves room for system users at the low end and ad-hoc local users in 1000–9999 if ever needed.

Document the chosen ranges in the deployment values and don’t change them after first user creation (would require renumbering all home directory ownership).

MFA

Mandatory for infra-admins. Optional but encouraged for everyone else. Kanidm supports:

• WebAuthn (YubiKey, platform authenticators) — preferred where supported (phishing-resistant)
• TOTP (Google Authenticator, Aegis, etc.) — fallback
• Backup codes (recovery)

For SSH, MFA is enforced via PAM (pam_kanidm supports the same MFA methods as web auth — TOTP via PAM prompt, WebAuthn via plugin). Set up MFA-required groups; passwords-only groups skip MFA but lose access to sensitive services.

Sessions and tokens

Sane defaults:

• OIDC ID token lifetime: 1 hour (re-auth via refresh token, not full re-login)
• Refresh token lifetime: 30 days (with sliding window)
• Browser session: 12 hours (forces re-login after a workday)
• Linux PAM cache (kanidm-unixd): 1 hour for credentials, longer for group lookup; configurable
• Service account / API token: indefinite, scoped, revocable

Tighten these for higher-sensitivity services if needed.

Phased plan

Phase 1: deploy Kanidm + integrate first targets

1. Add k0s/platform/kanidm-values.yaml (Helm chart or raw manifest, encrypted bootstrap password via SOPS).

2. Add Kanidm install task in deploy-workloads.yaml.

3. Create base groups in Kanidm: infra-admins, developers, viewers, with POSIX attributes (gidNumber from chosen range).

4. Create operator user accounts (one per real human), set strong passwords, enroll WebAuthn for infra-admins.

5. First integration: pick the highest-value of these two for proof-of-concept:

   • Grafana (easiest OIDC, low blast radius — confirms web/OIDC flow end-to-end)
   • Linux SSH on one host (e.g. backup2 — confirms PAM/NSS flow end-to-end, validates the cache + fallback story)

   Either is fine. Doing both in Phase 1 is also reasonable since they exercise different code paths.

Phase 2: integrate remaining web UIs

In rough order of value:

1. Argo CD — the control plane for every vibes workload, and the only integrated service currently running on an admin account its own manifests describe as interim. Native oidc.config; disable the local admin once SSO works (keep the credential in OpenBao as break-glass).
2. Forgejo — every code interaction goes through it
3. Headlamp — cluster admin
4. OpenBao — secrets access
5. Zot — via Kanidm’s LDAP gateway (less interactive; mostly API-driven)
6. Triage the untriaged (Kuvasz, Matomo, turnstone) — decide in or out, and record the decision either way

For each: configure OIDC client (or LDAP bind) in Kanidm, configure service to use it, define group→role mapping, document the bootstrap account and how to recover it, test login + permission boundaries.

Phase 3: roll Linux PAM out across the enrolled host set — DROPPED 2026-08-22

Not deferred. Dropped, because it solves a problem this fleet does not have. Two independent structural reasons, either sufficient on its own:

Infra operators never touch host login. They use incus exec, which authenticates to the Incus API with the client cert in /configs/incus/ and lands them as root inside the instance. SSH, host users, PAM and NSS are all absent from that path. Enrolling ~31 hosts would have created accounts nobody logs into, plus a kanidm-unixd daemon and an offline-cache failure mode on each.

Developers must be moby. Rootless docker owns the compose projects under moby’s uid, reachable only through moby’s DOCKER_HOST socket (wired into moby’s .bashrc). A named account cannot drive those stacks without becoming moby, so per-user accounts would be bypassed with sudo -iu moby immediately. The shared account is a consequence of the rootless design, not laziness — Kanidm cannot change it.

The original plan (an enrolled/not-enrolled split of roughly 31/17 hosts, drawn on “does a human ever interactively log in here”) is withdrawn along with the phase. Its dividing line was sound; the premise that humans log in at all was not.

Consequences elsewhere in this document:

• POSIX uid/gid attributes are now near-irrelevant — they exist for kanidm-unixd, which is not being deployed. See the corrected range entry in the decision log.
• pam_mkhomedir and internal-CA trust in roles/init/ stay. Both are harmless, fleet-wide, and useful regardless.
• The ansi break-glass key was never in question; with this phase dropped it is simply the only host access model, as it is today.

Bring this back only if humans start logging into hosts directly and routinely — e.g. incus exec is restricted away from operators, or a workload requires interactive multi-user shell access. Both are listed under Re-evaluation triggers.

Phase 3 (replacement candidate): operator identity at the Incus API

The attribution gap this project should actually close is not on the hosts. It is at the Incus API — simultaneously the most privileged path in the fleet (incus exec is root in any instance, on any host) and the one where every operator shares a single client cert. “Who rebuilt that volume?” is currently unanswerable.

Incus supports OIDC directly: incus remote add --auth-type oidc (verified on the 7.3 client in the toolbox), so Kanidm can cover the operator path at the layer operators actually use.

Open questions, to answer before committing:

1. Authorization, not just authentication. OIDC establishes who; Incus’s fine-grained authz is OpenFGA, a separate mechanism. Confirm what IncusOS supports and whether per-operator scoping is achievable or whether everyone lands with full access anyway (in which case this buys attribution only — still worth something, but be honest that it is only that).
2. Bootstrap dependency. Putting Kanidm in front of incus exec means an IdM outage costs the recovery path used to fix things. The client cert must remain as break-glass, exactly as ansi does per-host. Non-negotiable — see the bootstrap section.
3. What the toolbox does. The container mounts /configs/incus/ and runs unattended Ansible against remotes; automation keeps cert auth regardless. Only human remotes would move to OIDC.

Worth doing before the remaining web UIs? Probably not — it is one credential shared by a handful of trusted operators, whereas Argo CD is a shared admin on the control plane for every workload. Sequence it after Phase 2 unless operator attribution becomes an audit requirement.

Phase 4: federate with org IdM (if applicable)

If RRCHNM stands up or already has an org IdM, configure Kanidm as a federation broker — users authenticate at the org IdM, Kanidm issues OIDC tokens to our services. We retain control of our group → role mappings while sourcing identity from the org.

Kanidm’s federation story is less mature than Authentik’s; if this becomes a hard requirement, re-evaluate the platform choice.

Phase 5: extended scope

Real candidates, deferred until the basics are solid:

• k8s API server OIDC (replace kubectl cert-based auth with OIDC for individual operators)
• SCIM for automated user provisioning from the org IdM (if Kanidm supports it sufficiently by then; consider Authentik swap if not)
• Per-project group hierarchies as the org grows

Decision log

• Kanidm over Authentik / Keycloak. Linux PAM/NSS is first-class in Kanidm; with Authentik or Keycloak it’s a Phase-4 advanced bolt-on (LDAP outpost + sssd, or pam_oauth2). For our scope (small team, OIDC web services + Linux SSH), Kanidm is the better shape. Authentik remains the fallback if SAML/SCIM/polished UI become hard requirements.
• In-k0s deployment. REVERSED 2026-08-21 → dedicated Incus VMs (huginn/muninn, group idm_servers). The original reasoning — “SSO outage is bounded, simpler than another VM” — did not survive contact with the build:
  • The circular dependency is the real cost. An IdM that gates access to the platform, running on that platform, means a k0s incident removes web login to Argo/Headlamp/Grafana at exactly the moment you need them. ansi covers the Linux path; nothing covered the web path.
  • k0s bought no availability to trade against it. Every PVC here is local-path RWO — node-local storage — so a single-replica StatefulSet is pinned to whichever node its volume landed on and cannot reschedule if that node dies. The HA was illusory, and worse, looked real.
  • Kanidm fits k8s ingress badly. It mandates TLS on every hop with no plaintext-behind-a-proxy mode, so it cannot use the internal Gateway’s terminating listener. Passthrough needs TLSRoute (experimental in Cilium v1.19, with open bugs where TLSRoutes attach to HTTPS listeners); re-encrypt needs BackendTLSPolicy, which v1.19 does not implement although the CRD is installed and would silently never reconcile. Every route led to a bespoke exception.
  • On VMs it is the ordinary kuvasz/zot shape: own heimdall cert via step-cli, HAProxy SNI-passthrough, hand-curated dns_zones record. Two instances on hyperion + theia, both AQ114, HAProxy health-checking both.
  • Cost accepted: cert renewal is ours to run (step-cli timer, SIGHUP reload) rather than cert-manager’s, and there is one more pair of VMs to patch.
• ACME dictates the topology; Phase 0 is ONE instance reached directly. The architecture below describes clients → VIP → HAProxy SNI-passthrough → either instance. That cannot be built first, because ACME validates a name by connecting to wherever it resolves: with idm.rrchnm.internal pointing at a VIP, heimdall’s HTTP-01 callback lands on whichever backend the LB picks, not necessarily the one that requested the cert. So idm.rrchnm.internal is an A record straight to huginn, kanidmd terminates its own TLS on :443, and there is no VIP and no PROXY protocol. muninn is deliberately excluded from the role: installing kanidmd there would have it request a certificate for a name that resolves to huginn and fail the challenge on a timer, forever. Restoring the fleet in front needs a JWK provisioner on heimdall (issue by authenticated token rather than by reaching back to the requester) — that is the actual prerequisite for HA here, and it comes before replication.
• The database goes on a type: block volume, not filesystem. Incus attaches filesystem volumes to VMs over virtiofs, which does not support MAP_SHARED; kanidm’s SQLite runs in WAL mode and needs it for the -shm file. Measured on huginn: /srv/kanidm (virtiofs) → OSError: [Errno 19] No such device; /root (virtio-blk) → fine. kanidmd reported it as SQLITE_IOERR_SHMMAP (extended_code 5386). The MariaDB volumes are block-typed for the same underlying reason (tc.log mmap) — that precedent was already documented in opentofu/README.md and was missed when this volume was declared. Generalized there now.
• db_path lives outside the unit’s StateDirectory, and that trade is deliberate. The package default /var/lib/private/kanidmd is on the instance root disk, so an instance rebuild loses the directory. Putting it on the protected volume keeps it — at the cost of taking over ownership management from systemd. DynamicUser=yes allocates a fresh uid every boot, so the only durable handle on the data is the static group kanidmd: hence the setgid 2770 data dir, the 0660 database, and a ReadWritePaths= drop-in (the unit is otherwise ProtectSystem=strict by implication). The sharp edge: anything touching those files as root leaves them group-unwritable and the service dies with SQLITE_READONLY — kanidmd configtest does exactly that, since it creates the DB if absent. Open risk to watch: the first reboot is when the transient uid actually changes and this handling gets tested for real. A lower-drift variant exists — mount the same block volume at /var/lib/private/kanidmd and revert db_path to the default, letting systemd manage ownership — but it rests on how systemd treats a pre-existing mount there, which is unverified.
• OIDC for web, Kanidm-LDAP for Zot, kanidm-unixd for Linux. Each protocol where it fits.
• Group-based authorization, not per-service role assignments. Single source of truth for who has what access, propagates on next login. Cost: requires discipline about not configuring service-local roles for IAM users.
• MFA mandatory for infra-admins only at first. Can tighten later. WebAuthn preferred over TOTP. PAM-side MFA enforced via Kanidm groups.
• Enrol by “does a human log in here”, not by criticality. SUPERSEDED 2026-08-22 — no host gets kanidm-client. The split (~31 of 48) was drawn on the right question; the answer turned out to be “none of them”. Infra operators reach instances through incus exec — Incus API auth, root in-instance, no SSH and no PAM in the path — and developers must be moby, because rootless docker owns the compose projects under that uid. Named Linux accounts would have gone unused by one group and bypassed with sudo -iu moby by the other. Full reasoning in Phase 3. The corollary is that the operator attribution gap lives at the Incus API, not on the hosts, which is the new Phase 3 candidate.
• POSIX UID/GID range allocated explicitly (10000–19999 users, 20000–29999 groups). CORRECTED 2026-08-22 — Kanidm requires 65536–524287. kanidmd 1.11.1 carries a named validity check, gidnumber range validity, whose failure text reads: “The automatically allocated gidnumbers for posix accounts was found to allocate numbers into systemd-reserved ranges. These can no longer be used. action: Modify the gidnumber of affected entries so that they are in the range 65536 to 524287 OR reset the gidnumber to cause it to automatically regenerate.” The bands below are systemd’s — including 61184–65519 for DynamicUser=yes, which is where kanidmd’s own transient uid comes from. So do not hand-assign: let Kanidm auto-allocate above 65536. The local side cannot collide from either direction — init_login_defs_caps caps local UID_MAX/GID_MAX at 9999, and even uncapped Debian tops out at 60000. Cross-reference: daemon system users are pinned to a reserved 9000–9999 block stepping by 10, per ansible/group_vars/all.yaml (system_user_ids); that block is why the local cap exists and is unrelated to Kanidm. Largely moot in practice now that Phase 3 is dropped — POSIX attributes exist for kanidm-unixd, which is not being deployed.
• Bootstrap accounts are non-negotiable. Every web service keeps a local admin account. Every host keeps the ansi break-glass key. Kanidm itself keeps idm_admin. All in OpenBao / off-site, rotated quarterly.
• Service-to-service auth is OUT of scope. API tokens / mTLS / AppRole / Kanidm service accounts. Don’t try to make services authenticate as humans.

Re-evaluation triggers

• RRCHNM stands up or unifies on an org-wide IdM → switch from “Kanidm as primary” to “Kanidm as federation client” or remove it entirely
• Compliance regime requires SAML, SCIM, audit logging beyond what Kanidm provides → consider Authentik or Keycloak migration; OIDC contracts with services don’t change
• Operator count grows past ~10 → user lifecycle automation (SCIM, just-in-time provisioning) becomes critical; revisit Kanidm vs Authentik on this dimension
• A service we want to integrate doesn’t support OIDC AND doesn’t support LDAP → SAML fallback (would push us toward Authentik), or wrap with oauth2-proxy for forward-auth pattern
• A breach implicates IAM → token lifetimes tighten, MFA mandatory everywhere, possibly audit log review (the kanidm-unixd cache TTL is no longer a lever — Phase 3 is dropped)
• Humans start logging into hosts directly and routinely → the premise behind dropping Phase 3 fails, and Linux PAM comes back. Concretely: incus exec is restricted away from operators, a workload needs interactive multi-user shell access, or developers stop being able to work as moby (e.g. a move off rootless docker). Any one of these, re-open Phase 3 as originally scoped — the enrolled/not-enrolled split was sound reasoning against a premise that has since changed back.
• Operator attribution becomes an audit requirement → promote the Incus OIDC candidate (Phase 3 replacement) ahead of the remaining web UIs. Today incus exec is root-in-any-instance behind one shared client cert, which is the largest unattributed privilege in the fleet and the first thing an auditor will find.
• We adopt a service mesh (Cilium service-mesh, Istio, Linkerd) → per-service mTLS becomes automatic; review whether it changes the service-to-service auth picture
• Kanidm’s project trajectory falters (loss of maintainers, security issues, etc.) → migration plan to Authentik exists because the OIDC contracts with services are interchangeable