Skip to content
RRCHNM Infra

Runbook — Kanidm IdM (HA pair)

Operational runbook for the fleet identity provider. Design rationale + decision log: /operator/notes/systems/identity-access-mgmt.md. Role/playbook index: ansible/README.md. This file is the how-to-operate, not the why.

Topology (as-built)

client ──▶ idm.rrchnm.internal (A → vip-int, VRRP master epimetheus)
        └▶ HAProxy fleet (atlas / prometheus / epimetheus), SNI-passthrough, mode tcp
             ├─ server idm-1  huginn:8443  (PRIMARY, active)
             └─ server idm-2  muninn:8443  (READ-ONLY replica, backup)
                   └▶ kanidmd terminates TLS (heimdall cert) + reads client IP from PROXY-v2
  • Group: idm_servers = huginn (hyperion) + muninn (theia); IPs in fleet.yaml.
  • Replication: native Kanidm over :8444, mutual-TLS pinned by each node’s replication cert. huginn = allow-pull (supplier), muninn = pull + automatic_refresh (read-only consumer). ~15s incremental cadence.
  • Certificates: issued by heimdall’s JWK provisioner — a token minted with the provisioner password, no ACME network challenge (that’s what lets both hosts hold a cert for the one shared name behind a VIP). Rotated by kanidm-cert-rotate.sh on an 8h timer; 24h cert.

Invariants — do NOT violate

  • Never change kanidm_domain / origin (idm.rrchnm.internal). A kanidmd domain rename invalidates every enrolled WebAuthn credential.
  • muninn is READ-ONLY. When huginn is down, authentication works but writes do not — credential resets, passkey enrolment, group/account edits all require the primary. Don’t linger in that degraded mode.
  • HAProxy is active/backup, not round-robin — writes must never land on the read-only replica. muninn only serves when huginn is down.
  • recover-account idm_admin is once-per-directory (huginn only). Never run it on muninn — muninn inherits the account via replication.
  • Cutover steps are load-bearing in order (see Maintenance below); a half-applied cutover is an outage.

Health checks

Terminal window
# End-to-end through the VIP (DNS → HAProxy → proxy-v2 → kanidm:8443 → TLS):
curl -sS https://idm.rrchnm.internal/status                       # → true


# HAProxy backend view (run on any haproxy host):
echo "show stat" | sudo socat stdio /run/haproxy/admin.sock \
  | awk -F, '/idm.rrchnm.internal/{print $1,$2,$18}'
#   → https-idm… idm…-1 UP   (huginn, active)
#     https-idm… idm…-2 UP   (muninn, backup)   BACKEND UP


# Replication is flowing (run against muninn, the consumer):
incus exec theia:muninn -- sh -c \
  'journalctl -u kanidmd -n 20 --no-pager | grep -iE "Incremental|refresh"'
#   → "Incremental Replication Success"  sock_addrs: [<huginn-ip>:8444]

Failover (huginn loss)

Automatic. HAProxy health-checks :8443; when huginn’s check fails, the idm backend fails over to muninn (backup). curl …/status still returns true; logins keep working (auth is a read). Writes pause until huginn returns.

Recovery:

Terminal window
incus start hyperion:huginn
# verify: haproxy stat shows idm-1 UP again (traffic returns to huginn — backup
#         only serves while the active is down); muninn logs resume
#         "Incremental Replication Success" against <huginn-ip>:8444.

There is nothing to reconcile by hand — muninn is read-only, so no writes diverged while huginn was down; muninn just resumes incremental pull.

Certificate operations

  • Renewal is automatic: kanidm-cert-renewer.timer → kanidm-cert-rotate.sh → mints a JWK token → issues a 24h cert → SIGHUP kanidmd (sessions survive). Safe to run by hand: ssh <host> /usr/local/sbin/kanidm-cert-rotate.sh.
  • Rotate the JWK provisioner bundle (compromise, or hygiene):
    1. step crypto jwk create pub.jwk enc.jwk (offline; set a new password).
    2. group_vars/step_ca_servers.yaml → step_ca_jwk_provisioner_key = pub.jwk.
    3. group_vars/step_ca_servers.sops.yaml → step_ca_jwk_provisioner_encrypted_key = enc.jwk (compact single-line JWE; the role normalizes flattened-JSON but don’t hand-wrap it).
    4. Password → both group_vars/step_ca_servers.sops.yaml (step_ca_jwk_provisioner_password) and group_vars/idm_servers.sops.yaml (kanidm_jwk_provisioner_password) — the CA verifies with the public key, the kanidm client mints with the password.
    5. make step-ca (heimdall re-renders ca.json + full restart), then make kanidm (rewrites the 0600 password file). Verify: step ca provisioner list on heimdall shows acme + jwk; a manual kanidm-cert-rotate.sh issues.

Replication — re-seed / cert exchange

Replication is pinned by each node’s replication certificate (kanidmd show-replication-certificate), stored per-host in host_vars/{huginn,muninn}.yaml (kanidm_replication_cert, public — not a secret). The peer’s cert is what each server.toml pins (huginn→consumer_cert, muninn→supplier_cert).

Bring up / rebuild muninn (two-phase — neither cert exists until the node has run with replication enabled once):

  1. Ensure kanidm_replication_enabled: true in group_vars/idm_servers.yaml.
  2. Apply (make kanidm) with kanidm_replication_cert still empty for the new node → renders [replication] header only → kanidmd generates its cert.
  3. Capture it: incus exec <remote>:<host> -- sh -c 'PATH=/usr/sbin:/usr/bin:$PATH kanidmd show-replication-certificate' → paste the certificate: "…" string into that host’s host_vars/*.yaml.
  4. Re-apply → the peer subsection renders → link. A fresh muninn (empty DB) will auto-refresh (full pull) from huginn on link (automatic_refresh: true). Do not recover-account on muninn.

Snapshot before risky replication changes (rollback floor):

Terminal window
incus storage volume snapshot create hyperion:hyperion_instances kanidm-db <label>
incus storage volume snapshot list   hyperion:hyperion_instances kanidm-db

Backups

kanidmd’s [online_backup] writes consistent dumps to /srv/kanidm/backups (05:30 UTC); the restic timer ships that directory to backup1/backup2 at 02:30 ET (alternating — the fleet’s standard window), repo idm.rrchnm.internal. Only the write primary ships backups — the read-only replica’s copy is identical (replicated), so kanidm_restic_enabled derives from kanidm_replication_primary and the role tears a replica’s timer down (tasks/backup-teardown.yaml); a replica left shipping would just fail every run (its key isn’t authorized on the targets). Passphrase: kanidm_restic_password (group_vars/idm_servers.sops.yaml) — losing it loses the restore; keep a copy in OpenBao beside the idm_admin recovery credential. Authorized restic client keys live in host_vars/backup{1,2}.yaml (restic_clients); add a host’s /etc/kanidmd/restic_ed25519.pub there and make backup TAGS=clients.

Maintenance-window procedures

Cutover to the VIP (or re-cutover after a rebuild)

Order is load-bearing; gate each step on the previous succeeding.

Terminal window
make haproxy    # VIP + idm backend. :8443 checks fail until kanidm flips — expected.
                # ⚠ CONFIRM THIS SUCCEEDS (haproxy -c passed, reload fired) BEFORE proceeding.
make kanidm     # huginn→muninn: listener→:8443 + proxy-v2 + nftables. :443 closes.  ◀ blip
make dns TAGS=knot   # idm A → .149 (VIP).                                            ◀ restored
  • Pre-lower DNS TTL first if you need a short blackout: the idm record is ttl: 60, but clients cached at the old TTL hold the old value until it expires — serve .172 at ttl: 60 a TTL-length ahead of the window so caches are short.
  • Verify each step: haproxy stat (backend present), ss -tlnp | grep 8443 on each host, server.toml [http_client_address_info] lists the 3 haproxy /32s, then curl …/status → true.

Rollback (VIP → direct-to-huginn, emergency)

group_vars/dns_resolvers.yaml idm → huginn’s IP; roles/kanidm/defaults kanidm_bind_port → 443, kanidm_trusted_proxies → []; nftables idm branch back to :443 from clients. make dns && make kanidm. Only huginn serves (no failover) — this is a break-glass, not a resting state.

Incident log

  • 2026-09-02 — Phase C cutover, ~50 min SSO outage. A trim_blocks bug in roles/haproxy/templates/site.cfg.j2 (an inline {% endif %} at the end of an http server line ate the trailing newline, gluing check to the next backend line → checkbackend → haproxy -c failed) failed make haproxy. The cutover continued to make kanidm/make dns anyway, so kanidm left :443 while HAProxy had no idm route. Fixed by restoring the compensating blank line (commit in the Phase C set). Lessons: (1) gate each cutover step on the previous succeeding; (2) render-test every template branch — the bug was in the http-server-loop branch, but only the idm redirect branch had been tested.