Skip to content
RRCHNM Infra

Slack-bot k0s migration

Move the interactive RRCHNM Slack-bot stack (pi-mom orchestrator + the agents it forwards to) out of docker-compose into the k0s vibes namespace. Phases 1-4 shipped June 2026 (commits 90686ac5, 1a0f7fd6, eea9ef83 + the phase-4 commit) and the cutover has since happened — the stack is live. The fleet then evolved past this plan: github-agent, web-browser, and the CDP backends (chromium, lightpanda) were decommissioned, and web-developer (not in this plan) was added. Live roster today: slack-bot-pi-mom (+ channel-tailer), general-agent, web-developer, accessibility-tester (k0s/vibes/slack/). Phases 5-6 were never built. The per-agent recipe and conventions below remain the operative pattern; per-phase entries are the June record with outcome notes added.

Architecture model (every phase follows this)

  • pi-mom pod (pi-mom + slack-channel-tailer; chat-web-ui dropped from the plan 2026-06-09) shares a filesystem (/workspace) as IPC → run as one Deployment, N containers, one RWO local-path PVC, replicas:1, strategy: Recreate. Slack Socket Mode → no inbound Service. fsGroup: 1000 (roy). No K8s API (automountServiceAccountToken: false).
  • Each agent = a stateless vibes-agent-wrapper HTTP server on :9000 → one Deployment + ClusterIP Service + replicas:1 (in-memory per-channel conversation must not split across pods), fsGroup: 999 (agent user), no SA token, readiness/liveness httpGet /manifest|/health.
  • Service names must equal the MOM_*_AGENT_URL host pi-mom dials: general-agent, github-agent, web-browser, accessibility-tester, database-agent-law, database-agent-apiary (and the CDP chromium / lightpanda). pi-mom reaches each via MOM_<X>_AGENT_URL=http://<svc>.vibes.svc.cluster.local:9000.
  • Secrets via ESO from OpenBao kv/eso/<name> using dataFrom: extract (mirror k0s/external-secrets/general-agent-externalsecret.yaml or the agent-controller precedent): the whole kv path materializes 1:1 (field name → Secret key), so seed every kv field UPPER_SNAKE with the bare credential name (ANTHROPIC_API_KEY, SLACK_BOT_TOKEN, GH_TOKEN, …). The Deployment binds each per-key by secretKeyRef (not blanket envFrom). For the agents, the wrapper now reads the bare names directly, so secret key == env var (name == key; envFrom-able). The bot is the lone exception: its upstream mom binary owns the MOM_* namespace, so the bot’s secretKeyRef maps the bare kv keys → MOM_SLACK_*/MOM_ANTHROPIC_API_KEY (that prefix lives in the manifest, not the kv key). extract pulls only the fields that are seeded, so optional integration keys can be omitted (mark them optional: true on the Deployment); a required key just won’t materialize and its secretKeyRef fails the pod — the intended fail-loud for the Anthropic key.
  • Egress is default-deny (netpol.yaml). All third-party APIs are public HTTPS → existing world:443. The only non-public destination is the DB agents’ on-prem Postgres (phase 5).
  • Images: pull the existing :latest images from ghcr.io/chnm/systems/* through Zot — no separate k8s build pipeline; the upstream build (GHA on chnm/systems) is the single source of truth. Post-Pattern-B (commit eea1c3a0, 2026-06-09): k0s manifests reference the upstream path directly (image: ghcr.io/chnm/systems/<path>:latest); containerd on each k0s node routes the pull through the in-cluster Zot mirror via /etc/containerd/certs.d/ghcr.io/hosts.toml written by the k0s_registry_mirror role. Each new image still needs a same-name passthrough block in ansible/group_vars/oci_mirrors.yaml’s ghcr.io content: list (prefix == destination == /chnm/systems/<path>) — Zot v2.1.6 requires content rules to admit a path. Image paths follow the category-namespace layout (reorg commit f05d8f2b): every vibes image lives at /chnm/systems/vibes/<category>/<name>. pi-mom + tailer are vibes/slack/bot / vibes/slack/channel-tailer; agents are vibes/agents/<name> (e.g. vibes/agents/general, vibes/agents/github-admin, vibes/agents/web-browser, vibes/agents/accessibility-tester); the CDP backends are vibes/browsers/chromium / vibes/browsers/lightpanda. The image path’s <name> ≠ the k8s workload/Service name (the dial-name) — workload/Service is the bare name pi-mom dials (e.g. image vibes/agents/github-admin → dial-name github-agent). If any image flips private on ghcr, the pull-through needs auth wired on the ghcr remote (today: read-only PAT in vars/github.sops.yaml).
  • Startup ordering: agents up before pi-mom; pi-mom’s prepare initContainer waits (fail-soft) for each MOM_*_AGENT_URL/manifest. pi-mom builds its forward_* registry once at startup → adding/removing an agent requires kubectl -n vibes rollout restart deploy/slack-bot-pi-mom.
  • PSA on vibes is baseline. Quota (quota.yaml) has headroom now but the browser/accessibility pods are heavy — re-check before phase 3/4 and raise if pods/requests.* would be exceeded.

Reference facts: roy uid/gid 1000; agent uid/gid 999; wrapper port 9000; HEALTHCHECK curls /manifest.

Reusable per-agent recipe (template for every remaining agent)

  1. Confirm the ghcr image is published at ghcr.io/chnm/systems/vibes/agents/<name>:latest (<name> is the ghcr image name under the vibes/agents/ category — NOT the dial-name <svc>). No moby build step; the upstream GHA is the source of truth (.github/workflows/docker-images--vibes.yml).
  2. Add a same-name passthrough block to ansible/group_vars/oci_mirrors.yaml’s ghcr.io content: list — prefix == destination == /chnm/systems/vibes/agents/<name>, stripPrefix: true (the substitution is a no-op, but the entry is what admits the path through Zot’s content rule check). Then make oci-mirrors to apply on the live mirror.
  3. k0s/vibes/<svc>.yaml (file + Deployment/Service named for the dial-name <svc> pi-mom dials): copy general-agent.yaml from phase 1; set image: to ghcr.io/chnm/systems/vibes/agents/<name>:latest (Pattern B — upstream path directly, containerd routes via hosts.toml), the Service name to <svc>, env, secrets, resources.
  4. k0s/external-secrets/<svc>-externalsecret.yaml — dataFrom: extract from kv/eso/<kv-path> (1:1 field→Secret-key); k8s Secret name/target.name = the dial-name <svc>. Add the apply task to playbooks/k0s/deploy-eso.yaml (no blocking wait); add the file to k0s/vibes/kustomization.yaml.
  5. Seed kv/eso/<kv-path> in OpenBao (operator) with UPPER_SNAKE bare credential names (the Deployment’s secretKeyRef maps each to the app env var) — via the matching block in the SOPS seed file k0s/platform/secrets/initial-kv-seeds.sops.yaml, then make k0s-openbao-kv-put.
  6. Add MOM_<X>_AGENT_URL to the slack-bot-pi-mom Deployment env, apply, then rollout restart deploy/slack-bot-pi-mom.
  7. Smoke: in-cluster curl …:9000/manifest; then a Slack message that routes to the new forward_<x> tool.

Status

Phase 1 — DONE (90686ac5, image-tag realignment in 4cf2a745): slack-bot-pi-mom (pi-mom + slack-channel-tailer, 2 containers, RWO PVC, Recreate, prepare init that waits for migrated agents) + general-agent (Deployment+Service) + their ExternalSecrets + kustomization / deploy-eso / oci_mirrors wiring. The phase-1 manifests are the templates for the rest.

Phase 2 — DONE, since DECOMMISSIONED (1a0f7fd6; the agent was later removed from the fleet): github-agent (Deployment+Service) reused the phase-1 template. Workload + Service named github-agent; image vibes/agents/github-admin (ghcr image name kept verbatim, descriptive rename deferred). Secret github-agent from kv/eso/github-agent (ANTHROPIC_API_KEY + GH_TOKEN). ExternalSecret + kustomization + deploy-eso wiring + pi-mom prepare init wait + MOM_GITHUB_AGENT_URL env added. No netpol / no quota change.

Phase 3 — DONE, since DECOMMISSIONED (all three later removed from the fleet): web-browser (Deployment+Service) + the two bare-name CDP backends chromium and lightpanda (Deployment+ Service each, hardcoded bare DNS targets in the web-browser agent prompt / scripts/lp-fetch.js / skills/web-search). Secret web-browser from kv/eso/web-browser-agent (kv path takes the -agent suffix matching general-agent / github-agent; k8s Secret name stays web-browser to match the Deployment/Service dial-name); single field ANTHROPIC_API_KEY; chromium + lightpanda carry no secrets, TCP-only :9222 probes). No netpol change (chromium/lightpanda are same-ns → existing rule 2; web-browser public web → existing world:443 rule 7). Quota check: chromium pod is the heaviest of the three at 1 cpu / 1Gi limit; total new requests ~400m cpu / ~896 Mi mem against 6 cpu / 12 Gi headroom — comfortable.

Phase 4 — DONE, still live: accessibility-tester (Deployment+Service) reuses the tier-1 agent template. Workload + Service + Secret named accessibility-tester (the host pi-mom dials via MOM_ACCESSIBILITY_TESTER_AGENT_URL); image vibes/agents/accessibility-tester. Runs pa11y + its own in-pod headless Chromium (--no-sandbox --disable-dev-shm-usage, so Chrome’s scratch lands in /tmp → roomier 1Gi /tmp emptyDir), so it’s the heaviest agent pod: 2 cpu / 2Gi limit (vs the others’ 1 cpu / 1Gi). Secret accessibility-tester (k8s name = dial-name) from kv/eso/accessibility-agent (operator-chosen seed name), single field ANTHROPIC_API_KEY. No netpol change (public web → existing world:443 rule 7). Quota re-checked (the plan’s gate): standing fleet ~8.25 cpu / ~8Gi at limits + 2 cpu / 2Gi here is well within vibes-quota (16 cpu / 24Gi limits) — no bump. ExternalSecret + kustomization + deploy-eso wiring + oci_mirrors passthrough + pi-mom prepare init wait + MOM_ACCESSIBILITY_TESTER_AGENT_URL env added.

Verify before live: the MOM_ACCESSIBILITY_TESTER_AGENT_URL var name was derived from the existing pattern (service name, dashes→underscores, uppercased: web-browser→MOM_WEB_BROWSER_AGENT_URL). Cross-check it against the slack-bot image’s start.sh / compose .env.slack-bot when the cluster is back — the var name determines the forward_* tool pi-mom registers.

All four phases went live; the cutover is complete — the compose stack is retired and the k8s pi-mom carries the real Slack app. The cross-cutting cutover caveats below are kept as the record of how it was done.

ESO convention (all five ExternalSecrets, this revision): the slack-bot + agent secrets use dataFrom: extract — the whole kv path materializes 1:1, Secret key = kv field name (matching agent-controller / agent-worker / restic-backup), replacing the old per-property data: mapping. kv fields are seeded UPPER_SNAKE with the bare credential name (SLACK_APP_TOKEN, ANTHROPIC_API_KEY, GH_TOKEN, …). The agents’ wrapper source was de-VIBES_’d (commit-this-revision) so the agents read the bare names directly — secret key == env var, no mapping. Only the bot keeps a secretKeyRef mapping (bare kv key → MOM_*), because the MOM_ prefix is upstream mom’s and can’t be renamed here. The seed blocks in initial-kv-seeds.sops.yaml were re-keyed to this scheme (slack-bot-pi-mom, general-agent, github-agent, web-browser-agent, + new accessibility-agent). Edit the SOPS file with sops, never a plain editor.

Remaining phases (for later impl)

Phase 5 — database-viewer ×2 (law, apiary) — NEVER BUILT

  • One image systems/database-viewer, two Deployments: database-agent-law and database-agent-apiary (Services matching the MOM_DATABASE_*_AGENT_URL). PGDB_DBNAME/PGDB_USER static env per Deployment.
  • Per-db credential isolation (preserve the compose intent, fix the shared env-file gotcha): ExternalSecret database-agent-creds from kv/eso/database-agent-creds with keys LAW_PROD_PASS / APIARY_PROD_PASS (+ optional *_DEVL_PASS); law binds PGDB_PROD_PASS←LAW_PROD_PASS, apiary ←APIARY_PROD_PASS. Each sees only its own.
  • netpol egress (the one new rule across the whole migration) — on-prem Postgres, hosts hardcoded in the agent prompt: 
    - toCIDRSet:
        - cidr: 10.112.12.120/32      # PROD postgres
        - cidr: 10.112.113.222/32     # DEVL postgres
      toPorts:
        - ports: [{ port: "5432", protocol: TCP }]
    Mirror netpol rule 8’s comment style. (If those IPs change, the agent image prompt/SKILL.md and this rule move in lockstep.)

Phase 6 — vibecoder workspace agents — NEVER BUILT AS DESIGNED

(The underlying goal — Slack-driven vibecoding of apps — was addressed differently: popquiz + rrroster run as k0s StatefulSets on the vibecoder image; see the popquiz/rrroster migration plan.)

  • The biggest lift. Per-project dev containers from separate gitignored repos (vibes-fastapi, vibes-nodejs), each an app + an embedded vibes-agent-wrapper agent, channel-pinned via MOM_WORKSPACE_URLS (name:url@CHANNELID). Stateful dev environments → own PVCs. Decide how those repos build/mount in k8s before planning concretely. Likely a separate effort.

Final — flip real Slack tokens — DONE (the live bot runs on the k8s stack)

  • If validated on a test Slack app (recommended path), the last step is swapping the real SLACK_APP_TOKEN/SLACK_BOT_TOKEN into kv/eso/slack-bot-pi-mom and rollout restart — once the fleet is complete enough that the real bot keeps its capabilities.

Cross-cutting

  • Cutover / Socket-Mode caveat: pi-mom is a single Socket-Mode connection — you cannot run the compose pi-mom and the k8s pi-mom on the same Slack app at once (duplicate replies). Stop compose pi-mom before the k8s one connects. At each phase the real bot only has the migrated capabilities (unmigrated forward_* tools are simply absent). Validate on a test Slack app to avoid a capability gap on the live bot.
  • Secret isolation: before seeding the real high-sensitivity tokens, decide whether to lock the operating agent out of reading them — see plan-agent-secret-isolation.md (operator note; write-only OpenBao seeding + a least-privilege agent kubeconfig that omits secrets, pods/exec|attach|ephemeralcontainers, and pod-create; plus the Incus and SOPS recipient controls that close the agent’s node-root and root-token read-paths).
  • cron: pi-mom’s start.sh seeds operator crons hardwired to #sysadmin (CLJUHQ30V) — they resume from the k8s bot on cutover. TAILER_CHANNEL_IDS on the tailer container seeds the channel-tail context (phase 1 set CLJUHQ30V; extend from the compose .env.slack-bot).

Verification (per phase)

  1. Image pulls: from any k0s node, crictl pull ghcr.io/chnm/systems/<path>:latest (the ghcr image path, e.g. vibes/agents/<name>, not the k8s <svc> dial-name) — containerd routes through Zot via hosts.toml (Pattern B). Bypass-Zot direct upstream pull is fine too if Zot’s down.
  2. ESO: kubectl -n vibes get externalsecret <svc> → SecretSynced (effect, not value — see the isolation note).
  3. Agent: in-cluster kubectl -n vibes run probe --rm -it --image=curlimages/curl -- curl -s http://<svc>.vibes.svc.cluster.local:9000/manifest; then a POST /prompt exercising a credentialed skill.
  4. pi-mom: logs -c pi-mom → probed each /manifest, registered forward_*, Socket-Mode connected; a Slack message routes to the new tool; single reply (no duplicate = clean handoff).

Open decisions

  • Secret isolation: the agent-view read-only kubeconfig role is the only piece of plan-agent-secret-isolation.md shipped so far (control B’s kubeconfig-side scope-down). Remaining work to land before seeding real tokens: write-only OpenBao seeding (control A), agent-only SOPS age key (control C) , and the Incus-side controls (control D). Until those land, decide per high-sensitivity field whether to seed real values now (agent- readable risk) or wait.
  • Test Slack app vs live cutover for validation. Resolved — cutover done.
  • Whether to share one vibes-agent-secrets (the Anthropic key) across agents vs a per-agent ExternalSecret (phases 1-4 used per-agent).

Phases 1-4 implemented 2026-06; remaining phases 5-6 + final flip pending. Templates: the phase-1→4 manifests infra/k0s/vibes/{slack-bot-pi-mom,general-agent, github-agent,web-browser,chromium,lightpanda,accessibility-tester}.yaml and infra/k0s/external-secrets/{slack-bot-pi-mom,general-agent,github-agent, web-browser,accessibility-tester}-externalsecret.yaml.