Voice client

Status (2026-08-30): proposal / design. Nothing built. Establishes what a voice-driven client needs from the (now headless) agent-controller, and — the load-bearing part — the handful of controller changes that turn “possible” into “good.” Written off the agent-dispatch-unification cutover: the controller is a headless dispatcher with a typed command API + event stream, and clients-irc proved a client is just a drive-plus-observe adapter. A voice client is the same shape with audio at the edges.

Goal

A hands-free client: speak a request, hear the answer; file tasks and get spoken notifications when they’re ready. It rides the same typed interface as clients-irc and pi-mom — no privileged path, no second brain. The controller stays the one place that owns state; the voice adapter owns only audio and translation.

Why it’s cheap now (and wasn’t before)

The irc-ectomy reduced “a client” to two seams:

Drive — HTTP to the typed API (/v1/ask, /v1/tasks, /v1/hubs, /v1/capabilities) carrying an actor.
Observe — subscribe to /v1/events and render.

clients-irc is a small binary doing exactly that (parse !cmd → API; SSE → IRC render). A voice client swaps IRC for audio. The invariant generalizes cleanly: nothing that owns state speaks a chat — or audio — protocol.

The pipeline

   mic ─▶ wake-word ─▶ STT (Whisper) ─▶ [intent] ─▶ controller API
                                                          │
 speaker ◀─ TTS (Piper) ◀─ [render] ◀─ /v1/ask stream + /v1/events (SSE)

The thin-adapter move — offload NLU to the controller. Don’t build a heavy intent parser. The controller already has /v1/ask; make an LLM capability be the NLU: transcribe → POST the raw text to a voice/general capability → speak the reply. Only the few lifecycle actions (“create a task to…”) take a second branch. The adapter is ~“STT → ask → TTS” + a thin task-intent branch — mirroring clients-irc’s “parse → API.” Intelligence lives in the capability, not the transport.

Controller changes — “possible” → “good”

Voice is a forcing function: each item below is exposed by voice but helps every interactive client, which is exactly why it belongs in the controller and not the adapter.

1. Streaming `/v1/ask` (the one real feature; two-sided)

Today /v1/ask blocks and returns the whole answer — for voice that is dead air until the agent finishes (a 5 s answer is 5 s of silence). Add a streamed variant, peer to the existing /v1/events?stream=1:

POST /v1/ask with stream=1 (or Accept: text/event-stream) → SSE of token deltas: data: {"delta":"…"} … terminal data: {"done":true,"ok":true,"duration_ms":N}.
Two-sided: the warm agent’s <endpoint>/prompt (vibes-agent-wrapper) must stream too (SSE / chunked from the model); the controller passes the stream through off-lock (the ask proxy already runs off the bot lock — Phase 3). Agents that can’t stream → controller buffers and emits one delta + done (graceful fallback, so a non-streaming agent still works).
The event log is unchanged: ask_started / ask_completed still fire for #firehose; the stream is the direct client channel, a peer to the log — the same split as /v1/events?stream=1 vs #firehose.
Payoff beyond voice: progressive replies in Slack and any interactive client.

2. Ask cancellation / barge-in (small; controller-owned)

Voice users interrupt — say “stop,” or start a new request mid-answer. The controller owns the in-flight round-trip, so it must be able to abort one: give each ask a correlation id; DELETE /v1/ask/{id} or client-closes-the-SSE → cancel the context, stop the agent proxy, emit ask_cancelled. The adapter, on detecting fresh speech, cancels the in-flight ask and starts the new one. Also useful for Slack (edit / retract).

3. Scoped event subscription — ✅ BUILT + VERIFIED (`e955c51c`, 2026-08-30)

/v1/events is a firehose. A voice client should speak only your stuff, not drain the world and discard 95 %. Server-side filters on the existing cursor read + live tail: GET /v1/events?kind=a,b&task=<id>&since=&stream=1 — kind is OR/comma-list, task exact, AND-composed. pi-mom wants a single thread’s events too — this isn’t voice-specific. (actor= is out: events carry no actor field.)

Landed on cc-quiet-yarrow (e955c51c): both paths filtered — cursor read + SSE (replay & live tail) — with the narrowing pushed into SQL so limit counts matching events. EventFilter.Match drives the live predicate; zero value = unchanged behavior, so existing clients (clients-irc) are unaffected. Tests: unit + cursor + a deterministic SSE test + a store test. Verified green on moby (the Dockerfile builder’s go vet + go test ./... + build). Ships with the next controller image roll — pure additive API, no manifest change.

4. A `voice` channel_type + session TTL (model extension; Phase 2 territory)

The channel_users map already anticipates multiple channel_types. Admit voice: a voice identity (paired-device token / passphrase) maps to an internal user for attribution + ACLs (the /link flow analog). Add session TTL + explicit end to /v1/ask’s session so context doesn’t bleed between users on a shared device (DELETE /v1/sessions/{id}, or close-on-silence).

5. Idempotency + preview on side-effecting calls (small; safety)

STT mishears — a misheard “create a task…” / “delete…” must not double-fire. POST /v1/tasks accepts a client-supplied Idempotency-Key (e.g. a hash of utterance + timestamp) → dedupe within a window; optional ?preview=1 returns the resolved intent (title, repo, pod/secret) without creating, for the adapter’s confirm dance. The confirmation dialog stays adapter-side; the controller just supplies safe primitives. Benefits every unreliable-channel client.

Net: one real feature (streaming ask), three additive API surfaces (cancel, scoped events, idempotency/preview), one model extension (voice channel_type + session TTL). No voice special-casing — it’s the API maturing under a client that finally demands streaming and interruption. Build streaming + cancel + scoped events and Slack + every future client get better for free.

What stays OUT of the controller

Adapter / edge / config, deliberately:

STT, TTS, wake-word, audio transport — the adapter / edge.
The confirm-before-side-effect dialog — adapter UX (built on the controller’s preview/idempotency primitives).
Concise-for-speech answers — a voice capability system prompt in capabilities.yaml, not controller code (shapes: [ask], endpoint = a wrapper tuned for short spoken replies).
Text normalization for TTS (numbers, code, URLs → speakable) — the adapter.

Request classes

Utterance	Adapter	Controller
“status of the popquiz deploy?”	`POST /v1/ask?stream=1 general`	warm agent streams → TTS as it speaks
“create a task to fix homepage a11y”	`POST /v1/tasks?preview=1` → confirm → `POST` w/ Idempotency-Key	draft, bind repo, spawn worker
“what’s pending?”	`GET /v1/tasks?status=pending&mine=1`	speak the list
(task → for_review)	`/v1/events?actor=…&kind=status` → TTS	“Task X is ready for review”

Attribution + security

Voice is ambiguous and spoofable. Rules:

Map the speaker to an internal actor via the voice channel_type (paired device / passphrase); never trust raw audio for privileged ops.
Anything with a lifecycle or side-effect goes through preview → explicit spoken confirm (“file a task titled X — yes?”), gated on a confirmed actor. Read-only ask needs no confirm.
Wake-word + push-to-talk over always-listening; the edge device holds no secrets (the in-cluster adapter holds AGENT_API_TOKEN, seeded like clients-irc’s).

Deployment shape

clients-voice Deployment in vibes (mirrors clients-irc): reaches agent-controller-api.vibes.svc.cluster.local:9101 (drive) + the event stream (observe); a CNP allowing clients-voice → agent-controller:9101 (mirror the agent-controller-ingest pattern that admits clients-irc — netpol.yaml is Ansible-applied via make k0s-vibes, not Argo).
STT/TTS on logi (ibuypwr, RTX 2080 SUPER) — Whisper (faster-whisper) + Piper. Shares the GPU with Ollama → watch contention; pin small models (whisper-base, a light Piper voice). A cloud STT/TTS is a valid MVP shortcut to defer the GPU decision.
Audio transport is the real build, and it’s off-cluster: a web page over WebRTC (mic + speaker, nothing to fabricate) for the MVP; later a Pi / ESP32 satellite (wake-word at the edge, stream audio to the adapter). The controller integration is trivial by comparison.

Sequencing

Order	Chunk	Controller change	Standalone
1	MVP — WebRTC page → `clients-voice` → Whisper → `POST /v1/ask general` (blocking) → Piper	none (today’s `/v1/ask`)	✅ proves the seam
2	Streaming ask — `?stream=1` + wrapper streams `/prompt`	#1 (two-sided)	✅ kills the dead air
3	Barge-in — ask cancel on new speech	#2	✅
4	Notifications — scoped `/v1/events` → TTS	#3 ✅ built (`e955c51c`)	✅
5	Task path — preview → confirm → idempotent `POST /v1/tasks`	#5	✅
6	Identity — `voice` channel_type + session TTL	#4	rides Phase 2 attribution
7	Satellite — wake-word + edge device	none (adapter/edge)	✅

Build the MVP against today’s API first — it needs zero controller change and tells you whether the audio loop is worth maturing before you spend on streaming.

Honest cost / risks

Audio transport is the actual work. STT/TTS/API-bridge is a weekend; a reliable low-latency mic/speaker path (echo cancellation, VAD, barge-in) is not. WebRTC MVP first.
Latency. STT + ask + TTS. A blocking warm ask was ~1 s (measured — general returned “pong” in 992 ms); with STT + TTS ≈ 2–3 s. Streaming (chunk 2) is what makes longer answers tolerable.
Streaming is two-sided. The wrapper /prompt must stream, or the whole chain buffers — the only change here that touches code outside the controller’s own API package.
GPU contention with Ollama on the single box; the MVP can start on cloud STT/TTS.
Mishears + destructive ops — mitigated by preview/confirm + idempotency (change 5), but the confirm UX must be genuinely hard to trigger or bypass by accident.

Cross-references

agent-dispatch-unification (the headless-controller cutover this builds on — the ask path in Phase 3, attribution in Phase 2, the clients-irc adapter precedent, the /v1/events stream) · irc-ai-agents (the controller’s design of record; the event log / firehose) · k0s/vibes/irc/capabilities.yaml (where a voice capability lands) · playbooks/ops/setup-ollama-servers.yaml + logi / ibuypwr (the GPU for STT/TTS).