Kata RuntimeClass for hostile workloads

Design note — Phase 0 passed, nothing installed (last updated 2026-08-21; migrated 2026-09-05)

Status is as the doc says: nested virtualization is verified on all four Incus hosts, and nothing beyond that has been built — no RuntimeClass object, no Ansible role, no labeled workers. Phase 1 (one worker, the Juice Shop canary) is the next step. A few operator-side security specifics are stated generically in this copy.

Status: planned — Phase 0 PASSED on all four Incus hosts (2026-08-21; results table below). Nested KVM is available to VM guests fleet-wide, so the plan’s one plan-stopping risk is retired and the gVisor fallback is moot. Nothing installed yet; no RuntimeClass object exists on the cluster. Next step is Phase 1 on levi. Everything from here is additive and reversible.

Depends on: nested virtualization exposed to Incus VM guests — satisfied (theia, hyperion, kyojin, atomsk all verified 2026-08-21).

Blocks: nothing today. Would let the cluster-hardening effort (plan-cyberhardening-k0s-cluster.md, operator note) close its container-escape containment goal (container escape → root on the worker VM).

Background: the cyberhardening doc’s threat model — “some workloads are the hostile party” — plus the growth of the vibes / turnstone agent tiers, where model-chosen code execution is the product, not an incident. k0s/sandbox/README.md already states the invariant this plan hardens.

---

Motivation

The cluster’s containment story is strong at every layer except the kernel. PSA admission, Cilium default-deny + per-namespace egress CNPs, ResourceQuota / LimitRange, dropped capabilities, automountServiceAccountToken: false — all of it is enforced by one shared Linux kernel per worker VM. A single kernel LPE (the nf_tables, io_uring, overlayfs, and cgroup-v1 families all produced usable container escapes in recent years) defeats every one of those controls at once, because they are all implemented by the thing that just got compromised.

That matters more here than on a generic cluster, because this cluster runs workloads that are hostile by design or by construction:

Workload	Why the kernel boundary is the one that counts
devl/owasp-juice-shop	RCE-by-design. Its entire purpose is to be exploited.
sandbox/playwright-mcp	Renders attacker-chosen web content in Chromium. Browser renderer RCE → pod.
vibes agent-workers	Per-task Jobs whose job is to run bash/git/gh on model-chosen input.
turnstone-server-{0,1}	Judge-gated tool sessions — same shape, “governed” but not sandboxed.
vdi/* (kali, arch, …)	Interactive shells handed to humans, with an explicit attack toolkit.

Today, all of these share a kernel with each other and with whatever else the scheduler happened to place on that worker — including platform services holding real state. The namespace isolation is real at the network and API layers and absent at the kernel layer.

The cost of the gap is not theoretical but it is also not urgent: the workers are themselves VMs, so a container escape yields root on a 4 vCPU / 8 GiB disposable Debian VM, not on an R540. That is the reason this is a plan and not a todo — it buys a second containment ring behind one we already have, for workloads where the first ring is under active, deliberate attack.

The second, quieter motivation: PSA is stuck. Four namespaces sit at baseline (vibes, sandbox, turnstone, gitops) with comments in ansible/vars/k0s-namespaces.yaml all saying some version of “tighten to restricted once confirmed compatible.” A hardware-isolated runtime changes the calculus for exactly those namespaces — baseline-inside-a-VM is a stronger posture than restricted-on-the-shared-kernel, so the tightening stops being the only lever.

---

Current state

Nodes. 8 workers, all Incus virtual machines (kind = "virtual-machine" in opentofu/fleet.tf’s fleet_resource_class_spec), 4 vCPU / 8 GiB / 32 GiB, Secure Boot on, Debian 13 trixie, k0s v1.32.2, containerd 1.7.26, runc only. Two workers per Incus host across theia / hyperion / kyojin / atomsk — all Dell R540s on Intel Xeon Cascade Lake (Gold 5218, Silver 4214R, Silver 4208).

So Kata here means nested virtualization (KVM inside an Incus VM inside IncusOS). Cascade Lake has VT-x/EPT and kvm_intel.nested=1 has been the upstream default for years, so this is likely fine — but nothing in the repo configures or asserts it (grep -rn "nested\|vmx\|svm\|kvm" across fleet.yaml, opentofu/, and ansible/ returns no hits related to CPU features), and it has never been tested. This is Phase 0 and it is a hard gate.

Headroom. kubectl top nodes today:

	CPU	Memory
Range across 8 workers	165–306 m (4–7%)	2245–3183 MiB (28–40%)

~4.5–5 GiB free per worker. CPU is nearly idle. Memory is the binding constraint for Kata, not CPU — every Kata pod pays a VMM + guest-kernel tax that runc pods do not.

Extension mechanism already exists. roles/k0s_registry_mirror proves the pattern: k0s’s bundled containerd reads partial-override drop-ins from /etc/k0s/containerd.d/*.toml and merges them with its generated config. That role renders 00-registry-mirror.toml and notifies a Restart k0s handler. A Kata runtime handler is the same shape of change to the same directory. No new mechanism to invent, no fighting k0s over /etc/k0s/containerd.toml.

Nothing Kata-adjacent exists yet. No RuntimeClass objects, no gVisor, no prior art anywhere in the notes corpus.

---

Proposed target state

An opt-in, node-scoped, explicitly-declared second runtime. Nothing changes for any workload that doesn’t ask for it.

                    worker VM (Incus guest, Debian 13, nested KVM enabled)
  ┌──────────────────────────────────────────────────────────────────────┐
  │  k0s containerd 1.7.26                                               │
  │    ├── runc          (default)   ──► pod ──► shared worker kernel     │
  │    └── kata-qemu / kata-clh      ──► pod ──► ┌──────────────────────┐ │
  │          containerd-shim-kata-v2             │ guest kernel + agent │ │
  │          /opt/kata/…                         │  (per-pod, isolated) │ │
  │                                              └──────────────────────┘ │
  │  Cilium veth ──(tc-redirect)──► tap into the pod VM                   │
  └──────────────────────────────────────────────────────────────────────┘

The RuntimeClass

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: kata
handler: kata-qemu            # the containerd runtime-handler name
# PodOverhead (GA since 1.24) — kubelet adds this to scheduling AND to the
# pod cgroup, so the VMM's own footprint stops being invisible to the
# scheduler. On 8 GiB workers this field is not optional.
overhead:
  podFixed:
    memory: 256Mi
    cpu: 250m
# RuntimeClass admission stamps this onto every pod that selects the class,
# so Kata pods can only land on workers that actually have the runtime.
scheduling:
  nodeSelector:
    rrchnm.org/kata: "true"

overhead values above are a budget, not a measurement — Phase 1 measures the real number and this file gets corrected. Order-of-magnitude expectation: ~160–250 MiB/pod for kata-qemu, ~100–160 MiB for kata-clh, plus ~1–3 s (qemu) or ~0.5–1.5 s (clh) of extra pod start latency.

Node label rrchnm.org/kata: "true" follows the existing rrchnm.org/tier: … convention in ansible/vars/k0s-namespaces.yaml.

The containerd drop-in

/etc/k0s/containerd.d/10-kata.toml, rendered by Ansible, deliberately mirroring k0s-containerd-drop-in.toml.j2’s shape and header comment:

version = 2

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu]
  runtime_type = "io.containerd.kata.v2"
  # Kata gives privileged pods a *guest* /dev, not the host's — this is what
  # makes `privileged: true` inside Kata a bounded rather than total grant.
  privileged_without_host_devices = true
  pod_annotations = ["io.katacontainers.*"]
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.kata-qemu.options]
    ConfigPath = "/opt/kata/share/defaults/kata-containers/configuration-qemu.toml"

One runtime_type (io.containerd.kata.v2 → shim binary containerd-shim-kata-v2) with a per-handler ConfigPath — rather than kata-deploy’s io.containerd.kata-qemu.v2 symlink-per-hypervisor convention. Fewer moving parts, and adding kata-clh later is one more block pointing at configuration-clh.toml. The shim binary needs to be on k0s-containerd’s PATH: symlink /opt/kata/bin/containerd-shim-kata-v2 into /usr/local/bin.

The Ansible role

New roles/k0s_kata_runtime, applied to k0s_workers only (controllers have no kubelet), with a make k0s-kata target in mk/k0s.mk:

1. Assert /dev/kvm exists and is usable — fail loudly, don’t skip.
2. Install the pinned kata-static-<ver>-amd64.tar.xz release tarball to /opt/kata (version pinned in defaults/main.yaml, same discipline as k0s_version / Cilium / Tetragon). Candidate for the existing oci-mirrors flow so the workers don’t fetch from GitHub directly — see open questions.
3. Symlink the shim onto PATH.
4. Render /etc/k0s/containerd.d/10-kata.toml; notify: Restart k0s.
5. Label the node rrchnm.org/kata=true — gated on a k0s_kata_enabled host/group var, so enabling a worker is a one-line inventory change, not a role edit.

k0s-kata must join the make k0s band, immediately after k0s-registry-mirror (mk/k0s.mk:66). This is not cosmetic: make k0s-reset rolls every node back to its post-cloud-init snapshot, which restores cloud-init’s seeded /etc/k0s/containerd.d/00-mirror.toml but wipes both /opt/kata and the Ansible-rendered 10-kata.toml. The RuntimeClass object survives in etcd (or is recreated), so the failure mode after a reset is a RuntimeClass pointing at a handler containerd no longer has — pods fail with an opaque shim error rather than an obvious “not installed.” Being in the band makes the reset path self-healing.

Deliberately not kata-deploy. That DaemonSet rewrites /etc/containerd/config.toml from inside the cluster, which fights k0s for ownership of node config and inverts the repo’s posture that node state is Ansible’s. The drop-in path is both supported by k0s and consistent with k0s_registry_mirror.

Opting a workload in

One line, explicit, in the manifest — no namespace-wide defaulting, no mutating webhook:

spec:
  runtimeClassName: kata

For the vibes agent-workers this lands in k0s/vibes/irc/agent-worker-template.yaml’s PodTemplate, which the agent-controller reads at spawn time — so the next task picks it up with no controller restart. That file’s own header already argues for exactly this: “worker shape — image, resources, securityContext — belongs to the worker, not the orchestrator.”

Why this approach (and not the alternatives)

Option	Isolation	Nested virt?	Cost on 8 GiB workers	Verdict
Status quo (PSA + CNP + Tetragon)	shared kernel	no	zero	insufficient — one LPE defeats all of it at once
Kata Containers	per-pod guest kernel	required	~150–250 MiB/pod	chosen — real kernel boundary, standard CRI integration, opt-in per pod
gVisor (runsc)	userspace kernel (ptrace/KVM)	not required	~50–100 MiB/pod	the fallback if Phase 0 fails — but syscall-compat gaps bite browsers, JITs, and Go runtimes; the Sentry still issues host syscalls
Dedicated Incus VM per workload	full VM	no (sibling, not nested)	very high	rejected — loses k8s scheduling/ingress/policy; we deliberately moved toward the cluster
Per-tier k0s worker pools (taints only)	shared kernel per tier	no	one more VM per tier	complementary, not a substitute — reduces blast radius, doesn’t add a boundary

Kata wins on being a real boundary with standard integration: it’s a CRI runtime handler, so RBAC, Cilium policy, quotas, kubectl exec, and kubectl logs all keep working unchanged. gVisor is genuinely competitive on cost and is the right answer if nested virt turns out to be unavailable — but Chromium (playwright-mcp, lightpanda) and the Node/Go runtimes in the agent images are exactly the workload classes where runsc’s syscall surface causes trouble, and those are the workloads this plan exists to contain.

---

Workload assessment

The admission test, borrowed from k0s/sandbox/README.md and sharpened for this decision:

Does this pod execute code chosen by something we don’t trust — and would a host-kernel escape from it reach something that matters?

Both halves must be yes. A pod that runs untrusted code but sits on a worker alone gains less than one that shares a worker with a platform service holding real state.

Tier A — strong candidates, do these first

Workload	Ns	Why	Notes
playwright-mcp	sandbox	Chromium rendering attacker-chosen pages. The namespace’s whole charter is “assume full compromise.”	Already baseline PSA, no secrets, no LAN. Needs real /dev/shm (per its README) — Kata provides it inside the guest, but validate the shm sizing survives the move. Best first real workload: single pod, stateless, blast radius already contained.
owasp-juice-shop	devl	RCE-by-design; the safest possible test subject because breaking it costs nothing.	Stateless, 1 pod, no PVC, no secrets. Effectively a free canary — put it in Phase 1, not Phase 2.
agent-worker Jobs	vibes	Per-task pods that shell out to bash/git/gh on model-chosen input. Ephemeral, no SA token, 2 GiB, PVC workspace.	The highest-value target in the cluster. Opt-in is one line in the PodTemplate and takes effect on the next spawn. Overhead is per-task and transient.
vdi/* (kali, arch, debian, alpine)	vdi	Interactive shells + browsers handed to humans; Kali is an explicit attack toolkit.	Namespace is empty today — so this is free to specify now and costs nothing until desktops come back. Do it as a manifest change whenever make k0s-vdi next runs.

Tier B — good candidates, once Tier A has run a while

Workload	Ns	Why	Caveat
turnstone-server-{0,1}	turnstone	The tool-running surface; k0s-namespaces.yaml already calls these “the sensitive surface.”	StatefulSet with a PVC and DB connections — more moving parts than Tier A. Move after vibes proves the shape.
lightpanda	turnstone	Headless browser; same untrusted-content argument as playwright.	Stateless — genuinely easy, just lower-traffic than playwright.
general-agent, web-developer, accessibility-tester	vibes	Long-lived agent pods that execute model-chosen code.	These hold ESO-materialized API keys, so Kata protects them rather than protecting the cluster from them. Real but second-order value.
slack-bot-pi-mom	vibes	Same agent shape, processes untrusted inbound Slack content.	2 containers; verify sidecar behavior under Kata.

Tier C — no meaningful benefit; leave on runc

prod/analytics-dashboard, platform/forgejo, platform/headlamp, gitops/argocd-*, observability/grafana, vibes/{ergo,gamja,dashboard,relay-client}, cert-manager, external-secrets, crowdsec-alerts, idrac-exporter, metrics-server, coredns, hubble-*.

Operator-curated code paths that don’t execute third-party input. Kata would cost memory and pod-start latency to defend against a threat that isn’t in the model. openbao-0 is a deliberate omission from Tier B — it is high-value, but it’s a target to be protected, not a source of untrusted execution, and Kata is the wrong tool for that job (network policy and RBAC are the right ones).

Tier D — technically incompatible; must NOT be Kata

Workload	Blocker
cilium, cilium-envoy (DS)	Host-network + eBPF program loading into the host kernel. Kata isolates it from precisely the kernel it must program.
tetragon (DS)	Same — eBPF kprobes on the host kernel.
alloy (DS)	hostPath log scraping + journal access.
konnectivity-agent (DS), nllb (static pods)	Node-local control-plane plumbing; nllb are k0s-managed static pods and not ours to annotate.
local-path-provisioner	Creates hostPath dirs on the node filesystem.

Rule of thumb: anything with hostNetwork, hostPath, or eBPF is disqualified outright. Every current DaemonSet in the cluster is in this category.

Tier E — contraindicated on performance / correctness grounds

turnstone/postgres-0, observability/{loki,prometheus}, crowdsec-0, platform/openbao-0, and especially vibes/popquiz-0 and vibes/rrroster-0.

Kata surfaces volumes through virtio-fs, not a direct host mount. That is fine for a code workspace and bad for anything doing small synchronous writes with strict durability expectations. popquiz and rrroster run SQLite on a local-path PVC — SQLite’s locking and fsync semantics over virtio-fs are a known sharp edge, and the cluster already has a documented etcd fsync-latency problem (disk-performance.md) that says the storage layer here has no margin to give away. Keep stateful datastores on runc.

Summary

Roughly 6–8 long-lived pods plus the transient agent-worker Jobs are worth moving. At a 256 MiB budgeted overhead that is ~2 GiB of additional cluster memory demand spread across workers that currently have ~4.5 GiB free each. It fits without resizing the workers — but it fits with less margin than the raw numbers suggest, because Kata pods can’t be spread arbitrarily: they land only on rrchnm.org/kata=true nodes.

---

Sequencing

Phase 0 — verify nested virt (hard gate, ~30 minutes)

Nothing else in this plan is worth doing until this passes. Check from inside the worker VMs, not from the host — IncusOS provides no general shell, and /1.0/resources doesn’t expose CPU feature flags, so kvm_intel.nested isn’t readable through the Incus API. The guest-side answer is the dispositive one anyway: if the guest has vmx and /dev/kvm, nesting works end to end.

cd .../infra/ansible          # ansible.cfg: inventory=fleet.yaml + private_key_file

# 1. Flags + device, across all 8 workers at once
ansible k0s_workers -m shell -a \
  'grep -c vmx /proc/cpuinfo; test -e /dev/kvm && echo "KVM: yes" || echo "KVM: NO"'

# 2. Functional proof — flags can be present while /dev/kvm still won't open
ansible k0s_workers -m shell --become -a \
  'command -v kvm-ok >/dev/null || apt-get install -y cpu-checker; kvm-ok'

Result	Meaning
vmx > 0 and KVM: yes	Pass — proceed to Phase 1.
vmx > 0, KVM: NO	Module not loaded: modprobe kvm_intel, check dmesg | grep -i kvm.
vmx == 0	Incus isn’t exposing VMX — diagnose at the Incus layer, not in k0s.

Without the ansible key, the same check rides the Incus API path make host-specs uses (needs incus-agent in the guest):

INCUS_CONF=/configs/incus incus exec theia:thor -- \
  sh -c 'grep -c vmx /proc/cpuinfo; ls -l /dev/kvm'

If vmx is absent, inspect the instance config — incus config show <host>:<inst> --expanded — and look for migration.stateful: true, which forces Incus off -cpu host onto a migration-safe baseline model that drops vmx. Checked on hyperion:thor (2026-08-21): absent, as are raw.qemu and any CPU-model override — limits.cpu: "4" is a vCPU count only. So Incus is launching these VMs with -cpu host and the guest question reduces entirely to kvm_intel.nested on the Incus host. Per-instance override of last resort: incus config set <inst> raw.qemu -- '-cpu host,+vmx', then restart.

That check also confirmed volatile.last_state.agent: STARTED on thor, so incus exec is a working path into the workers and the ansible key is not required for Phase 0. Re-run the check on one worker per Incus host — the four R540s carry different CPUs (Gold 5218 on theia/hyperion, Silver 4214R on kyojin, Silver 4208 on atomsk) and were racked at different times; don’t generalize from one node.

Expectation: this passes on all eight. Cascade Lake has VT-x/EPT, kvm_intel.nested has defaulted to Y upstream for years, and nothing in the IaC narrows the guest CPU model.

Phase 0 results — PASSED, all four hosts (2026-08-21)

Host	CPU	Instance probed	Class (kind)	vCPUs	vmx lines	/dev/kvm
hyperion	Xeon Gold 5218	thor	k0s_workers (VM)	4	8	present
theia	Xeon Gold 5218	zagreus	k0s_controller (VM)	2	4	present
kyojin	Xeon Silver 4214R	erwin	k0s_controller (VM)	2	4	present
atomsk	Xeon Silver 4208	jean	docker (VM)	4	8	present

The pass condition is vmx lines == 2 × vCPUs, not any fixed number. Intel guests with real VMX exposure print two matching lines per processor: the general flags: line containing vmx, and a separate vmx flags: line enumerating VMX secondary capabilities (vnmi, preemption_timer, invvpid, ept_x_only, …). The second line appears only once the kernel has read the VMX capability MSRs, so it is evidence VMX is functionally exposed rather than merely a CPUID bit. The 4s above are 2-vCPU controllers and are exactly as healthy as the 8s — don’t misread a low count as a degraded result. A count equal to 1× the vCPU count would mean the flag is advertised but the capability MSRs weren’t decoded; investigate rather than proceed.

Probe a VM, never a container. Several fleet classes are kind = "container" (caddy_proxies, dns_*, garage_cluster, oci_mirrors, caddy_kv_stores) and share the host kernel — their /proc/cpuinfo shows the host’s flags directly and proves nothing about passthrough to VM guests. All four probes above are kind = "virtual-machine" per fleet_resource_class_spec, which is what makes them dispositive.

One VM per host is sufficient: VMX exposure is a property of the host’s KVM module plus the CPU model Incus presents, and thor’s expanded config confirmed no k0s class carries a CPU-model override (no migration.stateful, no raw.qemu; limits.cpu is a count). So the untested k0s workers on theia, kyojin, and atomsk inherit the same result. Phase 1 lands on levi (kyojin), covered by erwin.

Don’t bother installing cpu-checker for kvm-ok: /dev/kvm existing at all implies the module loaded and VMX initialized, and Phase 1 is a far better functional proof — installing Kata and booting the Juice Shop canary exercises nested KVM harder than kvm-ok does, and it’s on the critical path regardless.

Verify: boot a throwaway VM inside a worker VM — qemu-system-x86_64 -enable-kvm -m 128 -display none -nodefaults -no-reboot (fails fast with Could not access KVM kernel module if nesting is off; reaching SeaBIOS / “no bootable device” means acceleration is live). If this fails and can’t be fixed at the Incus layer, stop and re-scope to gVisor — the workload assessment above transfers unchanged; only the runtime and the Ansible role differ.

Phase 1 — one worker, one canary

Pick levi (kyojin, lowest current memory use at 28%, and it already hosts Juice Shop).

1. Write roles/k0s_kata_runtime + make k0s-kata; apply to levi only.
2. Apply the RuntimeClass; label levi with rrchnm.org/kata=true.
3. Move devl/owasp-juice-shop — the free canary — to runtimeClassName: kata.

Verify, all of it, before touching anything else:

• Pod reaches Running; kubectl exec and kubectl logs both work.
• uname -a inside the pod shows the Kata guest kernel, not the worker’s 6.12.x+deb13-amd64. This is the single assertion that proves the boundary exists.
• Cilium policy still enforces. Kata plugs the CNI veth into the pod VM via tc-redirect; confirm the devl default-deny and the egress CNP still drop what they should, and that Hubble still shows flows for the pod. Treat a failure here as plan-stopping, not as a tuning exercise.
• Measure real memory overhead (systemd-cgls / node memory delta) and correct the overhead.podFixed values in the RuntimeClass.
• Measure pod start latency delta.
• Confirm k0s restart / node reboot brings the drop-in back cleanly and containerd still lists both handlers.

Phase 2 — the sandbox tier

Move sandbox/playwright-mcp. Label a second worker so the tier isn’t single-homed. Re-validate /dev/shm sizing and the crash fix its README documents, plus browser-heavy memory behavior under virtio-fs.

Verify: the playwright.mcp.rrchnm.internal MCP flows in /claude/CLAUDE.md still work end-to-end — navigate, snapshot, screenshot — and screenshot extraction via kubectl cp still works.

Phase 3 — the agent tier

Add runtimeClassName: kata to k0s/vibes/irc/agent-worker-template.yaml. Takes effect on the next spawned task with no controller restart. Watch vibes-quota — pod overhead counts against ResourceQuota, so requests.memory: …/12Gi will start including it.

Verify: a full task lifecycle — spawn, PVC attach at /workspace, agent shells out, Job completes, agent-pvc-reaper cleans up.

Phase 4 — Tier B + the vdi manifests

turnstone-server, lightpanda, the long-lived vibes agents; add runtimeClassName: kata to the vdi/ manifests so desktops come back isolated. Expand node labelling to whatever the placement math requires.

Phase 5 — make it policy, not habit

Two things that stop this from silently rotting:

1. A smoke test asserting that every pod in sandbox, devl, and vdi has runtimeClassName: kata — the repo already treats gateway-IP drift as a smoke-test failure rather than a silent one; same discipline here.
2. Fold the runtime choice into k0s/sandbox/README.md’s admission test, so “does it go in sandbox?” and “does it run on Kata?” are one decision.

If Kyverno lands (cyberhardening item 12), a mutating policy defaulting runtimeClassName per namespace replaces (1) — but explicit-in-manifest matches the repo’s house style and needs no admission webhook.

Phase N — cleanup

Nothing to decommission; runc remains the default and every non-opted pod is untouched. If the plan is abandoned mid-flight, see Rollback.

---

Risks

• Nested virt unavailable. RETIRED 2026-08-21 — verified on all four Incus hosts (Phase 0 results). The gVisor fallback is no longer load- bearing; the comparison table above is kept as the decision record only. Stability under sustained Kata load remains unproven — that’s Phase 1’s job, not a separate risk.
• Tetragon and EDR go blind inside Kata pods. This is the real trade-off and it deserves to be stated plainly. Tetragon’s kprobes, and host-side EDR, observe the host kernel. Processes inside a Kata pod execute against the guest kernel and become invisible to all of them — host-kernel enforcement rules stop applying to exactly the workloads they were written for. We trade detection inside the sandbox for a boundary around it. That is usually the right trade for hostile-by-design workloads and it is not free. Two mitigations, neither complete: Cilium/Hubble network visibility is unaffected (enforcement is on the host-side veth), so network-layer detection survives intact; and Kata escape attempts that reach the host kernel — the thing we actually care about — remain visible to Tetragon. Decide this consciously before Phase 3, and record the decision.
• Memory pressure on 8 GiB workers. Kata pods are pinned to labeled nodes, so overhead concentrates rather than spreads. Mitigated by declaring overhead.podFixed (so the scheduler stops being blind to it), correcting it from Phase 1 measurements, and labelling ≥2 workers before Phase 3. Escape hatch: bump k0s_workers mem in fleet_resource_class_spec — the hosts have 64 GiB and are not the constraint.
• virtio-fs I/O regressions. Mitigated by Tier E: no stateful datastore moves. Watch the vibes agent-worker PVC at /workspace in Phase 3 — that’s the one Tier A workload doing real file I/O.
• k0s regenerating containerd config. The drop-in directory is the supported merge point and k0s_registry_mirror has been living there without incident. Phase 1 explicitly tests restart + reboot survival.
• Cilium ↔ Kata datapath interaction. Kata’s tcfilter networking model plus Cilium’s eBPF host-routing is the least-proven interface in this plan. Phase 1 gates on it; see open questions.
• Upgrade burden. A pinned Kata version means a guest kernel we now own and must patch — a new artifact in the same class as k0s_version / Cilium / Tetragon. Add it to upstream-watch.md.
• Secure Boot. No conflict: the worker VM’s Secure Boot governs its boot chain; Kata direct-kernel-boots its guest inside an already-booted VM.

Rollback

Fully additive, so rollback is clean at every phase:

1. Remove runtimeClassName: kata from the manifests; re-apply. Pods reschedule onto runc immediately.
2. kubectl delete runtimeclass kata; remove the rrchnm.org/kata node labels.
3. Remove /etc/k0s/containerd.d/10-kata.toml and restart k0s. containerd returns to a runc-only handler set.
4. rm -rf /opt/kata and the shim symlink.

No etcd state, no CRDs, no storage migration, nothing to un-migrate. A stalled rollout can also simply be left — labeled nodes that no pod selects cost nothing.

---

Open questions

• Does Incus expose VMX to VM guests on these hosts? Answered 2026-08-21: yes, on all four. See Phase 0 results.
• kata-qemu or kata-clh? Cloud Hypervisor boots faster and uses less memory; QEMU is the better-trodden path with wider device support. Both can be installed side by side as two handlers off one tarball. Suggest starting kata-qemu for compatibility, benchmarking kata-clh in Phase 2, and switching if the memory delta is material on 8 GiB nodes. (kata-fc / Firecracker is out — it needs the devmapper snapshotter and lacks virtio-fs.)
• Does Cilium eBPF host-routing need adjustment for Kata’s tcfilter networking? The interaction is documented upstream but not proven here. Answer empirically in Phase 1 by testing policy enforcement and Hubble flows on the Juice Shop canary before anything else moves.
• Do we accept the Tetragon/EDR blind spot? See Risks. Needs an explicit decision recorded before Phase 3, not an implicit one discovered later.
• Mirror the Kata tarball through oci-mirrors/Zot, or fetch from GitHub? Consistent with the fleet’s supply-chain posture to mirror it; it’s a release tarball rather than an OCI artifact, so it may not fit the existing oci-mirrors flow cleanly. Decide when writing the role.
• Which workers get labeled? Placement interacts with k0s-cluster-rebalance.md. Starting proposal: levi (kyojin) + one AQ114 worker, so the Kata tier spans two sites and two Incus hosts.
• Does Kata + restricted PSA compose cleanly? If yes, the sandbox and vibes namespaces could get both, and the “tighten to restricted later” comments in k0s-namespaces.yaml finally resolve.

---

Related docs

• plan-cyberhardening-k0s-cluster.md (operator note) — the threat model this plan serves; this plan closes its container-escape containment goal and narrows its host-kernel enforcement scope for Kata-run pods.
• k0s/README.md (systems repo) — namespace layout and workload list; the workload assessment above is keyed to it.
• k0s/sandbox/README.md (systems repo) — the untrusted-workload admission test this plan extends from “which namespace” to “which runtime.”
• disk-performance.md — why Tier E keeps stateful datastores off virtio-fs.
• k0s-cluster-rebalance.md — topology; informs which workers get labeled.
• upstream-watch.md (operator note) — where a pinned Kata version and its guest kernel need to be tracked.
• ansible/roles/k0s_registry_mirror/ — the drop-in + Restart k0s pattern roles/k0s_kata_runtime copies.
• k0s/vibes/irc/agent-worker-template.yaml — the highest-value opt-in site; one line, picked up at next spawn.

---

Last updated: 2026-08-21