Firewall architecture

This page captures the firewall architecture across the entire infrastructure: bare-metal IncusOS hosts, k0s cluster VMs, GPU/passthrough VMs, and ad-hoc instances. It defines what each class of host should expose, who’s allowed to reach those services, and where the rules live.

The networks map in ansible/group_vars/all.yaml is the source of truth for source-network classification. Firewall rules reference those classes rather than hard-coded CIDRs, so adding/changing a campus subnet only requires updating one file.

Strategy: enforcement layers

Three layers, each scoped to what it can actually see and govern:

Layer	Scope	Where defined	Who owns
IncusOS host firewall	Coarse perimeter on the bare-metal host’s primary IP — Incus API (8443), etc.	IncusOS network seed (rendered from `ansible/playbooks/incus/templates/network.yaml.j2`)	IncusOS appliance
VM/container nftables	Coarse perimeter on each VM’s primary IP — what’s reachable from outside the cluster’s pod network	`ansible/roles/nftables/` (per-VM)	This repo
CNI NetworkPolicy (Cilium)	Fine-grained pod-level — namespace isolation, label selectors, L7 (HTTP path, gRPC method)	`k0s/<ns>/netpol.yaml`, plus cluster-wide default-deny in `apply-network-policies.yaml`	This repo

The key invariants:

Host nftables can’t see pods. Cilium uses an eBPF datapath; pod-to-pod traffic may never traverse the host kernel’s filter hooks. Don’t try to enforce pod policy at the VM nftables layer.
Cilium can’t lock down node-IP services. kubelet’s 10250, the API server’s 6443, etcd’s 2380 — those listen on the node’s IP, not a pod IP. Cilium has no policy primitive for them.
IncusOS hosts are managed by IncusOS, not Ansible nftables. Don’t try to apply this repo’s nftables role to bare-metal IncusOS hosts; their firewall posture is part of the IncusOS appliance contract.

Network source classification

ansible/group_vars/all.yaml defines four classes of network. Use these as the basis for firewall source rules:

Class	What it is	Firewall stance
`private`	Our infra subnets (CIDRs: `networks.private` in `group_vars/all.yaml`)	Trusted. Allow all admin traffic (SSH, API server, kubelet, etcd peers, jump-box access).
`client`	End-user campus subnets (`networks.client`)	Semi-trusted. Allow public service ports (NodePort range, Gateway API LoadBalancer VIPs, public web endpoints). No direct admin access.
`hpc`	HPC head + worker subnets (`networks.hpc`)	Need-to-know. Block by default. Allow only explicit interop ports (e.g. inbound from a specific HPC head node to a specific service VIP).
`public`	Internet — campus public v4/v6 ranges (`networks.public`)	Untrusted. Allow only externally-published services (Gateway API ingress, VPN endpoints if any). Never SSH, never admin ports.

Anything not matching one of these (private RFC1918 outside ours, link-local, etc.) gets the same treatment as public — default-deny.

Outbound (`output` chain) and `forward`

forward: leave at policy accept on k0s VMs. Cilium injects iptables/eBPF rules for pod-to-pod traffic, and nftables forward blocking would interfere with the CNI datapath. Pod egress restrictions belong in Cilium NetworkPolicy, not here.

output: policy accept is the pragmatic default. A locked-down output chain is a security-budget item. If you decide it’s worth doing later, allowlist per-host:

DNS (UDP/TCP 53) to upstream resolvers
NTP (UDP 123) to time servers
HTTPS (TCP 443) to image registries (registry-1.docker.io, ghcr.io, quay.io, internal Zot at the LoadBalancer VIP) and apt mirrors
Inter-cluster ports (per the inbound table above, mirrored)

Lots of effort, breaks routinely when something legitimate adds a new endpoint. Start with accept, narrow only if there’s a specific threat model that demands it.

Rollout plan

A locked-down host firewall is an easy way to silently break a cluster. Phase the rollout:

Phase 1: extend the nftables role to all VM-class hosts

The role currently runs only on k0s workers. Add include_role calls for:

k0s controllers (in setup-cluster.yaml controller play)
backup2 (already covered by setup-backup-servers.yaml Play I — init+nftables roles)
GPU VMs (logi on ibuypwr — wire when productionizing the Ollama serving path)

Keep the template with policy accept for now — this is just establishing coverage, not enforcement.

Phase 2: switch template to default-deny on one node

Apply the template above to a single k0s worker (not a controller — controllers losing 6443/2379 mid-rollout will lose the cluster). Use --limit:

ansible-playbook playbooks/k0s/setup-cluster.yaml \
  --tags firewall --limit <one-worker>

Run the smoke tests:

Cluster-internal pod-to-pod: see k0s-cross-subnet. Pin one netshoot pod to the firewalled node, verify cross-host pod ping.
kubelet reachability: kubectl logs <pod-on-firewalled-node>, kubectl exec, kubectl top node <that-worker> — all should work.
NodePort: curl http://<sasha-ip>:<exposed-nodeport>/ from the jump box (which lives in private).
Hold ~30 min and observe. Cluster controllers may take longer than smoke tests to surface issues — let it sit.

Phase 3: roll to remaining workers, then controllers

Order: workers (one at a time, smoke tests between) → secondary controllers → primary controller (odin). One node at a time.

If anything goes unhealthy and recovery isn’t obvious, revert by editing the template back to policy accept on that node and re-running the role.

Phase 4: tighten NodePort range to specific ports

Once the set of exposed NodePorts is stable, replace 30000-32767 with just the actually-used ports. Friction every time a new NodePort is added, in exchange for a tighter perimeter.

Phase 5: consider lockdown of `output` chain

Per-host-class allowlist for outbound traffic. Significant effort. Only worth it if the threat model includes pod-compromise lateral-movement defense, and Cilium NetworkPolicy isn’t sufficient.

Decision log

Choices made above that future-you may want to revisit:

Default-deny inbound, default-accept output and forward. Inbound is where untrusted-network risk lives; output and forward are where Cilium owns the policy and where premature lockdown breaks things. Revisit if a specific threat model demands locked-down output (compromise containment, regulated environments).
Static @k8s_nodes set instead of Cilium host firewall. The latter is more powerful (label-based, Cilium-policy-managed) but adds another moving part. For a fleet of 10 nodes the static set is easier to reason about. Switch when the fleet grows past ~10 nodes or dynamic node addition becomes routine.
networks from all.yaml as the source of truth. Keeps firewall and Ansible inventory unified. If all.yaml becomes stale, every firewall apply propagates the staleness — keep it accurate.
IncusOS bare-metal hosts intentionally out of scope for the nftables role. Their posture is part of the IncusOS appliance contract; if we ever need richer host-level firewall on them, that’s a separate doc on extending IncusOS or running a privileged firewall VM.
Allow full NodePort range initially (30000–32767). Phase 4 narrows this. Trade-off: friction per new exposed service vs slightly looser perimeter while the service set is in flux.

Re-evaluation triggers

Re-read this doc and possibly tighten rules if:

A specific compliance regime applies (HIPAA, PCI, FedRAMP, etc.) — those typically demand stricter egress controls and audit logging beyond this baseline.
A node or pod is breached — incident retrospective should question whether tighter rules would have helped contain it.
The cluster grows past ~10 nodes — the static @k8s_nodes set becomes annoying to maintain; switch to Cilium host firewall or templated-from-inventory.
A new threat model surfaces (e.g. internal pivot from a compromised pod) — Cilium NetworkPolicy is the right layer for that, but host firewall may also need adjustment.
Campus network policy changes (e.g. a new subnet is added to client or private) — all.yaml should be updated and the firewall re-applied across all hosts.