Bridge meshing across hosts

This document compares methods for providing cross-host connectivity between Incus instances attached to per-host tfbr0 bridges (the TF-managed bridges declared in opentofu/system_network.tf; not to be confused with the auto-created incusbr0 that the default profile uses). The five IncusOS hosts (kyojin, theia, hyperion, atomsk, ibuypwr) sit across multiple LAN subnets that are L3-routable via campus routing. The goal is for an instance attached to a bridge on any host to reach instances on the other hosts’ bridges directly, with whatever isolation, performance, and operational properties best fit a production deployment.

This was written while evaluating Incus as the XCP-ng replacement (that migration has since happened — Incus is production). Decisions here scale beyond the current 5-host setup; “homelab simplicity” is not the framing.

Where we are today

kyojin:tfbr0 — 10.164.202.0/24, NAT’d through kyojin’s host IP (10.112.113.202)
theia:tfbr0 — 10.114.104.0/24, NAT’d through theia’s host IP (10.112.12.104)
hyperion:tfbr0 — 10.114.103.0/24, live (hyperion has been operational since 2026-05-13)
ibuypwr and atomsk have no tfbr0 — ibuypwr serves only the logi GPU VM (macvlan); atomsk’s instances are likewise macvlan/LAN-attached (tfbr0_subnets in system_network.tf covers only kyojin/theia/hyperion)
Hosts on separate L2 broadcast domains; L3 routing between them via campus
No mesh — instances on each bridge are reachable only via their host’s NAT, and they can’t reach other hosts’ bridge subnets at all

XCP-ng equivalents (context for the migration)

XCP-ng pools handle cross-host networking through:

Pool-wide management network — control plane traffic between hosts. Implicit, established at pool creation.
Pool-wide private/internal networks — VM-level networks that span all hosts in the pool, typically backed by VLAN trunking on the physical NIC.
Cross-pool networking — manual, usually shared L2 (trunked VLAN at the switch) or external routing.

The XCP-ng model assumes shared L2 between hosts (trunk VLANs at the top-of-rack switch), which simplifies VM mobility and broadcast domains. Incus doesn’t make that assumption — it offers several composition primitives (bridges, VXLAN, OVN, macvlan) and lets you decide.

The implication for migration: XCP-ng admins arriving at Incus may expect “internal networks just work across hosts” by default. They don’t. We have to pick and configure one of the methods below.

Methods

1. Native L3 routing via underlay

Have the campus router (or whatever network sits between the two host subnets) route each bridge’s CIDR to its respective host. Disable NAT on both bridges so source IPs are preserved.

kyojin announces (or has a static route): 10.164.202.0/24 → kyojin host IP
theia announces: 10.114.104.0/24 → theia host IP
Each Incus host adds a route for the other bridge’s subnet pointing at the other host

How: Static routes at the campus router, plus ip route add on each host (or routes in IncusOS network seed). Disable ipv4.nat on the bridges.

Pros: Zero extra software. No encapsulation overhead. Native L3 separation per host. Diagnosable with standard tools (traceroute, tcpdump on physical NICs). Fits a campus already running BGP/OSPF cleanly.

Cons: Requires network admin cooperation. Bridge subnets become campus-visible — needs IP-allocation policy alignment (you can’t use arbitrary RFC1918 ranges that conflict with existing campus). If routing changes, bridge connectivity breaks. No built-in encryption.

Production-readiness: High, if your campus or DC network already supports dynamic routing and your team has the operational maturity to manage routes alongside compute.

Scales: Cleanly. Adding a host = adding one more route announcement.

2. Incus bridge with VXLAN tunnel

Incus’s bridge network type supports tunnel.<name>.protocol config to extend a single bridge across hosts via VXLAN (or GRE). Both hosts run the same logical bridge with the same subnet; instances see flat L2 across hosts.

# Sketch (per-host, both pointing at each other's host IP):
config = {
  "ipv4.address"            = "10.123.50.1/24"   # SAME on both hosts
  "ipv4.nat"                = "true"
  "tunnel.cluster.protocol" = "vxlan"
  "tunnel.cluster.id"       = "42"
  "tunnel.cluster.local"    = <this-host-IP>
  "tunnel.cluster.remote"   = <other-host-IP>
}

Pros: Native to Incus, no extra packages on IncusOS. L2 mesh — VMs can keep their IP across warm-migration (since the L2 segment exists on both hosts). UDP encapsulation traverses any routed path.

Cons: Both hosts must use the same subnet, losing per-host CIDR distinction. Broadcast traffic (ARP, mDNS) crosses the inter-subnet path — fine for ~2–4 hosts, doesn’t scale to many because each host needs a tunnel to every other (full mesh). VXLAN UDP must be permitted between subnets — if campus blocks it, this fails the same way k0s overlay does (see k0s-cross-subnet). Not encrypted.

Production-readiness: Medium. Acceptable for small fleets with controlled networks; problematic at scale or with strict L2 hygiene.

Scales: Poorly past ~4 hosts (full-mesh tunnel count grows quadratically; broadcast amplification grows linearly). Past that, OVN or L3 routing wins.

3. WireGuard tunnel between hosts + L3 routes

Run WireGuard at the host level between every pair of hosts, then static routes through the WireGuard interface for the other host’s bridge subnet. Disable NAT on the bridges so source IPs are preserved.

kyojin: route 10.114.104.0/24 → wg0 (peer = theia’s WireGuard endpoint)
theia: route 10.164.202.0/24 → wg0 (peer = kyojin’s WireGuard endpoint)

Pros: L3 separation preserved (per-host CIDRs stay distinct). Encrypted host-to-host. Modern, well-understood, performant. Routing config lives in code (systemd-networkd .netdev + .network files via IncusOS network seed). No central control plane.

Cons: WireGuard userland (wg, wg-quick) isn’t part of stock IncusOS — kernel module is present, but configuration via wg requires either an IncusOS image including the package, or systemd-networkd’s WireGuard support (which IncusOS does use, but you’d be pushing config via the network seed). Test thoroughly. Full-mesh peering at scale (same N² problem as VXLAN). No automatic peer discovery.

Production-readiness: High for ≤10 hosts with explicit peer config. Requires WireGuard tooling on IncusOS — either custom IncusOS build or systemd-networkd-only config — verify before committing.

Scales: Good to ~10–20 hosts with manual peer config. Beyond that, hub-and-spoke or a mesh manager (Tailscale, Netbird, Headscale) makes sense.

4. Incus OVN

Incus has first-class OVN (Open Virtual Network) support. Create networks as --type=ovn and they become distributed virtual networks with geneve overlay between hosts plus virtual routers managed by OVN.

incus network create my-ovn --type=ovn
incus network create my-ovn-router --type=ovn ...

Pros: Designed for SDN at scale. Distributed L3 (each host has a local OVN controller; routing is distributed, not chokepointed). Network ACLs, floating IPs, multi-tenant networks, security groups all built in. Used in production by mid-to-large Incus deployments. The “right answer” if you’re thinking about isolation policy, multi-team ops, or growing past a handful of hosts.

Cons: Significant infrastructure burden. Requires:

OVN central databases (NB and SB) — clustered for HA (3-node Raft minimum)
OVN controllers running on every Incus host
Underlay supporting geneve UDP between all hosts (same firewall concern as VXLAN)
Operational expertise — OVN is its own world (ovn-nbctl, ovn-sbctl, debugging via ovs-vsctl on hosts)

OVN is not bundled with stock IncusOS. You’d need either a custom IncusOS image that includes the OVN packages, or to run OVN central infrastructure outside IncusOS (in a privileged VM or external bare metal). The latter is more flexible but adds ops surface.

Production-readiness: High (if you’re willing to invest in OVN ops). This is what mid-sized OpenStack and Kubernetes-with-OVN-Kubernetes deployments use.

Scales: Designed for hundreds of hosts and thousands of networks. The right answer if you’re going past 10+ Incus hosts.

5. Macvlan to host LAN (sidestep — no bridge mesh)

Don’t mesh the bridges. For instances that need cross-host connectivity, attach them to host macvlan instead of tfbr0. They get a campus LAN IP directly.

This is what fleet.tf already does for k0s VMs (device { type = nic, properties = { nictype = macvlan, parent = local.<host>_nic, hwaddr = ... } }).

Pros: Simplest possible cross-host networking. Zero extra config beyond per-instance NIC device. Each instance is a first-class campus host. Existing pattern in this repo for k0s.

Cons: Each instance consumes a campus LAN IP — IP allocation policy matters at scale. Instances are reachable from the entire campus by default — relies on host firewalls (nftables in cloud-init) for isolation. No NAT hiding the implementation detail. Doesn’t help if you specifically want a private network that’s invisible from campus.

Production-readiness: High for workloads that should be campus-visible (k0s nodes, services with proper authn/authz). Wrong choice for “private internal network for inter-service traffic” use cases.

Scales: Trivially. Adding a host adds zero networking complexity.

Comparison matrix

Method	Layer	Encrypted	Extra infra	Per-host CIDR distinction	Hosts before scale issues	Migration path
1. Native L3 routing	L3	No	Campus routing config	Yes	Many	Underlay change to add
2. Incus VXLAN bridge	L2	No	None	No (shared subnet)	~4	Replace with OVN
3. WireGuard + routes	L3	Yes	WireGuard config	Yes	~10–20	Layer onto OVN, or hub-and-spoke
4. Incus OVN	L3 (overlay)	Optional	OVN central + controllers	Logical, not physical	Hundreds	This is the destination
5. Macvlan to LAN	L2 (LAN)	No	None	N/A	Many	No mesh — workloads stay on LAN

Production considerations for this prototype

These are the questions a XCP-ng replacement decision should answer:

Which workloads need bridge meshing in the first place? k0s already uses macvlan and gets cross-host connectivity for free via the LAN. Bridge meshing is for non-k8s private networks — admin VMs, monitoring backends, internal databases, prototypes. If most workloads will end up on Kubernetes or LAN-attached VMs, the bridge-mesh demand may be low; option 5 + ad-hoc per-workload routes may suffice.
What’s the trust model between hosts and between subnets? If campus network is trusted end-to-end, encryption is optional (plain VXLAN or native routing fine). If hosts span untrusted segments (WAN, multi-DC, multi-tenant lab), WireGuard or OVN-with-encryption become required.
What’s the team’s operational maturity? OVN is powerful but has a real learning curve. WireGuard is approachable. Native L3 routing requires close cooperation with the network team. Choose what your team can operate at 3am during an incident.
What’s the expected fleet size in 12–24 months? Already at five (kyojin/theia/hyperion/atomsk/ibuypwr). If steady-state is “5–10”, VXLAN or WireGuard scale fine. If “20+”, OVN starts paying for itself.
What’s the IP allocation policy? Macvlan (option 5) needs LAN IPs. Native routing (option 1) needs campus-routable subnets. VXLAN (option 2) needs only one shared private subnet. Pick the option whose IP demand fits the policy.

Recommended progression

For the current standalone-host fleet (now):

Stay with what’s already working: k0s VMs use macvlan to LAN, test instances stay on per-host tfbr0 without mesh. Don’t add tunneling complexity until you hit a workload that genuinely needs private cross-host networking.

When the first such workload appears:

Add option 2 (Incus VXLAN bridge tunnel) as the path of least resistance. Native to Incus, no extra packages, takes ~30 minutes to configure across both hosts. This trades the per-host CIDR mnemonic for L2 mesh — acceptable for a prototype.

As the fleet grows past ~4 hosts, or when policy/security requirements harden:

Migrate from VXLAN bridge to option 4 (OVN). This is where Incus starts paying back the XCP-ng comparison — OVN is more capable than XCP-ng’s networking and gives you the multi-tenant, ACL-aware, distributed-routing properties production environments expect. Plan the OVN deployment as a project in itself: central DB cluster, IncusOS image rebuild to include ovn-controller/ovn-host, runbook for OVN ops.

If/when the campus network team is willing to partner:

Option 1 (native L3 routing) is the cleanest underlay. If you can negotiate routable subnets per host and have campus dynamic routing carry them, you avoid encapsulation entirely. Worth asking.

WireGuard (option 3) is the right answer if you find yourself spanning untrusted network segments (cross-DC, off-campus, mixed cloud). Don’t reach for it just for in-DC encryption — option 1 with VLAN segmentation usually suffices.

What this rules out

Building a cluster-wide Incus bridge by relying on host-LAN bridging isn’t an option in this repo’s de-clustered design. Cluster-wide pools/bridges require Incus clustering, which we deliberately moved away from (see incus-declustering). The methods above are for standalone Incus daemons with bridges meshed externally.
Using OVN without IncusOS modification. Stock IncusOS doesn’t include the OVN packages. Going with option 4 is a commitment to either custom IncusOS images or running OVN central elsewhere — both are real ops investments.

Open questions before deciding

Does the campus network team support routable per-host subnets? (gates option 1)
Can our IncusOS images include wireguard-tools or just rely on systemd-networkd? (gates option 3 feasibility)
Is there appetite for running OVN central services? (gates option 4)
What’s the IP allocation policy for instance traffic? (constrains options 1 and 5)

Answers to these turn this from “five options” into “one or two practical choices.”