Incus OVN integration

Parked — gated on re-clustering (migrated 2026-09-05)

Nothing here is deployed and the gate is unchanged: cross-host OVN requires an Incus cluster by upstream design, and the fleet remains standalone daemons. Until the de-clustering decision is revisited, the maintainer-blessed path for cross-host private networking on this shape is VXLAN multicast — option 2 in the bridge-mesh note.

Status: the IncusOS side is feasible — ovn-controller and OVS are bundled in the base image and exposed via the incus admin os system ovn service config. The real blocker is standalone Incus daemons: per the maintainer, “Incus expects to own the OVN deployment and be in control of all virtual switches,” which means cross-host OVN networks require an Incus cluster, not the standalone-daemon model we landed on after de-clustering. So OVN as a cross-host SDN substrate is gated on reversing incus-declustering.md, not on fighting IncusOS.

For the broader bridge-meshing decision (where OVN sits as one option among several), see incus-bridge-mesh.md. This doc is the OVN-specific deep dive.

What this gets us

The current network model is bimodal:

• macvlan for instances that need a LAN identity (k0s nodes, backup2, eren/mikasa/armin, logi). Each VM is a first-class campus host with its own MAC + DHCP-reserved or cloud-init-assigned IP.
• per-host bridged (tfbr0 in system_network.tf) for scratch/test instances. NAT’d through the host, host-local — instances on different hosts’ bridges can’t reach each other directly.

OVN unifies these into a third pattern: logical L2/L3 networks that span hosts without anyone of them needing a campus LAN identity. Cross-host workflows on private networks become first-class instead of a bridge-mesh workaround. ACLs become declarative. Multiple isolated tenant networks become a config item, not an architectural commitment.

This is the same SDN substrate that backs OVN-Kubernetes, OpenStack Neutron’s modern default, and Kubevirt clusters — well-trodden patterns, just outside this repo’s current operational envelope.

How Incus integrates with OVN

                                 ┌────────────────────────────────────┐
                                 │  ovn-central (NOT bundled in       │
                                 │  IncusOS — runs separately)        │
                                 │  - northbound DB (intent)          │
                                 │  - southbound DB (state)           │
                                 │  - ovn-northd (NB→SB compiler)     │
                                 │  3-replica HA recommended;          │
                                 │  OCI containers on isolated subnet │
                                 └─────────────┬──────────────────────┘
                                               │ ovsdb protocol (TLS)
                  ┌────────────────────────────┼────────────────────────────┐
                  │                            │                            │
       ┌──────────┴──────────┐      ┌──────────┴──────────┐      ┌──────────┴──────────┐
       │  kyojin (IncusOS)   │      │  theia (IncusOS)    │      │  hyperion (IncusOS) │
       │  ┌──────────────┐   │      │  ┌──────────────┐   │      │  ┌──────────────┐   │
       │  │ ovn-controller│  │      │  │ ovn-controller│  │      │  │ ovn-controller│  │
       │  │ (BUNDLED)     │  │      │  │ (BUNDLED)     │  │      │  │ (BUNDLED)     │  │
       │  └──────┬───────┘   │      │  └──────┬───────┘   │      │  └──────┬───────┘   │
       │  ┌──────┴───────┐   │      │  ┌──────┴───────┐   │      │  ┌──────┴───────┐   │
       │  │ Open vSwitch │   │      │  │ Open vSwitch │   │      │  │ Open vSwitch │   │
       │  │ (BUNDLED)    │   │      │  │ (BUNDLED)    │   │      │  │ (BUNDLED)    │   │
       │  └──────┬───────┘   │      │  └──────┬───────┘   │      │  └──────┬───────┘   │
       │   incus daemon      │      │   incus daemon      │      │   incus daemon      │
       │   (today: standalone│      │   (today: standalone│      │   (today: standalone│
       │    — see blocker)   │      │    — see blocker)   │      │    — see blocker)   │
       └─────────┬───────────┘      └─────────┬───────────┘      └─────────┬───────────┘
                 │                            │                            │
                 └─────── Geneve overlay (UDP/6081) ──────────────────────┘

Per-host pieces (all already in IncusOS):

• Open vSwitch (ovs-vswitchd + ovsdb-server) — userspace switching daemon plus its local config DB. Manages OVS bridges that act as the local fabric for OVN logical switches.
• ovn-controller — agent that connects to OVN central’s southbound DB, translates logical flows into OpenFlow rules pushed into the local OVS instance.

Per-cluster pieces (NOT in IncusOS — must be deployed externally):

• OVN northbound DB (NB) — declarative intent. Logical switches, routers, ACLs, ports.
• OVN southbound DB (SB) — runtime state. Physical hosts (chassis), bindings of logical ports to chassis, MAC learning tables.
• ovn-northd — daemon that compiles NB → SB.
• HA: 3 replicas in a Raft quorum.

Incus’s role:

• Translates incus network create --type=ovn ... into NB DB writes.
• When an instance with nictype=ovn starts, Incus tells OVN central to bind that logical port to this host’s chassis. ovn-controller on the host then plumbs the OVS port into the VM’s tap/veth.

The VM-facing config replaces the inline macvlan or bridged-profile NIC:

device {
  name = "eth0"
  type = "nic"
  properties = {
    nictype = "ovn"
    network = "ovn-private-1"   # name of the incus_network of type ovn
    hwaddr  = "5e:cd:5a:a4:9d:47"
  }
}

The IncusOS picture (mostly solved)

IncusOS distinguishes two configuration surfaces:

Concept	Examples	Configured via
Applications	incus, migration-manager, operations-center, shared-api, plus non-primary debug, gpu, incus-ceph, incus-linstor	applications.yaml seed
Services	OVN, plus storage / kernel / network we already use	incus admin os system <service> API

OVN is a service, not an application — same surface we use for storage and kernel passthrough. The applications.yaml seed isn’t where it would live.

Configuring the service

Per the IncusOS reference, the OVN service exposes:

Setting	Purpose
enabled	Whether to start ovn-controller + OVS on this host
database	OVN southbound DB connection string (where ovn-central lives)
tls_client_certificate / tls_client_key / tls_ca_certificate	Mutual TLS to ovn-central
tunnel_address	This chassis’s underlay IP for Geneve tunnels
tunnel_protocol	Tunnel encapsulation (geneve / vxlan / stt)
ic_chassis	Whether this chassis is a gateway for OVN interconnection

The shape would be (modeled on the existing incus admin os system storage edit we use):

INCUS_CONF=/configs/incus incus admin os system ovn edit kyojin:
# editor opens; YAML similar to:
config:
    enabled: true
    database: ssl:10.123.50.10:6642,ssl:10.123.50.11:6642,ssl:10.123.50.12:6642
    tls_ca_certificate: |
      -----BEGIN CERTIFICATE-----
      ...
    tls_client_certificate: |
      -----BEGIN CERTIFICATE-----
      ...
    tls_client_key: |
      -----BEGIN PRIVATE KEY-----
      ...
    tunnel_address: 10.112.113.202
    tunnel_protocol: geneve

(The exact YAML shape will be confirmed by incus admin os system ovn show kyojin: once we attempt this — IncusOS docs list the keys but don’t give a verbatim example.)

Recent integration history (worth tracking)

The OVN service in IncusOS is fresh and accumulating fixes:

• Dec 2025: bug where the service Start() only launched ovs-vswitchd.service without ovsdb-server.service first, leaving the OVS unix socket missing. Subsequent OVN config attempts then failed with database connection failed (No such file or directory). Fixed.
• Jan 2026: an external_ids:ovn-remote= empty-value error reported on the same forum thread; resolved once the underlying Start() ordering was corrected.
• Sep 2025 (Incus 6.17): OVN networks gained Geneve/VXLAN/STT tunnel support previously available only on bridged networks.

Not yet “deploy and forget.” Worth a sanity test on a throwaway environment before committing.

The standalone-daemon blocker (decisive)

This is the actual gate, not the IncusOS packaging question.

Per Stéphane Graber (Incus maintainer) on the IncusOS forum: “Incus expects to own the OVN deployment and be in control of all virtual switches.” In practice this means an Incus daemon assumes its OVN deployment is exclusive to that daemon — a standalone daemon can’t share OVN networks with other standalone daemons. The chassis-binding logic in Incus’s network code looks up port placements through cluster member identity, which standalone daemons don’t share.

Concretely, that gives three states:

Setup	OVN scope	Works?
Standalone Incus, single host	Single-host OVN networks (no cross-host span)	Yes — but single-host private networks don’t add value beyond tfbr0
Standalone Incus daemons, multiple hosts, sharing one ovn-central	Cross-host OVN networks	No — explicitly unsupported by Incus
Incus cluster, multiple members, sharing one ovn-central	Cross-host OVN networks	Yes — the documented and supported path

We de-clustered in May 2026 (incus-declustering.md) for specific operational reasons: simpler failure modes, no cluster-database recovery scenarios, warm migration via incus copy --refresh covering the workflows we needed. Adopting OVN means revisiting that decision.

Graber’s recommended alternative for standalone setups

Same forum thread: for inter-host private networking on standalone Incus, the explicit suggestion is VXLAN multicast tunnel (or VLAN/GRE if the underlay supports it) — the option-2 path in incus-bridge-mesh.md. This is the maintainer’s blessed path for our shape; OVN is offered for clusters.

OVN central placement (if we re-cluster)

ovn-central is not in IncusOS. It runs separately. Patterns that fit our stack:

Option	Where	Pros	Cons
A. Three Debian VMs, one per IncusOS host	New VMs (added to opentofu)	HA, isolates ovn-central from IncusOS lifecycle, fits cross-host fault tolerance	Three more VMs to operate, monitor, back up. Bootstrap order: ovn-central must be up before OVN-typed Incus networks can come up.
B. Three OCI containers via system-container profile	Same hosts, lighter-weight than VMs	Smaller footprint than option A. Recommended pattern from the IncusOS forum thread.	Still three things to operate; OCI image source/update story to design.
C. Single-node ovn-central on one Debian VM	A new VM somewhere (e.g. on kyojin_instances)	Simple. Tolerated by ovn-central for dev/test.	Not HA. Loss of ovn-central = no new OVN ports come up (existing flows keep working until something changes). Not production-grade.
D. ovn-central on k0s	Three pods	Container-native; familiar tooling	Bootstrap inversion: k0s networking depends on OVN, but OVN runs on k0s. Probably untenable.

Forum guidance is to put ovn-central on an isolated network from the management plane (meshbr0-style, macvlan, or direct attach) so OVN tunnel control traffic doesn’t fight management traffic.

Option A or B is the textbook answer past prototype stage. Option C is a defensible “see if it works” target during evaluation.

Cross-site Geneve reachability

OVN uses Geneve (UDP/6081) for inter-chassis tunnels. RH461 hosts (kyojin, atomsk) sit on 10.112.113.128/25; AQ114 (theia, hyperion) on 10.112.12.64/26. Cross-site tunnels traverse the campus router.

Coordination needed: confirm UDP/6081 isn’t filtered between RH461 and AQ114. Same conversation we’d have for VXLAN; reusable.

MTU consideration: Geneve adds ~58 bytes of header. Logical switch MTU on OVN networks should be 1422 to fit inside the campus 1500-byte path. Default OVN MTU calculation usually does this automatically based on the underlay MTU, but verify after bring-up with ip link show inside an instance.

Operational surface

Two layers of tooling, both new:

# IncusOS service config (analogous to existing storage/kernel commands)
INCUS_CONF=/configs/incus incus admin os system ovn show <remote>:
INCUS_CONF=/configs/incus incus admin os system ovn edit <remote>:

# OVN tooling (runs against ovn-central; install on operator client)
ovn-nbctl show                       # logical topology (switches, routers)
ovn-nbctl ls-list                    # logical switches
ovn-nbctl lr-list                    # logical routers
ovn-nbctl acl-list <switch>          # network ACLs

ovn-sbctl show                       # runtime state (chassis, port bindings)
ovn-sbctl list chassis
ovn-sbctl list port_binding

# Per-host OVS state (would need to go through Incus admin API on IncusOS,
# since IncusOS is API-only — no shell on the host)
incus admin os system ovs show <remote>:    # if such a command exists; verify

The Incus side (incus network create --type=ovn ..., incus network show <name>, incus_network TF resource) hides most of this for the common case, but debugging packet drops, ACL hits, or chassis-binding failures means dropping into the OVN tooling.

Operator skill investment: non-trivial. Plan for multi-day OVN training before treating this as on-call-able infrastructure.

Migration shape (only relevant if re-clustering)

Assuming the re-clustering decision is made and prerequisites land, a phased migration:

1. Phase 0 — Re-cluster Incus across kyojin / theia / hyperion. ovn-central up (option A, B, or C above). One Incus host wired (one chassis registered in SB DB).
2. Phase 1 — Replace test/scratch instances first. test1/test2/test3 move from bridged-profile to a new ovn-test network. Validates the Incus↔OVN integration with disposable workloads.
3. Phase 2 — Optional: introduce a private OVN network for non-test workloads (admin VMs, internal monitoring backends, prototype services). k0s nodes stay on macvlan (they need LAN identity, not OVN logical network membership).
4. Phase 3 — Decommission tfbr0 if no instances remain on it.

logi (GPU VM) and backup2 (USB JBOD VM) stay on macvlan permanently — both need a real LAN presence and OVN doesn’t help them.

Tradeoffs vs the current model

Pros

• Cross-host private networking without a per-host bridge mesh.
• Network ACLs as first-class declarative resources, better than scattered nftables rules.
• Multi-tenant networks become a configuration item — useful if RRCHNM ever wants per-project network isolation.
• Aligns with where the SDN ecosystem is heading.

Cons

• Requires reversing the de-clustering decision — biggest cost.
• Adds ovn-central as load-bearing infrastructure (3 more VMs/containers; back up the NB DB like you’d back up etcd).
• Geneve overhead is small but real (single-digit % per-packet at minimum) vs raw bridge or macvlan.
• Debugging stack: OVN logical → OpenFlow → datapath → kernel netfilter. Steep operator learning curve.
• Cross-site Geneve depends on campus router cooperation (UDP/6081).
• Integration is fresh in IncusOS (recent fixes); not yet a mature deploy-and-forget combination.

What “next steps” look like, if pursuing

1. Decide on re-clustering Incus. This is the actual gate. Read incus-declustering.md, enumerate which de-clustering benefits we’d lose, decide whether OVN’s cross-host story is compelling enough. Until this answer is “yes,” everything below is moot — and Graber’s VXLAN-multicast recommendation is the right path for inter-host private networks on standalone setups.
2. (If yes:) Spin up a single-node ovn-central on a throwaway Debian VM (option C) for evaluation. Cheap, gives the team hands-on familiarity with ovn-nbctl/ovn-sbctl.
3. Configure the OVN service on one IncusOS host via incus admin os system ovn edit kyojin:. Point database at the eval ovn-central; provide TLS material; pick tunnel_address and tunnel_protocol.
4. Re-cluster Incus across the hosts that will share the network (kyojin + theia minimum; ibuypwr/hyperion/atomsk if/when they should participate) — separate decision, separate runbook.
5. Try incus network create --type=ovn from the cluster. Validates end-to-end. Document what works, what breaks, what error messages mean.
6. Pilot the prototype with test1/test2/test3 on the OVN network. Confirms cross-host reachability without going through campus LAN.
7. Productionize ovn-central: scale to 3 replicas (option A or B), put on isolated network, set up TLS/cert rotation, NB-DB backups.

If step 1 returns “no” or “not yet,” document the decision (in this file or in incus-declustering.md) and move to VXLAN multicast as the cross-host private-network substrate.

Related docs

• incus-declustering.md — context for why we’re standalone today. OVN’s blocker lives here. If we ever revisit OVN seriously, the de-clustering decision is the first thing to revisit.
• incus-bridge-mesh.md — full menu of cross-host network options. OVN is option 4 there; this doc drills into option 4 specifically. Graber’s blessed alternative for standalone setups (option 2, VXLAN multicast) is also there.
• incus-warm-migration.md — how we currently handle the cross-host workflows OVN would otherwise provide.
• k0s-cross-subnet.md — the analogous discussion for k0s pod networking.
• infra-firewall.md — the current host-firewall posture OVN ACLs would partially supersede.

External references

• OVN — IncusOS reference docs — service config surface (enabled, database, TLS, tunnel_address, tunnel_protocol, ic_chassis).
• IncusOS applications reference — confirms OVN is not in the applications catalog (it’s a service).
• How to set up OVN with Incus — Incus docs — cluster-side setup walkthrough.
• Tutorial for OVN on IncusOS? — forum thread — Graber’s “Incus expects to own the OVN deployment” quote and his recommendation of VXLAN multicast for standalone setups.
• IncusOS OVN Issues — forum thread — Dec 2025 / Jan 2026 bug history, ovn-central deployment guidance (3 OCI containers on isolated network).
• GitHub: lxc/incus-os — file issues / RFEs against IncusOS here.