Incus PCI passthrough

How to hand a PCI device on an IncusOS host to a VM running on it. Two scenarios validated in this stack:

GPU: NVIDIA RTX 2080 SUPER through to the logi VM on ibuypwr. Used the IncusOS kernel-config approach (blacklist_modules + pci.passthrough) to detach nouveau from the GPU before VM start. (The pattern was originally validated on the now-retired surtr VM in 2026-04 before the ibuypwr brick recovery; logi inherited the same passthrough setup post-recovery — see ibuypwr-brick.)
USB JBOD: ASMedia ASM2142 USB 3.1 controller through to the backup2 VM on kyojin. Used the dynamic-bind-at-VM-start approach (no host-side blacklist).

These are two different patterns. Picking the right one matters because the wrong one can either fail silently or break unrelated devices. See Choosing the pattern below.

All commands assume INCUS_CONF=/configs/incus (the local Incus client config directory).

Mental model

A PCI device on an IncusOS host can be in one of three driver states:

Bound to the host’s native driver — nouveau for an NVIDIA GPU, xhci_hcd for a USB controller, mlx5_core for a Mellanox NIC, etc. The host kernel manages it; the device is unavailable for passthrough.
Bound to vfio-pci — the Linux passthrough stub. The device is visible to userspace but no kernel driver claims it for normal use. Available for assignment to a VM.
Unbound entirely — no driver. Equivalent to (2) for passthrough purposes; vfio-pci binds it on demand.

Incus’ VM lifecycle handles state transitions for type: pci devices on a VM — when a VM with such a device starts, Incus dynamically unbinds the host driver and binds vfio-pci. When the VM stops, it unbinds vfio-pci and re-binds the host driver. For most cases, that’s all you need; no IncusOS-level config required.

You only need IncusOS-level config (pci.passthrough and/or blacklist_modules) when:

The host driver is shared with other devices you don’t want to disturb, AND Incus’ dynamic unbind doesn’t work for some reason. (For most cases the dynamic path works fine — see USB controller passthrough.)
You want vfio-pci pre-bound for some other reason (SR-IOV, stable identifiers, multi-VM, GPU mediation).

Choosing the pattern

Your host driver claims…	Use this pattern
Only the device you want to pass through (e.g. `nouveau` for a GPU — there’s only one GPU)	Static binding via blacklist + passthrough config. Driver detaches at boot; device sits in vfio-pci or unbound; VM picks it up cleanly. See GPU passthrough.
Multiple devices, only one of which you want to pass through (e.g. `xhci_hcd` for both the JBOD controller AND the chassis keyboard’s USB controller)	Dynamic binding at VM start. Don’t blacklist — that would break the unrelated devices. Let Incus do the unbind/rebind dance per-VM. See USB controller passthrough.

The static pattern is what the upstream IncusOS docs describe (their example uses GPUs). The dynamic pattern is what you want when the host driver is generic and shared.

Common pre-checks

IOMMU enabled

IOMMU must be enabled on the host for PCI passthrough to work. Confirm via the boot log:

INCUS_CONF=/configs/incus incus query kyojin:/os/1.0/debug/log \
  | jq -r '.[] | select(.MESSAGE | test("Command line:")) | .MESSAGE' | head -2

You’re looking for intel_iommu=on (Intel) or amd_iommu=on (AMD). IncusOS enables intel_iommu=on by default. iommu=pt is also recommended for passthrough performance but isn’t strictly required.

Identify the device

GPUs:

INCUS_CONF=/configs/incus incus query kyojin:/1.0/resources | jq '.gpu'

PCI USB controllers (or any other PCI device):

INCUS_CONF=/configs/incus incus query kyojin:/1.0/resources \
  | jq '.pci.devices[] | select(.product | test("USB"; "i"))'

Note vendor_id, product_id, and pci_address — needed for any of the configs below.

Check the IOMMU group

A PCI device can only be passed through cleanly if its IOMMU group contains nothing else (or only related downstream devices). Multi-device groups force you to pass through the whole group at once.

The resources API includes iommu_group per PCI device. For the ASMedia card on kyojin (group 1, sole device), this was a clean single-device group. For shared groups, you may need ACS override patches at the kernel level — IncusOS doesn’t ship with these enabled.

GPU passthrough

The static-binding pattern, validated on ibuypwr’s RTX 2080 SUPER (originally → surtr, currently → logi post-2026-05 recovery). Works because nouveau only drives NVIDIA GPUs — blacklisting it has no impact on other hardware.

1. Identify the GPU

INCUS_CONF=/configs/incus incus query kyojin:/1.0/resources | jq '.gpu'

Example output:

{
  "cards": [
    {
      "driver": "nouveau",
      "driver_version": "6.18.14-zabbly+",
      "pci_address": "0000:01:00.0",
      "product": "TU104 [GeForce RTX 2080 SUPER]",
      "product_id": "1e81",
      "vendor": "NVIDIA Corporation",
      "vendor_id": "10de"
    }
  ],
  "total": 1
}

2. Blacklist the host driver and configure passthrough

echo '{
  "config": {
    "blacklist_modules": ["nouveau"],
    "pci": {
      "passthrough": [
        {
          "vendor_id": "10de",
          "product_id": "1e81",
          "pci_address": "0000:01:00.0"
        }
      ]
    }
  }
}' | INCUS_CONF=/configs/incus incus admin os system kernel edit

Verify:

INCUS_CONF=/configs/incus incus admin os system kernel show
INCUS_CONF=/configs/incus incus query kyojin:/1.0/resources \
  | jq '.gpu.cards[0] | {product, driver, pci_address}'

driver: null means the GPU is detached and ready. The blacklist_modules setting writes to /etc/modprobe.d/; for nouveau (a late-loading graphics driver), this is sufficient. The pci_address field on pci.passthrough triggers IncusOS to attempt a runtime unbind+rebind, which works for GPUs because nothing actively holds the device after the framebuffer console hands off.

3. Add the GPU to a VM (OpenTofu)

In the incus_instance resource:

device {
  name = "gpu0"
  type = "gpu"
  properties = {
    gputype = "physical"
    pci     = "0000:01:00.0"
  }
}

Also set security.secureboot = "false" in the instance config — required for PCI passthrough in Incus VMs.

See logi’s entry in fleet.yaml (rendered by fleet.tf) for the full VM definition.

USB controller passthrough

The dynamic-binding pattern, validated 2026-05 on kyojin’s ASMedia ASM2142 USB 3.1 controller → backup2 VM (4-drive JBOD with ZFS pool for Borg backups). Works because we don’t blacklist — xhci_hcd is the driver for all USB 3.x controllers, and blacklisting it would also break the chassis keyboard’s USB controller (which is a separate Intel C620 chipset device using the same module).

1. Identify the device

INCUS_CONF=/configs/incus incus query kyojin:/1.0/resources \
  | jq '.pci.devices[] | select(.vendor_id=="1b21")'

Example output:

{
  "pci_address": "0000:17:00.0",
  "vendor_id": "1b21",
  "product_id": "2142",
  "product": "ASM2142/ASM3142 USB 3.1 Host Controller",
  "driver": "xhci_hcd"
}

2. Verify there’s a separate USB controller you don’t want to lose

If kyojin had only one USB controller, the static-binding pattern would be fine. It doesn’t:

INCUS_CONF=/configs/incus incus query kyojin:/1.0/resources \
  | jq '.pci.devices[] | select(.driver=="xhci_hcd") | {pci_address, product}'

{ "pci_address": "0000:00:14.0", "product": "C620 Series Chipset Family USB 3.0 xHCI Controller" }
{ "pci_address": "0000:17:00.0", "product": "ASM2142/ASM3142 USB 3.1 Host Controller" }

The Intel C620 drives the chassis keyboard’s USB ports. We need xhci_hcd to keep claiming it. So the static blacklist pattern is out — we use dynamic binding instead.

3. Skip kernel config; just attach the device to a VM

No blacklist_modules needed. No pci.passthrough IncusOS-level config needed either (we tried both with and without; doesn’t change Incus’ behavior for our case). Incus’ VM-start lifecycle handles the unbind+rebind dynamically:

device {
  name = "jbod-usb"
  type = "pci"
  properties = {
    address = "0000:17:00.0"
  }
}

When the VM starts:

Incus unbinds xhci_hcd from the ASMedia controller (specifically that one — the Intel controller is unaffected).
Incus binds vfio-pci to it.
The VM boots with the ASMedia controller assigned and all 4 JBOD drives visible inside.

When the VM stops:

Incus unbinds vfio-pci.
Incus re-binds xhci_hcd.
Host can see the JBOD again (USB endpoints reappear in incus query .../resources).

Throughout, the Intel chipset USB controller stays bound to xhci_hcd and the keyboard works.

4. Required VM config flags

Same as GPU:

security.secureboot = "false" — required for any Incus VM PCI passthrough.

See backup2’s entry in fleet.yaml (rendered by fleet.tf) for the full VM definition.

5. Empirical verification of the swap

To confirm the swap actually happens on VM start/stop, watch the driver state:

# Pre-start: host owns the device
incus query kyojin:/1.0/resources | jq '.pci.devices[] | select(.vendor_id=="1b21") | {driver}'
# → {"driver": "xhci_hcd"}

incus start kyojin:backup2
sleep 5
incus query kyojin:/1.0/resources | jq '.pci.devices[] | select(.vendor_id=="1b21") | {driver}'
# → {"driver": "vfio-pci"}     ← VM has it

incus stop kyojin:backup2
sleep 5
incus query kyojin:/1.0/resources | jq '.pci.devices[] | select(.vendor_id=="1b21") | {driver}'
# → {"driver": "xhci_hcd"}     ← back to host

USB endpoints behind the controller (e.g. the JBOD’s TDAS hubs) follow the controller — visible to the host when bound to xhci_hcd, invisible when bound to vfio-pci (visible only inside the VM).

Kernel config schema reference

Config fields accepted by incus admin os system kernel edit (struct SystemKernelConfig):

Field	Type	Description
`blacklist_modules`	`[]string`	Kernel modules to blacklist. Written to `/etc/modprobe.d/` — only effective for modules that aren’t loaded from initramfs. Useful for late-loading drivers like `nouveau`; does not work for early-loading drivers like `xhci_hcd`.
`pci.passthrough[].vendor_id`	`string`	PCI vendor ID (hex, e.g. `10de`)
`pci.passthrough[].product_id`	`string`	PCI product ID (hex, e.g. `1e81`)
`pci.passthrough[].pci_address`	`string`	Optional; if set, IncusOS attempts a runtime unbind+rebind to vfio-pci. Opportunistic — silent failure if the host driver won’t release.
`memory.persistent_hugepages`	`int`	Number of persistent hugepages to allocate
`network.buffer_size`	`string`	Max buffer size for `tcp_rmem`/`tcp_wmem`/`rmem_max`/`wmem_max`
`network.queuing_discipline`	`string`	Value of `net.core.default_qdisc`
`network.tcp_congestion_algorithm`	`string`	TCP congestion algorithm (default: `bbr`)

Note: The IncusOS API is marked as subject to change. A system reboot may be required for some changes to fully take effect.

Gotchas

`blacklist_modules` doesn’t reach the kernel cmdline

IncusOS writes blacklist_modules to /etc/modprobe.d/, not to the kernel cmdline. For drivers loaded by initramfs (USB, NVMe, common storage), modprobe.d entries are too late — initramfs has already loaded the driver. The block fails silently.

You can verify by inspecting the actual kernel cmdline post-boot:

INCUS_CONF=/configs/incus incus query kyojin:/os/1.0/debug/log \
  | jq -r '.[] | select(.MESSAGE | test("Command line:")) | .MESSAGE' | head -2

IncusOS’s own modprobe.blacklist=nvidiafb is on the cmdline (baked in at image build), proving the cmdline-blacklist mechanism exists — IncusOS just doesn’t propagate user-supplied blacklist_modules to it as of 2026-05.

For drivers that load late enough (graphics drivers like nouveau/nvidia), this works fine. For early-loading drivers, you need the dynamic Incus pattern instead.

Runtime unbind via `pci.passthrough` can silently fail

pci.passthrough[].pci_address triggers an opportunistic unbind+rebind at config-edit time. The IncusOS code explicitly ignores errors from these operations — the file gets written to modprobe.d regardless, expecting the rebind to take effect at next boot.

Common failure modes:

vfio-pci not loaded yet at the time of the unbind+new_id write. The new_id write goes to a sysfs path that doesn’t exist; it fails silently.
Host driver won’t release because of active children (e.g. USB controller with USB devices currently bound behind it).
Driver autoprobe re-binds after unbind if no other driver claims first.

Diagnostic: check whether the driver actually flipped:

INCUS_CONF=/configs/incus incus query kyojin:/1.0/resources \
  | jq '.pci.devices[] | select(.vendor_id=="<vid>") | {pci_address, driver}'

If driver is still the host driver and not null/vfio-pci, the unbind failed. For the GPU case this normally works; for USB controllers we observed it consistently failing — leaving the dynamic-Incus pattern as the working alternative.

Incus’ dynamic unbind is reliable; the IncusOS pre-binding is the fragile path

For most cases, just attach the PCI device to a VM and start it. Incus does its own unbind+vfio-bind at VM-start time, and re-binds the host driver at VM-stop. This works regardless of whether IncusOS’s pci.passthrough config managed to pre-bind anything.

We spent significant effort fighting the IncusOS pre-binding for the JBOD case — chasing kernel cmdlines, rebooting, trying blacklist combinations — before discovering Incus’ dynamic path Just Worked. The pre-binding is mostly useful for cases where Incus’ own attempt would fail (e.g. devices needed at boot before any VM exists, or SR-IOV virtual functions).

XCP-ng / Xen does this differently

For reference: XCP-ng uses xen-pciback with kernel cmdline xen-pciback.hide=(0000:17:00.0). The pciback driver loads in initramfs and “hides” specific PCI addresses from the host before any other driver can bind. This is the surgical-per-device approach Linux/KVM also supports via vfio-pci.ids= on the cmdline.

IncusOS doesn’t expose either cmdline knob. The dynamic Incus pattern is the IncusOS-native equivalent — same end result (one device handed to one VM, others untouched), different mechanism.

CLI `-d` flag won’t add new devices

incus init/launch -d <name>,<key>=<value> overrides properties of an existing device (from a profile), not creates new ones. Trying to add a fresh pci device this way fails with Cannot override config for device "X": Device not found in profile devices.

Use the YAML-via-stdin form instead:

cat <<EOF | incus init kyojin:<image> kyojin:<name> --vm
config:
  security.secureboot: "false"
devices:
  jbod-usb:
    type: pci
    address: "0000:17:00.0"
EOF

Or incus config device add <instance> <name> pci address=<addr> after init.

Cross-remote image init not supported

incus init kyojin:test images:debian/12 --vm fails with Error: The remote isn't a private server because images: is a public remote. Copy the image first:

incus image copy images:debian/trixie/cloud kyojin: --vm --alias debian-trixie-cloud
incus init kyojin:debian-trixie-cloud kyojin:backup2 --vm < config.yaml

Sources

IncusOS kernel reference
incus-osd/api/system_kernel.go — SystemKernelConfig struct
incus-osd/internal/kernel/kernel.go — ApplyKernelConfiguration (modprobe.d writes + opportunistic runtime rebind)
incus-osd/internal/rest/api_debug.go — debug log endpoint (read-only; no host shell)
Incus PCI device documentation