Skip to content
RRCHNM Infra

Incus-native OCI containers

Reference for running upstream Docker / OCI images as Incus instances (rather than as system containers via images: or as Docker workloads on docker_servers). Validated on IncusOS 202605181246 + Incus 7.0.0 against docker:nginx:alpine and docker:alpine on kyojin (Broadcom tg3) and hyperion (Intel).

Background: the kresd-via-OCI probe in internal-dns “Alternatives evaluated” §“OCI containers (cznic/knot-resolver:latest6)” (#5 in the alternatives list) listed three OCI blockers that, between them, looked like “Incus-native OCI is unworkable on this fleet.” Re-probing in 2026-05-25 found that one of them (kyojin IPVLAN) is sidesteppable, one (safe_mount /run lxc.log error) is benign log noise, and the remaining two are real operational gotchas — but plannable around.

TL;DR

  • Incus-native OCI works on this IncusOS build. Vanilla launch + macvlan ingress + nginx entrypoint → status: Running, 26 processes, host_name=eno1. Confirmed on both Broadcom-tg3 and Intel-NIC hosts.
  • Use nictype=macvlan parent=eno1, not the daemon-default IPVLAN. IPVLAN fails on kyojin’s Broadcom tg3 with kernel EBUSY from netdev_configure_server_ipvlan — that part of the kresd note is correct. macvlan on the same NIC works fine for both VMs (already proven by jean/connie/sasha) and OCI containers.
  • ERROR utils - safe_mount:1332 - Invalid argument - Failed to mount "none" onto ".../rootfs/run" in lxc.log is benign. It’s standard noise from LXC’s tmpfs-on-/run setup. Containers run fine despite it. Don’t chase it. Confirmed by the linuxcontainers forum Postgres-as-OCI thread and reproduced on images:alpine/edge (system, not OCI) containers too on the same hosts.
  • Three real OCI gotchas to plan around: entrypoint must be long-running (image CMD respected; /bin/sh exits without a tty), /run tmpfs wipes on every start (image entrypoint must mkdir -p required subdirs like /run/php, /run/nginx), and rootfs persistence requires explicit storage-volume mounts.

The IPVLAN-vs-macvlan distinction

Incus’s OCI launch path defaults to attaching the instance’s primary NIC as nictype=ipvlan parent=<profile-default-parent> unless overridden. On kyojin’s Broadcom NetXtreme 5720 (tg3 driver), IPVLAN child creation kernel-rejects:

netdev_configure_server_ipvlan: Device or resource busy

This is a driver-level issue with tg3’s handling of the DHCP-bound parent NIC, not anything Incus does. Intel NICs (igc/ixgbe on theia/hyperion/ibuypwr) are unaffected. The note in internal-dns.md “Alternatives evaluated” correctly identified this as a kyojin blocker for the kresd probe; what it missed is that macvlan attached to the same parent works fine — confirmed by both:

  1. The whole existing VM fleet (jean/connie/sasha/eren/mikasa/armin/lugia/…) which already uses nictype: macvlan parent: eno1 per incus config show atomsk:jean (and the macvlan NIC pattern in fleet.tf, driven by fleet.yaml).
  2. A throwaway OCI probe with explicit incus config device override <name> eth0 nictype=macvlan parent=eno1 network= (the network= is required — it shadows the profile’s network: incusbr0 key, which would otherwise conflict with nictype=macvlan).

The kernel-level difference: IPVLAN child interfaces share the parent’s MAC and L3-multiplex; macvlan child interfaces get their own MAC and L2-broadcast on the parent’s segment. tg3 supports the latter, not the former.

The safe_mount /run log noise

incus info --show-log <instance> for any OCI container on this build shows:

lxc <name> ... ERROR utils - ../src/lxc/utils.c:safe_mount:1332
  - Invalid argument
  - Failed to mount "none" onto "/opt/incus/lib/lxc/rootfs/run"

This appears regardless of whether the container is actually running. The probe saw it on:

  • A nginx OCI container that was confirmed running via API (status: Running, pid 604599, 26 processes)
  • A failed alpine OCI container (entrypoint /bin/sh exited)
  • Containers on every storage pool tried (local, kyojin_instances)
  • Both security.privileged=true and unprivileged
  • Both Broadcom and Intel hosts

It’s a non-fatal warning emitted during LXC’s pre-pivot rootfs setup. The thread on Postgres as OCI does not start shows the same line on a container that briefly entered RUNNING. When debugging an OCI launch that’s not staying up, skip this line and look at the entrypoint / process exit signal instead — incus query <remote>:/1.0/instances/<name>/state shows the real pid and process count.

The real gotchas

Default entrypoint is the image’s CMD; shell-image CMDs exit on no-tty

OCI launches respect the image’s CMD. For service images (nginx, postgres, your custom Docker images) this is what you want — they have long-running entrypoints by design. For base images (docker:alpine, docker:debian, docker:busybox) the CMD is /bin/sh or /bin/bash, which exits immediately when stdin isn’t a tty, dropping the container straight to STOPPED.

Override via the oci.entrypoint config key. JSON-list form:

incus config set <remote>:<name> oci.entrypoint='["/bin/sleep","infinity"]'

oci.cwd, oci.uid, oci.gid are the other supported override keys. There is no oci.cmd (the kresd probe note’s third bullet); the entrypoint key takes the full argv as a JSON array.

/run is a fresh tmpfs every start — subdirs must be recreated

LXC mounts /run as tmpfs per Linux convention. Anything baked into the image at /run/<subdir> (php-fpm sockets, nginx pidfile dirs, supervisord state, postgres pidfiles) does not persist across container starts. The image’s entrypoint script must mkdir -p what it needs before launching the daemon.

This is documented upstream: Stéphane Graber’s recommendation in the OCI containers issues thread is to add mkdir -p /run/nginx /run/php (or equivalent) to the container image’s entrypoint. Many official Docker images already do this; some don’t. If a custom image fails to start with “cannot bind to /run/X” or similar, this is the cause — fix the image’s entrypoint, not Incus.

Rootfs wipes on every stop+start

The kresd probe’s second bullet: “OCI container rootfs wiped on every incus stop+incus start.” Still true. Persistent state requires explicit Incus storage volumes mounted at the relevant image paths. Pattern from the kresd probe (which never shipped but had this part working):

incus storage volume create <pool> <vol-name>
incus config device add <name> <dev-name> disk \
  pool=<pool> source=<vol-name> path=/var/lib/<app>

Same pattern Docker users know as named volumes — different ergonomics. Plan storage volumes the same way you’d plan named docker volumes for the same image.

Campus IP assignment is the same as for VMs

OCI containers attached to macvlan-on-eno1 get an L2 presence on the host’s LAN segment, same as any VM. IP assignment follows the same rules:

  • RH461 (kyojin + atomsk VMs at 10.112.113.128/25): DHCP-reservation-only — random MACs get no IPv4. Coordinate a reservation with network admin and set the reserved MAC explicitly via incus config device set <name> eth0 hwaddr=<reserved>. See ansible/inventory.yaml for the existing reservation pools (host192-199 and host243-249 blocks) and ansible/README.md “Per-site MAC + IP conventions” for the format.
  • AQ114 (theia/hyperion VMs at 10.112.12.64/26): Static-only — no DHCP at all. Set ipv4.address=<addr> on the device, configure the network inside the container at startup (the IncusOS cloud-init.network-config path used by VMs doesn’t apply to OCI containers — the image’s entrypoint has to configure the interface, or the image needs to be one that accepts an IP= env var, etc.).

This is not OCI-specific — it’s the same fleet-wide network-conventions story.

Launching one end-to-end

For a custom Docker image, the full recipe on kyojin (replace <your-image> and <reserved-mac> accordingly):

Terminal window
# One-time: add the docker.io OCI remote to the workspace client.
incus remote add docker https://docker.io --protocol=oci


# 1. Init from upstream OCI image (or from your private registry — point at thoth).
incus init docker:<your-image>:<tag> kyojin:<name>


# 2. Override the profile's eth0 to macvlan on eno1 with a reserved MAC.
incus config device override kyojin:<name> eth0 \
  nictype=macvlan parent=eno1 network= hwaddr=<reserved-mac>


# 3. (Optional) Mount a persistent volume for app state.
incus storage volume create kyojin:<pool> <name>-data
incus config device add kyojin:<name> data disk \
  pool=<pool> source=<name>-data path=/var/lib/<app>


# 4. (Optional) Override the entrypoint or env if the image needs it.
incus config set kyojin:<name> oci.entrypoint='["/path/to/binary","arg1","arg2"]'
incus config set kyojin:<name> environment.FOO=bar


# 5. Start. Verify via API (incus list may render OCI containers oddly).
incus start kyojin:<name>
incus query kyojin:/1.0/instances/<name>/state | jq '{status,pid,processes,network:.network.eth0}'

The incus list CLI sometimes renders application containers as empty rows when they’re up but have no IPv4 yet — don’t trust it as a liveness check, use the state API.

When to choose OCI vs. the alternatives

This fleet has three ways to run a Docker image’s workload. Pick deliberately:

PathWhen it fitsWhen it doesn’t
Incus-native OCI (this doc)One-off services that fit cleanly behind a Caddy or Gateway; you want Incus ops semantics (incus exec, snapshots, migration — see incus-warm-migration) and don’t need Compose-style multi-container stacks.The image needs sidecars (logging, init, etc.) you’d express as a Compose file or k8s Deployment. The image expects systemd. The image is part of a larger orchestrated bundle.
Docker on docker_servers (eren/armin rootless, mikasa rootful)Compose stacks, anything with sidecars, anything where the upstream’s deploy story is docker compose up.You want Incus snapshots / migration on the workload, or the image is a single-process service that doesn’t benefit from Docker daemon overhead.
k0s workloadOrchestrated, scaled, ingress via Cilium Gateway, secrets via ESO/OpenBao.One-off, low-traffic, or doesn’t need the k8s machinery. The make k0s-down blast radius matters.

The Docker-in-Incus pattern from internal-dns “Alternatives evaluated” (Trixie container + docker run --network host) is a fourth path. Useful when you want Incus ops shell around a Docker workload but don’t want the workload spread across docker_servers; cost is nested containerization. Pick this if the OCI gotchas above are deal-breakers for a specific image (e.g. the image fundamentally can’t tolerate /run tmpfs wipe and you don’t want to fix the upstream image).

References

Last updated: 2026-05-25