Molecule testing for Ansible

This document plans a test suite for the ansible/ tree using Molecule with the delegated driver, where test instances are provisioned on our existing Incus hosts. The goal is a CI-runnable suite that catches regressions in roles and playbooks before they hit kyojin / theia, without standing up a parallel test backend (Vagrant, EC2, etc.) — Incus is already the substrate this repo manages, and reusing it keeps the test environment close to production.

This is a planning document, not a build log. Decisions called out below should be revisited once the first scenario lands.

Scope

In scope:

ansible/roles/* — currently nftables, expected to grow.
ansible/playbooks/incus/tasks/* — small task files imported by the host setup playbooks, easy unit-of-test targets.
Idempotency, syntax, lint coverage across the whole tree.

Out of scope (initially):

playbooks/k0s/setup-cluster.yaml — provisions a real Kubernetes cluster across multiple VMs. Higher-level integration test, not a Molecule scenario; tracked separately.
playbooks/incus/setup-{kyojin,theia,hyperion}.yaml as end-to-end runs — these expect bare-metal IncusOS hosts and configure storage pools, profiles, etc. Test the imported task files individually instead.
OpenTofu modules under opentofu/ — outside Ansible’s scope.

Why `delegated` + Incus

Molecule’s container-native drivers (molecule-podman, molecule-docker) are the path of least resistance, but:

Most of our roles touch system-level concerns (nftables, systemd, networking) that need a real init system and full kernel surface — containers force compromises (privileged mode, systemd-enabled images, cgroup tweaks) that diverge from production.
We already operate Incus, with both container and VM workloads. The CLI, image catalog, snapshot/restore, and profile model are all things this team uses daily.
Incus VMs give true kernel isolation when needed; Incus containers (LXC, not OCI) give near-VM fidelity at container speed and run systemd cleanly.

Using delegated keeps Molecule out of the provisioning business and lets us drive Incus directly with the tooling we already trust. The downside — we own the lifecycle scripts — is real but bounded; see Lifecycle.

Other drivers considered:

molecule-podman — fast, but systemd-in-container friction and no kernel-module testing. Worth keeping as a secondary scenario for lint-heavy roles where speed matters.
molecule-vagrant — adds a VirtualBox/libvirt dependency to CI runners that don’t otherwise need it. We’d be standing up a second virtualization stack alongside Incus.
molecule-incus — no maintained first-party driver exists at the time of writing; the community options are thin. Wrapping the incus CLI from a delegated scenario is simpler and more transparent than depending on an unmaintained plugin.

Architecture

                       ┌──────────────────────────────────┐
   CI runner / dev ──▶ │ molecule test (scenario)         │
                       │  ├─ create.yml  ─────────────────┼──▶ incus launch / restore snapshot
                       │  ├─ converge.yml (the role)      │       on test host (kyojin or
                       │  ├─ idempotence                  │       a dedicated test host)
                       │  ├─ verify.yml (assertions)      │
                       │  └─ destroy.yml ─────────────────┼──▶ incus delete / snapshot restore
                       └──────────────────────────────────┘

Test instances live on a designated Incus host. Two reasonable placements:

Reuse kyojin or theia with a dedicated project (incus project create molecule) and profile that constrains CPU/RAM and pins instances to a non-production network. Cheap, no new hardware, but test load shares cycles with prod.
Stand up a small dedicated test host (a single small box, or even a nested Incus VM on one of the existing hosts). Cleaner isolation, more moving parts.

Start with (1), gated by a project-level resource limit. Move to (2) if test load grows or if we want to test changes that touch host-level Incus config without risk.

Connection plugin

Molecule’s delegated driver doesn’t dictate how the converge play reaches the instance. Two options:

community.general.incus connection plugin — talks to instances via the incus exec API. No SSH server, no key management, no network reachability requirement. The instance just needs to exist in the Incus database on the control host (which is where Molecule runs).
SSH — install openssh-server, inject a key in create.yml, expose port 22 over the test bridge. More moving parts, but matches how production playbooks actually connect.

Default to the incus connection plugin for unit-style role tests (faster, fewer dependencies). Add an SSH-based scenario for any role whose tasks differ meaningfully under the ssh connection (rare, but e.g. anything that inspects ansible_connection or relies on SSH-agent forwarding).

Lifecycle (`create.yml` / `destroy.yml`)

delegated means we write these. Sketch:

- name: Create Molecule instances on Incus
  hosts: localhost
  gather_facts: false
  vars:
    incus_remote: kyojin            # or local: if running on the host itself
    incus_project: molecule
    incus_profile: molecule-default # constrains cpu/memory, attaches test bridge
    base_image: images:debian/12
  tasks:
    - name: Launch instance
      ansible.builtin.command: >
        incus launch {{ base_image }} {{ item.name }}
        --project {{ incus_project }}
        --profile {{ incus_profile }}
        {% if item.type | default('container') == 'vm' %}--vm{% endif %}
      loop: "{{ molecule_yml.platforms }}"
      changed_when: true

    - name: Wait for instance agent
      ansible.builtin.command: >
        incus exec {{ item.name }} --project {{ incus_project }} -- true
      register: ready
      until: ready.rc == 0
      retries: 30
      delay: 2
      loop: "{{ molecule_yml.platforms }}"

    - name: Register with Molecule
      ansible.builtin.add_host:
        name: "{{ item.name }}"
        groups: molecule
        ansible_connection: community.general.incus
        ansible_incus_remote: "{{ incus_remote }}"
        ansible_incus_project: "{{ incus_project }}"
      loop: "{{ molecule_yml.platforms }}"

- name: Destroy Molecule instances
  hosts: localhost
  gather_facts: false
  tasks:
    - name: Delete instance (force)
      ansible.builtin.command: >
        incus delete --force {{ item.name }} --project molecule
      loop: "{{ molecule_yml.platforms }}"
      failed_when: false

Key knobs (set per-scenario in molecule.yml):

base_image — images:debian/12, images:ubuntu/22.04, etc. Match what production runs.
type — container (LXC) by default; vm for kernel-module or systemd-init tests where LXC’s shared kernel would mask bugs.
incus_profile — small (1 CPU, 512 MiB) for lint-and-converge; bumped for anything provisioning real services.

Reset strategy between runs

Three options, roughly in order of speed:

Snapshot-restore (preferred for inner-loop dev) — after first launch, incus snapshot create <name> clean. Subsequent runs incus snapshot restore <name> clean instead of relaunching. ~1s vs ~10–30s for a fresh launch. Implemented as an option in create.yml keyed off whether the instance already exists.
Delete + relaunch (default for CI) — destroy.yml deletes; next create.yml launches fresh from the image. Slower but unambiguously clean. Good default for CI where determinism matters more than speed.
Idempotent re-converge — never reset, rely on the role’s own idempotency to bring the instance back to baseline. Don’t do this. Hides first-run bugs (package install, user creation, file initialization) and entangles tests.

molecule test runs the full destroy → create → converge → idempotence → verify → destroy cycle and matches option 2. molecule converge (dev loop) reuses the existing instance and pairs naturally with option 1.

Verifier

Default to Ansible-based verifier (verify.yml with ansible.builtin.assert). Reasons:

No extra dependency — already have Ansible.
Same skill set as the rest of the codebase.
Sufficient for state assertions (file exists, service running, port listening, command output).

Reach for Testinfra (pytest-based) only when assertions get genuinely complex or when a role warrants table-driven tests across many parameters. Don’t introduce both.

For each role, the verifier should at minimum check:

Idempotency (Molecule’s built-in idempotence step covers this — second run reports changed=0).
Service is enabled and running (where applicable).
Configuration files exist with expected ownership and a content sanity check.
A behavioral probe (e.g. for nftables: nft list ruleset contains the expected chain; a curl to a blocked port fails).

Per-target plan

`roles/nftables`

First scenario, highest value. Container-friendly (LXC supports nftables in unprivileged containers as long as the host kernel has the modules — IncusOS does).

Platforms: Debian 12 container (matches kyojin/theia base), maybe Ubuntu 22.04 as a second platform.
Verifier: nft list ruleset parses cleanly, expected chains/sets present, default policy as configured, a representative rule actually filters traffic (use nc from a second instance on the same bridge).
Multi-instance: yes — one “target” instance with the role applied, one “probe” instance to generate traffic. Both in the Molecule scenario’s platforms list.

`playbooks/incus/tasks/*`

Many of these are small, focused task files (storage pool creation, profile configuration). They aren’t roles, so Molecule’s role-centric model doesn’t fit cleanly — wrap each in a thin “scenario playbook” under molecule/<scenario>/converge.yml that imports the task file with representative variables.

Caveat: testing Incus-configuring tasks requires Incus to be installed inside the test instance. That means either a custom Incus-on-Debian image (build once, push to a local image server) or running these tests against a VM instance with stock IncusOS as the base. The IncusOS path is closer to production. Defer until the nftables scenario is solid.

`playbooks/incus/setup-.yaml`, `playbooks/k0s/`

Treat as integration tests, not Molecule scenarios. Plan: a separate make test-integration target that provisions a throwaway Incus VM, runs the full setup playbook against it, runs a smoke check, destroys it. Out of scope for this doc.

Layering: lint → syntax → molecule

The full pipeline, fastest to slowest:

yamllint ansible/
ansible-lint ansible/
ansible-playbook --syntax-check on every playbook (cheap, catches missing vars and bad imports).
molecule test -s <scenario> for each role/scenario.

Stages 1–3 run on every push. Stage 4 runs on changes under ansible/roles/ or ansible/playbooks/incus/tasks/ (path-filtered) — full molecule runs are expensive enough that we shouldn’t pay for them on doc-only changes.

CI integration

Open question: where does CI run? This repo doesn’t currently have a CI config visible. Options:

Forgejo Actions on the in-cluster Forgejo. Runner would need network access to an Incus host (the test backend) and the right TLS client cert / token to talk to its API. Most aligned with the “everything we run, we run ourselves” posture of the rest of the stack.
GitHub Actions with a self-hosted runner deployed onto kyojin/theia. Simpler tooling, but introduces an external dependency for what is otherwise an air-gappable workflow.

Either way, the runner needs:

The Docker workspace image (or its toolchain): uv, Ansible, the community.general collection.
An Incus client config (~/.config/incus/config.yaml) with a remote pointing at the test host and a client cert trusted by that host.
Credentials for the molecule project on that host (project-scoped certificate is enough; no need for full admin).

Decision deferred — pick once Forgejo Actions readiness is settled in the broader repo.

File layout

Proposed:

ansible/
├── roles/
│   └── nftables/
│       ├── defaults/
│       ├── handlers/
│       ├── tasks/
│       ├── templates/
│       └── molecule/
│           └── default/
│               ├── molecule.yml
│               ├── create.yml
│               ├── destroy.yml
│               ├── converge.yml
│               └── verify.yml
└── molecule/
    └── _shared/
        ├── create.yml      # symlinked or imported by per-role create.yml
        └── destroy.yml

The shared create.yml / destroy.yml keeps the Incus lifecycle logic in one place. Per-role molecule.yml overrides platforms, profiles, and base images. Resist the urge to abstract further until there are at least three roles with scenarios.

Open questions

Image source — images: (LinuxContainers community images) is the obvious starting point. Do we want to mirror them locally for offline CI? Probably yes, eventually; not blocking.
Test host failure mode — if the Incus test backend is down, every PR fails. Acceptable? Or do we want a secondary container-only Molecule scenario as a fallback that runs without Incus?
Secrets in tests — none of the planned scenarios need real secrets, but if a future role does, we’ll need a SOPS-decrypt step in create.yml or fixture files. Cross that bridge when we hit it.
Coverage target — what fraction of roles/tasks should have Molecule scenarios before we call this “done”? Suggest: every role gets a scenario; task files get scenarios opportunistically as bugs surface. Don’t aim for 100% — diminishing returns past the system-level concerns.

Next steps

Build the nftables Molecule scenario as a working reference. Two platforms (Debian 12, Ubuntu 22.04), shared create.yml/destroy.yml, Ansible verifier with both state and behavioral assertions.
Add make test-roles target wrapping molecule test across all role scenarios.
Wire lint + syntax-check + role tests into whatever CI we end up with.
Document the Incus molecule project setup (profile, network, certs) in the repo — probably as a small section in this file once it’s real.
Revisit task-file testing and the integration-test strategy for setup-cluster.yaml separately.