Internal DNS

Source of truth

Container placements and IPs below are current as of 2026-09; fleet.yaml is the authority if they drift. Phase sections are dated records of the 2026-05 build-out.

This document captures the design for an internal DNS service: recursive resolver for general queries, authoritative for rrchnm.internal (or its successor), and a backend ExternalDNS in k0s can eventually write to. It records the architecture decisions, the chicken-and-egg avoidance rules, and a phased plan for standing it up.

Status (2026-05-29)

All three phases shipped. The original “combine recursive + authoritative in one daemon” plan pivoted on 2026-05-21: authoritative role moved off the resolvers to Knot DNS on a dedicated host (arceus on hyperion), and the resolvers became forward-only. Drove by the cert-manager DNS-01 → RFC2136 requirement (the file plugin can’t accept dynamic updates without an etcd backend; Knot does RFC2136 natively). See the “Decision log” and “Alternatives evaluated” sections below for the pivot rationale.

As-built summary:

Component	Reality	Phase
Authoritative	Knot DNS 3.5.x on arceus (hyperion). Two zones, both primary, no AXFR: rrchnm.internal (static, re-rendered + knotc zone-reload per Ansible run) and acme.rrchnm.internal (dynamic, journal-backed, RFC2136 via TSIG key acme-update.rrchnm.internal. HMACSHA256). Configured by setup-dns-servers.yaml Play 2 (tags: knot). Zone source: playbooks/dns/templates/knot-zone-rrchnm.j2 renders auto-A records from inventory + CNAMEs from dns_service_aliases + hand-curated A/CNAME from dns_zones + Gateway A records from ansible/vars/k0s-gateways.yaml.	Phase 3 — done
Recursive resolvers	CoreDNS 1.14.3 on three Lake Guardians: mesprit (kyojin), azelf (theia), uxie (atomsk). Binary install from upstream, sha256-pinned in group_vars/dns_resolvers.yaml. Forward-only — no file plugin. Three server blocks in Corefile.j2: acme.rrchnm.internal → arceus, rrchnm.internal → arceus (cache 300s), . → GMU campus (cache 3600s, health_check 5s). Bind on the host’s primary LAN IP.	Phase 1 — done
uxie on-prem	Moved on-prem from AWS in 2026-05-21. MAC-bound DHCP reservation on ibuypwr (RH461 reservable subnet 10.112.113.128/25 since 2026-06-09; previously DHCP-volatile on RH463 client subnet 3). Third entry in fleet’s default_dns_nameservers.	Phase 1.5 — done
Recursive forwarder	Campus DNS only (129.174.18.18 / 67.98 / 253.66); IP literals; no public-resolver fallback. Regulated-environment posture. Campus-DNS outage = external-DNS outage for the fleet, but *.rrchnm.internal keeps resolving authoritatively (resolver → arceus).	Phase 1 — done (campus-only deviation, see below)
Fleet resolver config	TF locals in opentofu/system_network.tf: default_dns_nameservers = [mesprit, azelf, uxie] only; default_dns_search = [rrchnm.internal]. Templated into every instance’s cloud-init.network-config.	Phase 2 — done
Search domain	rrchnm.internal set as a plain (search+routing) Domain on every fleet VM’s link. ping mesprit resolves bare.	Phase 2 — done
Dynamic updates	RFC2136 + TSIG against arceus for acme.rrchnm.internal. cert-manager’s heimdall ClusterIssuer carries the RFC2136 solver (selector-gated to test.rrchnm.internal only during incremental DNS-01 migration). TSIG secret in ansible/group_vars/dns_authoritatives.sops.yaml (knot_acme_update_tsig_secret, hmac-sha256), mirrored at deploy time into the cert-manager/rfc2136-tsig Secret.	Phase 3 — done (DNS-01 plumbing)
ExternalDNS	Still not deployed. Knot’s RFC2136 endpoint is in place if/when ExternalDNS is added.	Phase 3 — partial (manual dns_zones covers current needs)

Deviation from plan, worth flagging: the deployed Corefile forwards exclusively to GMU campus DNS. The original Phase 1 design specified 1.1.1.1 + 9.9.9.9 + GMU. Rationale captured in playbooks/dns/templates/Corefile.j2 and group_vars/dns_resolvers.yaml.

Why we need this

Three current pain points motivate building it:

1. *.rrchnm.internal doesn’t resolve anywhere. IncusOS hosts present TLS certs with kyojin.rrchnm.internal and theia.rrchnm.internal SANs; nothing actually resolves those names. The Incus client workflow works around this via fingerprint pinning (add the remote once with a transient /etc/hosts entry or hostname-resolving jump box, switch the remote’s addr to an IP literal afterward — see incus-os-factory-reset.md (operator note)), but it adds onboarding friction and rules out browser/curl-by-hostname for ad-hoc API access. Internal DNS makes the hostnames resolve from the LAN, so the SAN check passes for free without operator setup.
2. Single point of failure for resolution. All infra hosts and VMs currently depend on GMU campus DNS (129.174.18.18, 129.174.67.98, 129.174.253.66). Campus DNS outages would cascade into our cluster losing image pulls, apt updates, and external service resolution.
3. No place for ExternalDNS to write records. k0s has ExternalDNS deployed (per project memory) but no provider configured. Standing up a DNS backend ExternalDNS can talk to (CoreDNS-with-etcd, BIND-via-RFC2136, or PowerDNS) is a prerequisite to automating Gateway API record management.

Two DNS roles, often confused

Important to keep distinct:

Role	What it does	Examples
Recursive resolver	Answers queries for any domain by chasing referrals from root → TLD → authoritative	GMU campus DNS, 1.1.1.1 (Cloudflare), 9.9.9.9 (Quad9)
Authoritative	Answers only for zones it owns; listed in NS records of those zones	ns{1,2,3}.rrchnm.org (authoritative for rrchnm.org)

RRCHNM’s existing ns{1,2,3}.rrchnm.org are authoritative-only. They cannot replace GMU as a recursive resolver — a host pointed at them as primary DNS would fail to resolve apt.debian.org, image registries, etc. They could, in principle, host an rrchnm.internal zone (or a delegated-private subdomain), but they don’t currently — and asking the team that runs them to do this is a coordination task we’d want to avoid by running our own.

Architecture: split daemon model (as-built since 2026-05-21)

Two daemons, two roles, two host classes:

1. Authoritative — Knot DNS 3.5.x on arceus (hyperion). Serves rrchnm.internal (static, Ansible-rendered) + acme.rrchnm.internal (dynamic via RFC2136). One primary, no secondaries — the AQ114 host’s reliability bias bakes in some HA.
2. Recursive — CoreDNS 1.14.3 on mesprit + azelf + uxie. Stub-forward rrchnm.internal + acme.rrchnm.internal → arceus; forward everything else → GMU campus DNS.

The original plan had a single daemon doing both. The pivot to split was driven by the RFC2136 dynamic-update requirement for cert-manager’s DNS-01 solver — CoreDNS’s file plugin can’t accept dynamic updates without an etcd backend; Knot does RFC2136 natively. See the “Decision log” and “Alternatives evaluated” sections below for the alternatives evaluated (kresd, etcd-backed CoreDNS, acme-dns, OCI containers, etc.).

Software choice: CoreDNS (recursive) + Knot DNS (authoritative)

Original plan picked CoreDNS for both roles. The 2026-05-21 pivot kept CoreDNS for recursion and added Knot for authority.

Software	Recursive	Authoritative	Dynamic updates	Role here
CoreDNS	Yes (forward plugin)	Yes (file, etcd, k8s plugins)	Only via etcd backend	Recursive resolvers (mesprit/azelf/uxie). Stub-forward + recurse.
Knot DNS	No	Yes (mature, primary/secondary, ACL-driven)	RFC2136 native	Authoritative (arceus). Static + dynamic zones, in-controller knotc zone-reload.
BIND9	Yes	Yes	RFC2136 nsupdate	Considered for authority — Knot won on operational simplicity + journal-backed RFC2136.
PowerDNS	Two daemons	Yes (DB-backed)	REST API	Considered — heavier than Knot for our scale.
Unbound + NSD	Excellent split	Excellent split	No DDNS native	Considered — lacks RFC2136 on NSD.

Why this split: CoreDNS’s plugin model is wonderful for recursion + cache, but the file plugin is static-only — dynamic updates require pulling in the etcd plugin and operating an etcd quorum just for DNS. Knot DNS treats authoritative-with-dynamic-updates as its default mode (zonefile-load: difference, journal-backed). One daemon doing what it’s designed for + one daemon doing what it’s designed for is simpler than CoreDNS-with-etcd.

Where it runs: four Incus containers across four hosts

Production reality:

Container	Host	IP	Role
arceus	hyperion (AQ114)	static	Knot DNS authoritative
mesprit	kyojin (RH461)	DHCP-reserved	CoreDNS recursive
azelf	theia (AQ114)	static	CoreDNS recursive
uxie	atomsk (RH461)	DHCP-reserved	CoreDNS recursive

Current addresses: fleet.yaml (the inventory source of truth).

All four are Incus containers (not VMs) for the bootstrap reasons in “Chicken-and-egg avoidance rules” below — containers start from an image template already on disk, no cloud-init / DNS dependency at boot. Tofu-managed via their fleet.yaml entries, rendered by fleet.tf (originally a dedicated opentofu/dns.tf, since consolidated). Configured by ansible/playbooks/setup-dns-servers.yaml — Play 1 baseline (init + nftables), Play 2 (tags: knot) installs Knot on arceus from the CZ.NIC apt repo, Play 3 (tags: coredns) installs CoreDNS on the resolvers from upstream binary release pinned by SHA256.

Listener binding: each daemon binds on the host’s primary LAN macvlan IP via explicit bind (so other LAN clients can reach it and so the listener doesn’t collide with systemd-resolved’s 127.0.0.53:53 inside the container).

Naming: rrchnm.internal for now, monitor for migration to int.rrchnm.org

.internal is not formally reserved by IANA/ICANN as a private-use TLD (unlike .local for mDNS or .test). There’s been IETF discussion about reserving it, and ICANN has signaled it won’t delegate it as a public TLD, but no formal guarantee.

Name	Pros	Cons
rrchnm.internal (current)	Already in IncusOS host configs and cert SANs; no migration cost	Not formally reserved; small risk of future ICANN delegation
int.rrchnm.org (delegated subdomain you control)	Formally yours; no TLD risk	Migration: re-seed IncusOS with new domain, regenerate certs, update all configs

Sticking with rrchnm.internal for now. The migration cost is real (touch every IncusOS host’s network seed + factory-reset to regenerate certs + update every cloud-init network config) and the .internal-collision risk is low and slow. If a future IncusOS reinstall is happening anyway, bundle the domain change in then. Until then, document the trade-off and move on.

Chicken-and-egg avoidance rules

This is the part that makes internal DNS actually robust. The dependency layers (top consumes bottom):

┌─────────────────────────────────────────────┐
│  Layer 4: k8s workloads (Forgejo, Zot, ...) │
│  - Depend on cluster DNS (CoreDNS in k8s)   │
│  - Depend on infra DNS for external lookups │
├─────────────────────────────────────────────┤
│  Layer 3: k0s nodes + VMs                   │
│  - Need infra DNS for apt, image pulls,     │
│    rrchnm.internal name resolution          │
├─────────────────────────────────────────────┤
│  Layer 2: Infra DNS resolvers (THIS DOC)    │
│  - Authoritative for rrchnm.internal        │
│  - Recursive for everything else            │
│  - Must not depend on layers 3 or 4         │
├─────────────────────────────────────────────┤
│  Layer 1: IncusOS bare-metal hosts          │
│  - Bootstrapped without DNS (operator       │
│    talks via IP-based remote URLs)          │
└─────────────────────────────────────────────┘

The rule: each layer can only depend on layers below it. Concrete tactics:

1. Don’t put DNS in k0s

The big one. k8s needs DNS to pull container images and resolve hostnames during node registration. If DNS is a k8s Deployment, k8s outage → DNS outage → nothing recovers. This is why we run CoreDNS in Incus containers, not as a k8s pod. (Cluster DNS for in-cluster service resolution stays in k8s as kube-dns/CoreDNS — that’s fine, it’s a different concern.)

2. DNS resolvers must not depend on DNS to start

The resolver’s own boot path can’t require DNS resolution:

• Forwarder list uses IP literals: forward . 1.1.1.1 9.9.9.9 129.174.18.18 — never cloudflare-dns.com.
• The resolver’s own /etc/resolv.conf points at 127.0.0.1 so it self-resolves through itself once running, plus a public IP fallback for boot edge cases.
• Static zone file with IP literals only — authoritative answers require zero recursion.
• Pre-installed CoreDNS, no apt-during-boot. Either the Incus container template has CoreDNS installed at image-build time, or the container image is one with CoreDNS preinstalled. Container start = unpack + run binary; no network calls.

3. DNS containers boot without depending on Layer 3

Use Incus containers, not VMs. Containers start from an OS image template already on the host’s storage pool — no network bootstrap. A VM with cloud-init has a chicken-and-egg risk (cloud-init typically needs DNS to apt-update); a container started from a pre-built image just runs.

4. Hardcoded /etc/hosts on the DNS containers

On mesprit and azelf, /etc/hosts has entries for:

• The other DNS container’s IP (so mesprit can find azelf during config sync without DNS) — populated from inventory by the dns_bootstrap_hosts map in group_vars/dns_resolvers.yaml, so an IP change in inventory.yaml propagates automatically
• Any package mirror or pull-target needed during upgrades (deb.debian.org etc.)

Tiny safety net. Means a DNS-host-by-itself can still apt-update or ping its peer if needed.

5. HA across hosts

mesprit on kyojin + azelf on theia + uxie on atomsk. All three forward-only resolvers (stub-forward rrchnm.internal + acme.rrchnm.internal to arceus, the Knot DNS authoritative; recurse everything else to GMU campus DNS). The authoritative zone moved off the resolvers to arceus in 2026-05 — see “Decision log” below.

Each fleet VM’s resolver list (rendered into cloud-init.network-config from TF locals):

nameservers:
  addresses:
    - <azelf-ip>          # azelf  (theia)     — actual values: fleet.yaml
    - <mesprit-ip>        # mesprit (kyojin)
    - <uxie-ip>           # uxie   (atomsk)
  search:
    - rrchnm.internal

The original plan included a tertiary public-IP fallback (1.1.1.1) so a full internal-resolver outage would degrade rather than dead-end. The as-built configuration drops that fallback for two reasons that emerged during implementation:

1. systemd-resolved doesn’t fall through on NXDOMAIN. Mixing internal + public servers in one list means resolved can pick a public server as “current,” and any .rrchnm.internal query returns campus’s authoritative NXDOMAIN with no retry against the internal pair. Removing the public servers from the link list is what makes split-DNS actually work.
2. Regulated-environment posture. Forwarding to anywhere outside campus DNS is unnecessary egress.

Trade-off accepted: a full internal-resolver outage = full external-DNS outage for the fleet. Acceptable because (a) the resolvers run on different hosts (mesprit on kyojin, azelf on theia, uxie on atomsk), so simultaneous loss requires either both hosts down or both containers crashed; (b) apt/image pulls during normal operation are not in the chicken-and-egg path; (c) cold-boot recovery requires DNS to come up before non-DNS VMs do — see the Phase 2 open caveat about the boot-time race.

6. Provision DNS before anything that depends on it

In the deployment pipeline:

1. make infra provisions Tofu resources. The incus_instance.dns_server resources have no explicit depends_on from k0s/docker/lamp/etc. — they’re independent in the dependency graph — but tofu apply creates them as part of the same transaction, so timing is usually fine. The boot-time race surfaces when a non-DNS instance happens to reach cloud-init’s modules-final before the same-host DNS container’s CoreDNS systemd unit is healthy (observed once on kyogre).
2. k0s nodes come up after DNS is reachable; their cloud-init can resolve apt mirrors and image registries via the internal pair → campus forwarders.
3. ExternalDNS in k0s writes to the DNS backend later (Phase 3) — but its writes don’t gate any dependency.

A stricter ordering — two-phase tofu apply with -target=incus_instance.dns_server then make dns then full apply — would eliminate the race. Not yet wired into the Makefile; tracked as an open mitigation in Phase 2.

7. ExternalDNS is consumer-side, not producer-side

ExternalDNS in k0s writes records to the DNS backend; it doesn’t resolve through it. So:

• ExternalDNS lives in k0s (Layer 4) — fine
• It writes to the DNS backend (Layer 2) via an API
• Loss of ExternalDNS doesn’t affect resolution; loss of the DNS backend stops both writes (ExternalDNS) and reads (everyone)

No new chicken-and-egg. ExternalDNS adds nothing to the dependency graph.

8. Operator (jump box) doesn’t strictly need /etc/hosts

For Incus client operations, no /etc/hosts entries are needed — remotes can use IP-literal addr URLs in /configs/incus/config.yml (the cert is pinned by fingerprint, SAN content is irrelevant after first remote-add). For SSH/curl-by-hostname workflows, a /etc/hosts fallback for kyojin.rrchnm.internal and theia.rrchnm.internal (cost: two lines) keeps a bootstrap-recovery path open when DNS itself is broken — useful but optional.

Phased plan

Phase 1: stand up the resolvers — DONE (2026-05)

As-built deliverables:

• opentofu/dns.tf (since consolidated into fleet.yaml/fleet.tf) — incus_instance.dns_server iterated over local.dns_servers (azelf on theia, mesprit on kyojin). Containers, not VMs (per §3 of chicken-and-egg rules). Macvlan to host LAN.
• ansible/playbooks/setup-dns-servers.yaml — installs the pinned CoreDNS binary, renders zone + Corefile, enables systemd unit.
• ansible/playbooks/dns/templates/coredns.zone.j2 — Jinja template for rrchnm.internal, generated from groups['all'] so adding a host updates DNS automatically. Supports CNAME aliases via the dns_service_aliases map in group_vars/dns_resolvers.yaml.
• ansible/playbooks/dns/templates/Corefile.j2 — CoreDNS config (file plugin for the zone + forward . to campus DNS with health-check + 1h cache).
• ansible/playbooks/dns/templates/coredns.service.j2 — systemd unit.
• ansible/group_vars/dns_resolvers.yaml — CoreDNS version pin (1.14.3 + SHA256), zone name, upstreams, bootstrap /etc/hosts seeding, service aliases.

Smoke tests (still valid):

# From any LAN client (or this jump box):
dig @<mesprit-ip> mesprit.rrchnm.internal +short      # mesprit (IP: fleet.yaml)
# → mesprit's own IP

dig @<azelf-ip> example.com +short                     # azelf, recursive
# → real IP from campus upstream

# End-to-end via stub (after Phase 2):
resolvectl query mesprit.rrchnm.internal

Phase 2: point everything at it — DONE (2026-05)

Implementation diverged from the original Ansible-vars plan; landed in OpenTofu locals instead:

• opentofu/system_network.tf — default_dns_nameservers = local.internal_dns_nameservers (just the azelf + mesprit IPs; campus DNS deliberately not in the fleet’s list). default_dns_search = ["rrchnm.internal"] for both routing and bare-hostname resolution. fallback_dns_servers retained for use by dns.tf (azelf + mesprit themselves can’t depend on the pair they ARE) and as the upstream forwarder list inside CoreDNS.
• Every TF file with a cloud-init.network-config (kube.tf, docker.tf, gpu.tf, lamp.tf, caddy.tf, backup.tf, misc.tf — all since consolidated into fleet.tf) — templates addresses: …default_dns_nameservers + search: …default_dns_search into the netplan v2 block. DHCP branches also set dhcp4-overrides.use-dns: false so DHCP-supplied resolvers don’t layer on top.
• No Ansible group_vars var. The original plan specified ansible/group_vars/all.yaml: dns_nameservers as single source of truth. In practice the cloud-init.network-config is the only consumer (post-cloud-init resolver changes happen via netplan, which is what cloud-init wrote), and keeping a single source of truth in TF locals is cleaner than crossing the TF/Ansible boundary.

After this phase: *.rrchnm.internal resolves from every fleet VM; ping mesprit works via search domain; systemd-resolved is locked to internal pair (no campus-DNS NXDOMAIN-no-fallthrough trap). Verified end-to-end with resolvectl status showing DNS Domain: rrchnm.internal and DNS Servers: listing the internal resolvers (the addresses in that phase-era transcript predate the current assignments).

Open caveat: boot-time DNS race. With campus DNS removed from the fleet’s resolver list, new instances’ cloud-init runs apt-get update against deb.debian.org resolvable only through the internal pair. If mesprit/azelf aren’t reachable when cloud-init hits modules-final, the install fails (observed once on kyogre — Temporary failure resolving 'deb.debian.org' → modules-final exit 1). Recovery is manual (incus exec … apt-get install of openssh-server + python3-apt, then re-run Ansible). Mitigations not yet implemented:

• Add a bootcmd DNS-wait to the shared cloud-init template (lowest-friction)
• Re-order make up so make dns (CoreDNS install) precedes tofu apply for non-DNS instances (correct but requires two-phase apply)
• Keep one campus resolver in default_dns_nameservers as a boot-only fallback (partial regression of the split-DNS goal)

Phase 3: dynamic updates — DONE (2026-05-21) via Knot DNS on arceus, ExternalDNS still deferred

Knot DNS on arceus accepts RFC2136 UPDATEs (TSIG-authenticated) for the acme.rrchnm.internal subzone, journal-backed. cert-manager’s heimdall ClusterIssuer carries the RFC2136 solver pointed at knot.rrchnm.internal:53 (CNAME → arceus). Selector currently scopes to dnsNames: [test.rrchnm.internal] only during incremental DNS-01 migration; flip per-Certificate as migration proceeds.

ExternalDNS still not deployed — manual dns_zones entries (in group_vars/dns_resolvers.yaml, consumed by arceus’s zone template via vars_files:) cover the records that can’t be derived from inventory (kuvasz round-robin, git LB, garage round-robin). If/when ExternalDNS demand grows: configure ExternalDNS in k0s with --provider=rfc2136 --rfc2136-host=arceus.rrchnm.internal against the same TSIG key, no daemon changes on the DNS tier.

Decision log

Choices made above that future-you may want to revisit:

• Split daemon: Knot DNS for authority, CoreDNS for recursion. Originally planned as single-daemon CoreDNS for both. Pivoted 2026-05-21 to split when the cert-manager DNS-01 + RFC2136 requirement landed. CoreDNS’s file plugin is static-only; Knot DNS handles dynamic updates as its default mode. See “Alternatives evaluated” below for the path through five design pivots.
• Three CoreDNS resolvers + one Knot authoritative, not active-active authoritative. arceus is the single source of authority — eliminates the zone-file-drift risk that an active-active pair would have. The resolvers (mesprit/azelf/uxie) provide HA across three hosts; cert-manager retries against arceus give a 7-day budget for arceus outages before any cert actually expires.
• arceus on hyperion, not theia. AQ114 reliability bias (same logic as the controller placement in k0s-cluster-rebalance). hyperion has the most pool headroom of the IncusOS hosts (zfs-raidz1 ~5.4TB).
• rrchnm.internal (not formally reserved) over int.rrchnm.org (delegated subdomain). Migration cost outweighs the hypothetical TLD-collision risk. Re-evaluate if IncusOS gets reinstalled or if ICANN actually moves on .internal.
• Tofu-managed Incus containers, not Ansible-managed bare-metal install. Lifecycle is identical to other Incus instances in the repo. Apply, destroy, recreate idempotently.
• Public-IP fallback in resolver chain. Reversed at implementation. The deployed configuration has no public-IP fallback at any layer: CoreDNS forwards only to campus DNS, and fleet VMs talk only to the internal trio. Reasons: (a) systemd-resolved’s no-fallthrough-on-NXDOMAIN behavior makes mixed internal+public resolver lists actively harmful for split DNS; (b) the regulated-environment posture argues against unnecessary egress. Full internal-resolver outage = full external-DNS outage for the fleet. Mitigated by HA across three resolver hosts on three sites.

Alternatives evaluated for the authoritative tier

Option	Verdict
Knot DNS on dedicated arceus + CoreDNS on mesprit/azelf/uxie (shipped)	Physical security-boundary split; each daemon owns :53 on its own host; tight authoritative-server allowlist; single zone-file render. Resolver tier briefly ran Knot Resolver (kresd); reverted to CoreDNS — see “kresd reversion” below.
Knot DNS + kresd collocated on mesprit/azelf	Earlier design. Port-juggling (Knot on :5353), zone file rendered identically on both hosts (drift risk), authoritative ACL has to coexist with resolver’s wide-open :53. Rejected after user pushback.
BIND9 replaces everything	Heavier config for marginal gain. Knot DNS preferred.
acme-dns + cert-manager webhook (the original 2026-05-21 plan)	Blocked by the cluster’s apiserver-can’t-reach-ClusterIPs limitation (same root cause that retired the cert-manager admission webhook — see cilium-gateway-l7). cert-manager’s built-in RFC2136 sidesteps this (in-process, no APIService).
Stay on HTTP-01 entirely	Rejected — accumulates redirect / namespace / wildcard friction.
OCI containers for kresd 6.x (YAML config)	Probed; hit kyojin Broadcom tg3 IPVLAN-EBUSY (now sidesteppable per incus-oci-containers) + OCI rootfs /run wipe + entrypoint quirks. Reverted to Lua kresd 5.x, then reverted to CoreDNS entirely.
Docker-in-Incus for kresd 6.x	Most attractive of the YAML-enabling options; not pursued once kresd was reverted out of the resolver tier.
Source-build kresd 6.x via uv	~2-3h build + ongoing maintenance for an aesthetic preference. Rejected.

kresd reversion (2026-05-21)

Briefly shipped the resolver tier as Knot Resolver (kresd) during a full-CZ.NIC-stack pivot. Reverted to CoreDNS after kresd 6.x (YAML config) was blocked by Debian’s python3-metapackage release cycle (knot-resolver6 needs python3 >= 3.14; Trixie ships 3.13). With kresd stuck on Lua 5.x indefinitely, the case for a second CZ.NIC package across three resolver hosts collapsed — CoreDNS is a known-good fit for pure forward+cache, sha256-pinned upstream binary, single systemd unit, Caddyfile-syntax config consistent with the rest of the fleet. Migration tasks at the top of setup-dns-servers.yaml stop kresd@1, purge knot-resolver, remove CZ.NIC apt source, remove /etc/knot-resolver — idempotent, no-ops on clean hosts.

Re-evaluation triggers

Re-read this doc and possibly redesign if:

• Done in 2026-05: the third on-prem DNS replica is uxie, originally on ibuypwr, since relocated to atomsk (ibuypwr is now RH463, GPU-only). Brings the Lake Guardian trio fully on-prem (the AWS instance retired). hyperion could still host a 4th if AQ114-side resolver redundancy ever becomes useful (e.g., to survive simultaneous theia + ibuypwr outage).
• Compliance regime forbids egress queries to public DNS — already mostly there (no public fallback in fleet or in CoreDNS upstreams). Final step would be to switch the CoreDNS forward . block from campus DNS to an internal-only recursive resolver.
• We start needing DNSSEC or signed zones — CoreDNS supports it but the simpler path is BIND.
• ExternalDNS demand grows beyond one or two zones — etcd-backed CoreDNS or a switch to PowerDNS becomes worthwhile.
• A naming collision actually happens with .internal — migrate to int.rrchnm.org (or another delegated subdomain). Cost is a coordinated re-seed of all IncusOS hosts; doable if we have a maintenance window.
• The team grows or operator machines proliferate — onboarding repeatedly hits the first-remote add SAN dance (resolve hostname once, pin cert, switch to IP). Painful at scale; that’s the cue to actually build internal DNS so the SAN check passes without operator setup.