ACME for off-fleet clients

Historical worked example — May 2026

The kuvasz bring-up recorded here predates several changes: DNS-01 now works (zone authority moved to Knot on arceus, which accepts RFC2136 — landmine #3’s “static CoreDNS refuses UPDATEs” era is over for clients that target it); kuvasz itself has since moved to the armin docker VM behind the Incus-side HAProxy fleet, with its A record on the internal VRRP VIP; the XCP-ng-era LB pair named here is retiring. IPs reflect May 2026 — fleet.yaml is the authority. The six-landmine pattern, the diagnostic checklist, and the client-agnostic recipe are the durable content. The LB-side routing fix (landmine #6) is summarized conceptually; its ruleset lives operator-side.

Scenario: an ACME client running outside the Incus fleet (XCP-ng-side VM, third-party host, anything that didn’t go through this repo’s init Ansible role) wants to issue a cert from heimdall (the rrchnm internal step-ca) for a *.rrchnm.internal hostname.

Canonical worked example: the kuvasz.rrchnm.internal cert, bound up 2026-05-19 by a Caddy container on an XCP-ng docker VM (pochita at 10.112.113.210), fronted by the XCP-ng HAProxy pair (agni/rudra at .73/.74, VIPs at .70/.71). Six distinct landmines, each with its own fix. This note exists so the next person/client through the same gauntlet doesn’t re-debug them.

Related: internal-cert-authority.md + internal-ca-bootstrap.md (operator notes, not published), and internal-dns (zone authoring).

---

TL;DR

For any off-fleet ACME client wanting a heimdall-issued cert:

1. Trust — distribute the rrchnm root CA cert to the client; in Caddy point at it via the trusted_roots directive in the ACME issuer block (not the OS trust store — scoped narrower).
2. Resolution — add a DNS A record for the cert subject via the new dns_zones mechanism in ansible/group_vars/dns_resolvers.yaml; re-run make dns.
3. Challenge selection — DNS-01 doesn’t work yet (static CoreDNS rejects RFC 2136 UPDATEs). TLS-ALPN-01 needs PROXY-protocol handling if the network path uses TCP-passthrough load balancing. Use HTTP-01.
4. HAProxy frontend match — ensure a :80 frontend exists with a use_backend ... if { hdr(Host) -i <name> } rule routing the .well-known/acme-challenge/ path to the docker VM.
5. Reachability — the trap. If ICMP/ARP succeed but TCP times out from heimdall to the LB, the LB’s F5-return policy routing is black-holing internal-LAN replies. Fix on the LB side — see landmine #6.

---

The six landmines (in order encountered)

1. Client doesn’t trust heimdall’s serving cert

Symptom (Caddy log):

HTTP request failed; retrying  {"url": "https://heimdall.rrchnm.internal/acme/acme/directory",
"error": "performing request: ... tls: failed to verify certificate:
x509: certificate signed by unknown authority"}

Cause: the Caddy container’s OS trust store doesn’t include the rrchnm internal root. (Fleet hosts get this via the init Ansible role’s ca-trust task at init_ca_cert_path = /usr/local/share/ca-certificates/rrchnm-internal-ca.crt; off-fleet clients don’t.)

Fix — Caddyfile trusted_roots:

kuvasz.rrchnm.internal {
    reverse_proxy ...
    tls {
        issuer acme {
            dir            https://heimdall.rrchnm.internal/acme/acme/directory
            email          syschnm@gmu.edu
            trusted_roots  /etc/caddy/rrchnm-internal-ca.crt
        }
    }
}

The trusted_roots directive scopes the trust override to ACME calls only — the rest of Caddy keeps using the OS trust store for non-ACME TLS. Bind- mount the root cert into the container; permissions don’t matter (Caddy reads it once at config-load time).

Naming gotcha: the JSON config field is ca_root. The Caddyfile directive is trusted_roots (plural, accepts a space-separated list of PEM file paths). Mixing them gives unrecognized ACME issuer property: ca_root at parse time.

2. Caddyfile syntax: explicit issuer vs implicit shortcuts conflict

Symptom:

parsing caddyfile tokens for 'tls': cannot mix issuer subdirective
(explicit issuers) with other issuer-specific subdirectives (implicit issuers)

Cause: the tls <email> shorthand syntax (tls syschnm@gmu.edu {...}) is itself an implicit-issuer subdirective. You can’t combine it with an explicit issuer acme { ... } block.

Fix — move email inside the issuer block:

tls {                                    <-- no email arg
    issuer acme {
        dir            ...
        email          syschnm@gmu.edu   <-- email lives here
        trusted_roots  ...
    }
}

3. DNS-01 challenge fails REFUSED against CoreDNS file plugin

Symptom:

solving challenges: presenting for challenge:
adding temporary record for zone "rrchnm.internal.":
dns response error code "REFUSED" (5)

Cause: Caddy was configured for DNS-01 (often via an rfc2136 dns provider block) and is trying to publish _acme-challenge.<name>.rrchnm.internal TXT via RFC 2136 dynamic DNS UPDATE against mesprit/azelf. The CoreDNS file plugin serves static zones and refuses UPDATEs.

Fix today: drop the dns ... block from the Caddyfile; Caddy defaults to HTTP-01 + TLS-ALPN-01.

Fix future: wildcard cert use cases will need DNS-01 — and this has since HAPPENED, differently than planned: authority moved to Knot DNS on arceus, which accepts RFC2136 + TSIG natively (see internal-dns). Landmine #3 only applies to clients still pointed at the forward-only resolvers.

4. DNS A record missing for the cert subject

Symptom: Caddy log shows challenge_type: http-01 then a long hang with no follow-up. The hang is heimdall trying to resolve <name>.rrchnm.internal to know where to send the HTTP-01 callback.

Cause: the CoreDNS zone has no A record. The three existing mechanisms in coredns.zone.j2 don’t cover off-fleet services:

Mechanism	Source	Fit for off-fleet?
Per-host A records	inventory.yaml (ansible_host)	Bad fit — adds a fake “host” to inventory
CNAMEs	dns_service_aliases in group_vars/dns_resolvers.yaml	No — CNAME target must be an inventory host
Cilium Gateway A records	ansible/vars/k0s-gateways.yaml	No — k0s only

Fix: new fourth mechanism dns_zones added to group_vars/dns_resolvers.yaml, schema mirrors the XCP-ng ansible repo’s dns_zones var (cross-repo copy-paste consistency during migration). Rendered by coredns.zone.j2 between the service-alias CNAMEs and the Cilium-gateway A records.

dns_zones:
  - fqdn: rrchnm.internal
    resource_record_sets:
      - type: A
        name: kuvasz
        ips: [10.112.12.73, 10.112.12.74]   # agni + rudra HOST IPs, NOT the F5-NAT'd VIPs

Critical pattern: point internal-only hostnames at the LB hosts’ own IPs (.73/.74 for agni/rudra), not the F5-NAT’d VIPs (.70/.71). See landmine #6 below for the full reasoning — short version: the LB hosts’ mangle chain force-routes all responses from the public VIPs through the F5-return path, which breaks direct-LAN replies. Sourcing responses from the host IPs instead bypasses that trap entirely. Round-robin across both host IPs mirrors the master-master VRRP pattern of the existing VIP split.

Template whitespace gotcha (debugged live): with Jinja trim_blocks=True (Ansible default) and lstrip_blocks=False, putting {%- (left-strip) on a block tag immediately after an output line eats the output line’s trailing newline — records render concatenated on a single line. Pattern to follow: bare {% for %} / {% endfor %} (no minus) bracketing the output-producing inner loop; use {%- ... %} only on logic-only tags. See the existing inventory-A-record loop in coredns.zone.j2 as the reference idiom.

5. HTTP-01 viable, TLS-ALPN-01 blocked by PROXY-protocol on TCP backend

The XCP-ng HAProxy frontend pair routes kuvasz differently for the two paths:

Path	Backend mode	Send-proxy-v2
:80 HTTP	mode http (HTTP-aware forwarding)	no
:443 HTTPS	mode tcp (TLS passthrough + SNI)	yes

TLS-ALPN-01 cannot succeed through the HTTPS path without listener-side PROXY-protocol awareness in Caddy. HAProxy prepends a 16-byte binary PROXY-v2 header before forwarding TCP bytes; Caddy on container :443 sees PROXY v2 header || TLS ClientHello, tries to parse the header as TLS, handshake fails, challenge can’t complete. To enable it would need a Caddy global block:

{
    servers {
        listener_wrappers {
            proxy_protocol { allow 10.112.0.0/16 }
            tls
        }
    }
}

HTTP-01 is unaffected — mode http rewrites cleanly, no PROXY-v2 in the way. Simplest fix: explicit disable_tlsalpn_challenge in the Caddy issuer block keeps Caddy on the HTTP-01 path and avoids the PROXY-v2 dead end.

issuer acme {
    ...
    disable_tlsalpn_challenge
}

6. The HAProxy hairpin-routing trap — ICMP works, TCP times out

Symptom: from heimdall, ping to the LB addresses is fine (sub-ms, ARP REACHABLE), but nc -zv <ip> 80 times out — same on :443.

Cause (conceptually): the LB hosts carry F5-return policy routing — responses leaving with sport ∈ {80, 443} are marked and routed via the F5-return gateway instead of the normal LAN gateway. External F5’d traffic needs this (F5 SNATs requests, so replies must return through it); internal direct-LAN callers get black-holed — their SYN arrives, HAProxy’s SYN-ACK takes the F5 path, and never reaches them via the LAN.

Fix (LB-side, summarized): discriminate by the source IP of the response — only responses originating from the F5-NAT’d VIPs take the F5-return path; responses from the LB hosts’ own IPs (i.e. direct-LAN connections) use the normal gateway. This pairs with landmine #4’s pattern (internal-only hostnames resolve to the LB hosts’ own IPs, never the F5-NAT’d VIPs), which is what makes the discriminant reliable — no F5-side cooperation needed. Two earlier destination-based attempts failed before the source-IP filter landed; the concrete ruleset and iteration history are operator-only (preserved in this note’s operator-side stub).

---

Diagnostic checklist (next time)

For any future off-fleet client failing to ACME against heimdall, walk in order — earliest-failing landmine is the one to fix first:

1. Trust — from client: curl -v https://heimdall.rrchnm.internal/acme/acme/directory. Should NOT say “unknown authority.” If it does → step 1.
2. DNS — from heimdall (or any fleet host): dig +short <name>.rrchnm.internal. Should return real IP(s). If NXDOMAIN → step 4 (add to dns_zones).
3. TCP reach — from heimdall: nc -zv <ip> 80. Should succeed in <1s. If timeout (ping works, TCP doesn’t) → step 6 (LB mangle trap).
4. HAProxy routing — from heimdall: curl -v -H 'Host: <name>.rrchnm.internal' http://<vip>/.well-known/acme-challenge/probe.
   • 404 + Server: Caddy → path clean, just no live token.
   • 503 Service Unavailable → HAProxy backend marked DOWN (health check fails because Caddy answers 4xx to OPTIONS /; remove check or tune it).
   • Connection refused → no :80 frontend, or wrong ACL.
5. Caddy challenge port — container :80 must be port-published to the docker VM. Container’s Caddy must bind :80 for HTTP-01 callbacks. If the port-publish maps to a non-80 port externally, add alt_http_port <port> to the issuer block.

---

Reusable pattern (any off-fleet ACME client)

Caddy is one client. The same six landmines apply to lego, certbot, cert- manager-elsewhere, acme.sh, etc. Generic recipe:

1. Distribute root — drop the rrchnm internal root CA at a path the client reads. Caddy: trusted_roots. lego: LEGO_CA_CERTIFICATES. certbot: OS trust store + --server. acme.sh: CA_BUNDLE. Avoid stuffing it into the OS trust store unless other tools on the same host also need it.
2. Point at heimdall’s ACME directory — https://heimdall.rrchnm.internal/acme/acme/directory or the alias https://ca.rrchnm.internal/acme/acme/directory (the ca CNAME comes from dns_service_aliases in group_vars/dns_resolvers.yaml).
3. Add A record for the cert subject via dns_zones (this repo) AND/OR the XCP-ng ansible repo’s matching var. Run make dns. Verify with dig from heimdall’s vantage.
4. Use HTTP-01 unless wildcards are needed. disable_tlsalpn_challenge or equivalent if the network path has PROXY-protocol on :443.
5. Wire up the HAProxy :80 frontend rule (if traffic traverses HAProxy). use_backend http-<name>.rrchnm.internal if { hdr(Host) -i <name>.rrchnm.internal } !{ ssl_fc }. Match ip_whitelist ACL or equivalent.
6. Verify TCP reachability from heimdall to the LB VIPs. ICMP-works-but- TCP-times-out is the mangle-trap signature — fix on the LB side.

---

Changes landed (this repo)

• ansible/group_vars/dns_resolvers.yaml — added dns_zones block. Kuvasz A records point at the LB hosts’ own IPs (10.112.12.73/74, agni/rudra), NOT the F5-NAT’d VIPs (.70/.71). Inline comment documents the “internal-only hostnames go to host IPs” rule for future entries. Also renamed init_ca_cert → init_ca_root_cert everywhere in ansible/ (parallel rename for clarity vs intermediate; see commit history).
• ansible/playbooks/dns/templates/coredns.zone.j2 — render section for dns_zones; supports A + CNAME with optional per-record ttl.
• ansible/README.md — dns_zones documented as the fourth zone-record mechanism.

Changes landed (XCP-ng ansible repo, not this repo)

• route_web_for_lb mangle chain on the LB pair — final form filters by response source IP (landmine #6 above; ruleset details operator-only). heimdall’s HTTP-01 validation succeeded immediately after the fix — the run of acme:error:connection errors ended at 18:42:44 with status: valid.
• Optional cleanup deferred: tighten the LB host-firewall :80/:443 accept with source restrictions (visibility hygiene; not required).

End-to-end verification (2026-05-19, post-bringup)

heimdall step-ca log proves the chain end-to-end:

2026-05-19T18:42:44  http-01 challenge validated for kuvasz.rrchnm.internal
2026-05-19T18:42:45  cert issued
                     issuer="RRCHNM Internal Intermediate CA"
                     sans="map[dns:[kuvasz.rrchnm.internal]]"
                     serial=316998321906591282463559570770140021861
                     valid-from "2026-05-19T18:40:25Z"
                     valid-to   "2026-05-20T18:41:25Z"

24h cert lifetime is heimdall’s defaultTLSCertDuration claim (ca.json’s ACME provisioner claims block — set at 24h; see the CA design operator note). Caddy will auto-renew on its default ~2/3-lifetime schedule.

Final-state verification (2026-05-19, after the iteration ended): with DNS pointing at [.73, .74] and the mangle filtering on saddr:

# from heimdall — direct-LAN reach to LB host IPs (no F5 path)
$ incus exec hyperion:heimdall -- nc -zv 10.112.12.73 80
Connection to 10.112.12.73 80 port [tcp/http] succeeded!
$ incus exec hyperion:heimdall -- nc -zv 10.112.12.74 80
Connection to 10.112.12.74 80 port [tcp/http] succeeded!

# from operator workstation (10.112.112.x):
$ curl --cacert rrchnm-internal-ca.crt https://kuvasz.rrchnm.internal   # works
$ curl https://teachinghistory.org                                       # works (F5 hairpin)

Both paths green, no F5 involvement.

---

Last updated: 2026-05-19. Bound up the same session that introduced the dns_zones mechanism, the init_ca_root_cert rename, and the LB-side mangle-chain fix.