macOS: reaching *.rrchnm.internal

How to set up a macOS laptop to consume internal services — the dashboards and APIs served under *.rrchnm.internal behind heimdall-issued (internal step-ca) TLS certs: kuvasz, grafana, forge, bao, Garage, Zot, the Incus UIs, heimdall’s own ACME directory, etc.

This is the client/consumer side. The mirror-image server/issuer side — getting an off-fleet Caddy (or any ACME client) to actually obtain a heimdall cert — is acme-off-fleet-clients.md (operator note). CA chain design + rotation: internal-cert-authority.md (operator note), sop-ca-bundle-rotation.md (operator note). Internal zone authoring: internal-dns.

Two things a Mac needs, and they’re independent: (1) DNS that resolves the internal zone, and (2) trust of the RRCHNM internal root CA. A failure in either looks different — diagnose them separately (the checklist at the bottom).

---

TL;DR

mkdir -p ~/rrchnm

# 1. Root CA — extract from the repo (roots are public-by-design plaintext)
sed -n '/^ca_root_cert: |/,/END CERTIFICATE/p' \
  vibes/infra/ansible/group_vars/all.yaml | sed '1d;s/^  //' > ~/rrchnm/root-ca.crt

# verify it's the right root before trusting it:
openssl x509 -in ~/rrchnm/root-ca.crt -noout -subject -fingerprint -sha256
#   subject=CN=RRCHNM Internal Root CA
#   sha256 = 6F:96:28:01:B8:25:EC:33:03:59:71:F2:D8:DE:1C:13:49:EC:88:0F:F6:63:32:BC:68:C0:D8:6A:E5:8F:0D:B1

# 2a. Browsers (Safari/Chrome) → System keychain
sudo security add-trusted-cert -d -r trustRoot \
  -k /Library/Keychains/System.keychain ~/rrchnm/root-ca.crt

# 2b. CLI tools (curl/openssl/etc) → DON'T read the keychain; use a file bundle
cat /etc/ssl/cert.pem ~/rrchnm/root-ca.crt > ~/rrchnm/ca-bundle.pem
# add to ~/.zshrc:
export SSL_CERT_FILE="$HOME/rrchnm/ca-bundle.pem"
export CURL_CA_BUNDLE="$HOME/rrchnm/ca-bundle.pem"

# verify
curl -I https://kuvasz.rrchnm.internal      # clean HTTP/2, no curl: (60)

Reference values:

Thing	Value
Root CA CN	RRCHNM Internal Root CA (self-signed, valid 2026-05-17 → 2036-05-14)
Root SHA256 (colon form)	6F:96:…:0D:B1 (full string above)
Root SHA256 (step form)	6f962801b825ec33035971f2d8de1c1349ec880ff66332bc68c0d86ae58f0db1
Intermediate CN	RRCHNM Internal Intermediate CA (valid 2026-05-17 → 2028-05-16)
heimdall ACME directory	https://heimdall.rrchnm.internal/acme/acme/directory
Root in repo	ansible/group_vars/all.yaml → ca_root_cert
Intermediate in repo	ansible/group_vars/step_ca_servers.yaml → ca_intermediate_cert

---

1. DNS

The internal zone is served by the CoreDNS resolver trio, not public DNS — so the Mac must be on the campus network / VPN that points at those resolvers (see internal-dns). Verify:

scutil --dns | grep -A3 rrchnm        # is there a resolver scoped to rrchnm.internal?
dig +short heimdall.rrchnm.internal   # heimdall per fleet.yaml (step_ca_servers class)
dig +short kuvasz.rrchnm.internal     # the internal HAProxy VIP (haproxy_internal_vips, VRRP-floated)

If these NXDOMAIN / time out, it’s a network/VPN problem, not a cert problem — fix DNS first, the TLS error is downstream noise. No /etc/hosts hacks; if a name doesn’t resolve it usually needs a dns_zones entry on the resolver side (acme-off-fleet-clients.md §4, operator note), not a client workaround.

2. Trust — the macOS split-brain gotcha

macOS has two trust stores and they don’t talk to each other:

• System keychain — used by Safari, Chrome, and Apple frameworks (Secure Transport).
• OpenSSL/LibreSSL file bundle (/etc/ssl/cert.pem) — used by the command-line curl, openssl, and most Homebrew CLI tools.

Installing the root into the keychain fixes browsers only. A curl whose verbose output shows * CAfile: /etc/ssl/cert.pem is on the OpenSSL path and ignores the keychain entirely — you must give it the cert as a file. This is the #1 source of “I installed it but curl still fails.”

Browsers (keychain)

sudo security add-trusted-cert -d -r trustRoot \
  -k /Library/Keychains/System.keychain ~/rrchnm/root-ca.crt
# verify present + correct fingerprint:
security find-certificate -a -c "RRCHNM Internal Root CA" -p \
  /Library/Keychains/System.keychain | openssl x509 -noout -fingerprint -sha256
# remove later:
sudo security delete-certificate -c "RRCHNM Internal Root CA" /Library/Keychains/System.keychain

• Safari / Chrome — pick it up from the keychain automatically.
• Firefox — ships its own NSS store; ignores the keychain. Either set security.enterprise_roots.enabled = true in about:config (makes Firefox read the system store), or import root-ca.crt under Settings → Privacy & Security → Certificates → View Certificates → Authorities → Import.

CLI tools (file bundle)

Don’t edit /etc/ssl/cert.pem directly — OS updates clobber it. Build a combined bundle and point the standard env vars at it (already in the TL;DR):

cat /etc/ssl/cert.pem ~/rrchnm/root-ca.crt > ~/rrchnm/ca-bundle.pem
export SSL_CERT_FILE="$HOME/rrchnm/ca-bundle.pem"   # openssl + most tools
export CURL_CA_BUNDLE="$HOME/rrchnm/ca-bundle.pem"  # curl

Per-command override without the env vars: curl --cacert ~/rrchnm/root-ca.crt ….

Root alone is enough — when the server serves its chain. Caddy serves the full ACME chain (leaf + RRCHNM Internal Intermediate CA), so trusting the root lets the client build leaf → intermediate → root. You only need to add the intermediate to your bundle if a server is misconfigured to serve the leaf alone.

Python / Node don’t use either store. Python requests/httpx use certifi; point them with REQUESTS_CA_BUNDLE / SSL_CERT_FILE or certifi’s cafile. Node uses NODE_EXTRA_CA_CERTS=~/rrchnm/root-ca.crt.

Optional: the step CLI

Lets you fetch the root straight from heimdall (pinned by fingerprint, no trust-on-first-use) and inspect issued certs:

brew install step
step ca bootstrap --ca-url https://heimdall.rrchnm.internal \
  --fingerprint 6f962801b825ec33035971f2d8de1c1349ec880ff66332bc68c0d86ae58f0db1

---

Diagnostic checklist — curl: (60) unable to get local issuer certificate

Walk in order; each step rules out a layer.

1. Is it DNS? dig +short <name>.rrchnm.internal returns IPs? If not → §1.

2. Is it the keychain-vs-file split? Try curl --cacert ~/rrchnm/root-ca.crt -I https://<name>.rrchnm.internal.

   • Works → your earlier keychain install just isn’t seen by this curl. Set up the file bundle (§2 “CLI tools”). Browsers are already fine.
   • Still fails → go to step 3; it’s not a client-trust problem.

3. What chain is the server actually serving?

   echo | openssl s_client -connect <name>.rrchnm.internal:443 \
     -servername <name>.rrchnm.internal -showcerts 2>/dev/null \
     | grep -E '^ *[0-9]+ s:|^ *i:|^   i:'

   Read the issuer (i:) of cert 0 (the leaf):

   Leaf issuer	Meaning	Fix
   CN=RRCHNM Internal Intermediate CA, with a cert 1	Good heimdall chain	Pure client trust — §2
   CN=RRCHNM Internal Intermediate CA, no cert 1	Real cert, intermediate not served	Add intermediate to your bundle as a stopgap; fix server to serve full chain
   CN=Caddy Local Authority …	Not a heimdall cert — the server fell back to Caddy’s self-signed tls internal; ACME issuance never happened	Server-side, not your Mac. No client trust will ever fix it

Worked example: kuvasz serving issuer: "local" (2026-06-17)

echo | openssl s_client -connect kuvasz.rrchnm.internal:443 \
  -servername kuvasz.rrchnm.internal -showcerts 2>/dev/null \
  | grep -E '^ *[0-9]+ s:|^ *i:|^   i:'

s_client showed the leaf issued by Caddy Local Authority, and docker compose logs caddy on the host showed certificate obtained successfully {"issuer": "local"} with zero ACME activity. Root cause: the cert_authority: heimdall switch (host_vars/armin.yaml) was committed but the stack hadn’t been redeployed, so the container was still in selfsigned mode (TLS_MODE=tls_selfsigned). Fix was entirely server-side — redeploy from the toolbox container: make deploy-docker-compose FQDN=kuvasz.rrchnm.internal, which flips TLS_MODE=tls_heimdall and runs caddy-verify-cert. The lesson: a Caddy Local Authority issuer is a deploy/issuance problem on the host, never a trust problem on the Mac.

---

Renewals — nothing to do

heimdall leaf certs are short-lived (24h defaultTLSCertDuration); Caddy auto-renews on its ~2/3-lifetime schedule, transparently, as long as the ACME HTTP-01 path stays reachable server-side. The root you installed is valid until 2036 — re-run §2 only on a root rotation (sop-ca-bundle-rotation.md (operator note)), which is a deliberate, announced event, not something teardown/make step-ca triggers.

---

Last updated: 2026-06-17. Written from the kuvasz off-fleet-trust debug session — companion to acme-off-fleet-clients.md (operator note) (server side).