zot (thoth)

Operational reference for the thoth Incus container on atomsk (IP in fleet.yaml). Three systems share the one Zot daemon:

Pull-through OCI registry — Zot serving cluster-bootstrap image pulls (Cilium, cert-manager, ESO, OpenBao, etc.) with a warm cache that survives k0s-down/k0s-up cycles.
Release-watcher — operator-run reconcile that mirrors a curated list of upstream binary releases (k0s, helm, kubectl, oras, gateway-api-experimental, local-path-provisioner, actions-runner) into the same Zot as OCI artifacts, so consumers pull binaries from the LAN instead of github.com.
Native push target — in-fleet CI pushes images to thoth directly (e.g. the Forgejo Actions build of rrchnm-systems/docs pushes oci.rrchnm.internal/rrchnm-systems/docs). Zot currently accepts anonymous LAN pushes; adding push auth is an open hardening item (operator todo). Digest-pinned deploys are immune to tag overwrites either way.

All three share the same TLS cert, the same setup playbook (setup-oci-mirrors.yaml), and the same DNS names: oci.rrchnm.internal for image pulls, thoth.rrchnm.internal for the watcher’s push path (zot.rrchnm.internal is a legacy CNAME to the same host).

Trust model — read this first

The watcher verifies upstream signatures at fetch time (PGP / sha256 / cosign per the artifact’s upstream_verify.kind), then pushes the verified bytes to thoth via plain oras push — no re-signing. Consumers pulling from thoth trust the LAN path (same posture as apt-cacher-ng, dockerd’s registry-mirrors, or the in-cluster image pulls that already go through Zot’s pull-through cache).

So the trust chain is: upstream signing key → watcher verifies → over-the-LAN handoff to thoth → over-the-LAN handoff to consumer. Compromise of thoth or its LAN means everything downstream is compromised — but that’s already the threat model for the in-cluster image pulls and apt-cacher-ng, so it’s not a new exposure.

Drift detection is make mirror-oci-artifacts FORCE=true: re-fetches every mirrored version from upstream, re-verifies upstream sigs, and overwrites in thoth. Slow (full upstream re-fetch) but thorough.

Architecture overview

campus LAN (10.112.x.x)
│
├── k0s workers (containerd) ──┐
│                              │  image pulls →
├── docker fleet (dockerd) ────┤  https://oci.rrchnm.internal
│                              │  (containerd hosts.toml /
├── in-fleet CI (eitri) ───────┤   dockerd registry-mirrors)
│     image pushes →           │
│     oci.rrchnm.internal      │
│                              │
└── k0s nodes (consumer-side ──┤  binary pulls (oras pull when
    binary fetch, opt-in)      │  use_thoth_artifacts=true) →
                               │  https://thoth.rrchnm.internal
                               ▼
                   thoth (Incus container, on atomsk)
                   ├── zot.service :443
                   │   ├── pull-through cache (sync from the upstream
                   │   │   registries in group_vars/oci_mirrors.yaml)
                   │   ├── OCI artifact store (release-watcher pushes)
                   │   └── natively-pushed images (in-fleet CI)
                   └── step-cert-renewer.timer (refreshes the TLS cert)

operator workspace ── make mirror-oci-artifacts ──► SSHes to thoth, runs the
                      watcher play (hosts: thoth):
                        gh API → list upstream versions
                        diff against `oras repo tags`
                        for each new version:
                          curl upstream asset
                          verify upstream signature
                            (PGP / sha256 / cosign)
                          oras push (verified bytes, no re-signing)

There is no on-host cron. The prior on-thoth systemd timer was dropped in favor of workspace-only, operator-on-demand runs (pending future Ansible control servers). The only timer on thoth is the TLS cert renewer.

The roles in one daemon

Zot is general-purpose OCI distribution; it doesn’t know or care how content arrived. Everything ends up as OCI manifests + blobs under /var/lib/zot (a dedicated zot-cache Incus volume on atomsk’s instance pool):

Role	Populated by	Pulled by	Repo namespace
Pull-through cache	Zot’s sync extension on first-touch	k0s containerd / docker dockerd via mirror config	mirrors upstream paths (`quay.io/cilium/cilium` → `oci.rrchnm.internal/cilium/cilium`)
Release-watcher artifacts	`oras push` from the watcher run	k0s nodes via `oras pull` (when `use_thoth_artifacts=true`)	per artifact’s `zot_repo` in `vars/oci-artifact-mirror.yaml` (e.g. `github/k0sproject/k0s`, `chnm/systems/bin/kubectl`)
Native images	in-fleet CI `docker push`	Argo/containerd via digest pin	build-chosen (e.g. `rrchnm-systems/docs`)

Same daemon, same cert, same disk. The namespaces don’t overlap.

Trust + secret custody

Verification at fetch time, not at consume time

Per-artifact upstream_verify.kind in vars/oci-artifact-mirror.yaml:

kind	Verification	Currently used by
`pgp`	Fetch `<asset>.asc`, `gpg --verify` against a pinned keyring (`oci-mirrors/keyrings/<name>.gpg`)	oras
`cosign_pinned`	Fetch `<asset>.sig`, `cosign verify-blob --key` against a pinned PEM pubkey (`oci-mirrors/cosign-pubkeys/<name>.pub`)	k0s
`cosign_keyless`	`cosign verify-blob` against Sigstore Fulcio with cert-identity assertion	(available; nothing currently)
`sha256_url`	Fetch a per-asset `.sha256` URL, compare	helm, kubectl
`sha256_sums`	Fetch the release’s checksums asset, find our line, compare	(available)
`sha256_pinned`	Look up the version in an in-repo `sha256_by_version` map	gateway-api-experimental, local-path-provisioner, actions-runner

kind: none is not allowed — every artifact must have some upstream verification path. Past the gate, bytes are pushed as-is; consumers trust the LAN.

What’s on disk

Material	Where (source)	Where (thoth)	Custody
GitHub PAT	`github_token_read_releases` in `group_vars/all.sops.yaml` (SOPS; part of the `github_token_*` family)	`/etc/oci-mirror/github.token` (0400 root)	Operator-minted at github.com; read-only public-repo scope. Materialized by `setup-oci-mirrors.yaml --tags watcher`.
PGP keyrings	`playbooks/oci-mirrors/keyrings/<name>.gpg` (committed; public keys)	`/etc/oci-mirror/keyrings/` (0444)	Manual fetch/rotation per `keyrings/README.md`.
cosign pubkeys	`playbooks/oci-mirrors/cosign-pubkeys/<name>.pub` (committed; public keys)	`/etc/oci-mirror/cosign-pubkeys/` (0444)	Manual per `cosign-pubkeys/README.md`.
Per-version sha256 pins	`sha256_by_version` maps in `vars/oci-artifact-mirror.yaml`	—	Operator hand-pins; auditable via `git blame`.

Why no signing on our end

Earlier iterations cosign-signed every mirrored blob with a fleet-owned key, and consumers verified. Dropped because:

The signature only proves “the watcher signed this” — upstream authenticity is already covered by the verify-at-fetch gate.
Both compromise vectors (thoth, the watcher) would also possess the signing key. The signature didn’t defend against the threats that matter.
Every key rotation required a full re-sign sweep. Tedious + a new failure mode.
Consumers already trust the LAN path for image pulls; a separate posture for binaries from the same host was inconsistent.

The model is symmetric with the pull-through cache and apt-cacher-ng: verify-at-fetch, trust-the-LAN-on-pull.

On-host layout (thoth)

/var/lib/zot/                      ← Zot blob + manifest store (zot-cache volume)
/etc/zot/
├── config.json                    ← rendered from roles/zot/templates/zot-config.json.j2
└── tls/{cert,key}.pem             ← step CLI-issued + step-cert-renewer.timer

/etc/oci-mirror/
├── github.token                   ← GitHub PAT, 0400 root
├── keyrings/<name>.gpg            ← PGP keyrings, 0444
└── cosign-pubkeys/<name>.pub      ← cosign pubkeys, 0444

/var/tmp/oci-mirror/               ← watcher work dir (downloads; cleaned after push)
/var/run/oci-mirror.lock           ← flock; prevents concurrent runs

/etc/systemd/system/
├── zot.service
├── step-cert-renewer.service
└── step-cert-renewer.timer

Tools installed by --tags watcher: oras (push + pulls), cosign (verify-only — cosign_pinned/cosign_keyless paths), gnupg (PGP verify). No Ansible runs on thoth — the watcher play executes from the operator workspace over SSH.

In-repo layout

Everything lives under vibes/infra/ansible/ in chnm/systems:

Path	Purpose
`playbooks/setup-oci-mirrors.yaml`	Bring-up + maintenance. Tags: `init` (baseline) · `snapshot` (Incus rollback point) · `zot` / `step` / `renewer` (the `zot` role: daemon, config, cert lifecycle) · `watcher` (tools, dirs, PAT, keyrings, pubkeys).
`roles/zot/`	Zot binary + config + systemd units + step cert lifecycle (templates: `zot-config.json.j2`, `zot.service.j2`, `step-cert-renewer.*`).
`playbooks/mirror-oci-artifacts.yaml`	The release-watcher. `hosts: thoth`; tags `plan` (diff only) / `push` (default).
`playbooks/oci-mirrors/tasks/mirror-one-{artifact,version}.yaml`	Per-artifact reconcile; per-version download → verify → push.
`playbooks/oci-mirrors/tasks/fetch-via-oras.yaml`	Consumer-side helper (used by `setup-cluster.yaml` when `use_thoth_artifacts=true`).
`playbooks/oci-mirrors/keyrings/` + `cosign-pubkeys/`	Committed public trust material + READMEs with the add/rotate workflows.
`playbooks/oci-mirrors/templates/oci-mirror-metrics.prom.j2`	Prom textfile metric template (see Observability).
`vars/oci-artifact-mirror.yaml`	Single source of truth for what gets mirrored + how each is verified.
`group_vars/oci_mirrors.yaml`	Zot version/config knobs, upstream registry list, oras/cosign versions, path defaults.
`host_vars/thoth.yaml`	Plain stub — no host secrets (the PAT is in `group_vars/all.sops.yaml`).
`mk/fleet.mk` targets	`oci-mirrors` (bring-up), `mirror-oci-artifacts` (`ONLY=` `TAGS=plan\|push` `FORCE=true`).

First-time bring-up

From a clean state:

# 1. Provision thoth + Zot + cert lifecycle (needs step-ca on heimdall up first).
make oci-mirrors

# 2. Set the GitHub PAT: add/update `github_token_read_releases: ghp_xxx`
#    in ansible/group_vars/all.sops.yaml (sops edit), commit.

# 3. Trust material for the curated artifacts (check vars/oci-artifact-mirror.yaml):
#    - k0s (cosign_pinned): cosign-pubkeys/k0s.pub — per cosign-pubkeys/README.md
#    - oras (pgp): keyrings/oras.gpg — per keyrings/README.md
#    Both are committed public keys; cross-check fingerprints against upstream docs.

# 4. Re-run to materialize PAT + keyrings + pubkeys onto thoth.
ansible-playbook ansible/playbooks/setup-oci-mirrors.yaml --tags watcher

# 5. First reconcile (from the workspace — it SSHes to thoth).
make mirror-oci-artifacts

sha256_pinned artifacts error loudly until their sha256_by_version maps have entries — compute the hash out-of-band, paste into vars/oci-artifact-mirror.yaml, commit, re-run. Every new upstream release of a pinned artifact needs the same hand-pin (the cost of the no-unverified policy).

Smoke tests

Run after bring-up or a host reflash; bail at the first failure to localize.

Web UI — https://oci.rrchnm.internal/ renders Zot’s browse UI. TLS warning → the heimdall root isn’t in your trust store (see the macOS access guide).
Anonymous Distribution API — curl -sSi https://oci.rrchnm.internal/v2/ → HTTP/2 200; /v2/_catalog lists repos. 401 → anonymous policy broken; timeout → systemctl status zot / nftables :443.
Pull-through — docker pull oci.rrchnm.internal/docker.io/library/alpine:3.20 twice (second is cache-fast). x509 error → daemon trust store; hang → thoth’s egress to the upstream registry.

Day-to-day operation

Runs are operator-on-demand from the workspace; output lands in your terminal, not a journal on thoth.

Action	Command
Plan-only reconcile (no push)	`make mirror-oci-artifacts TAGS=plan`
Reconcile everything	`make mirror-oci-artifacts`
One artifact only	`make mirror-oci-artifacts ONLY=k0s`
Drift check / re-verify all	`make mirror-oci-artifacts FORCE=true`
List curated artifacts	read `vars/oci-artifact-mirror.yaml`
List mirrored versions	`oras repo tags thoth.rrchnm.internal/github/k0sproject/k0s`
Re-render Zot config	`ansible-playbook playbooks/setup-oci-mirrors.yaml --tags zot`
Re-materialize secrets/keys	`... --tags watcher`

Rotation runbooks

GitHub PAT

When: expiry (~90 days), suspected leak, scope change.

Mint the new token (read-only public-repo scope).
sops edit ansible/group_vars/all.sops.yaml → update github_token_read_releases. Commit.
ansible-playbook playbooks/setup-oci-mirrors.yaml --tags watcher — rewrites /etc/oci-mirror/github.token on thoth.
Revoke the old token.

PGP keyring / cosign pubkey

When: upstream rotates their release-signing key (rare; announced upstream). Follow the workflow in playbooks/oci-mirrors/keyrings/README.md (PGP) or cosign-pubkeys/README.md (cosign) — fetch/export the new key, cross-check the fingerprint against a recent signature (never trust the keyserver alone), commit, then --tags watcher to materialize. Already-mirrored versions don’t need re-verification (verify-at-fetch model); if upstream rotated a while ago, recent watcher runs may have failed verification — re-run and watch.

Adding a new artifact

Append one block to vars/oci-artifact-mirror.yaml; the next run picks it up. The file’s header documents the source: kinds (github release assets, github_raw files-at-tag, url_template for non-GitHub) and the upstream_verify kinds (see the table above — pick the strongest thing upstream actually publishes; sha256_pinned is the fallback and commits you to hand-pinning each release). Placeholders: __VERSION__ (with v), __VERSION_NOV__ (stripped).

make mirror-oci-artifacts ONLY=<name> TAGS=plan   # confirm the plan
make mirror-oci-artifacts ONLY=<name>             # push

To wire a consumer playbook, follow setup-cluster.yaml’s pattern: a use_thoth_artifacts conditional wrapping include_tasks: fetch-via-oras.yaml.

Consumer side — opting k0s in (and out)

setup-cluster.yaml has three GitHub fetches (the k0s binary, Gateway API CRDs, local-path-provisioner manifest). One flag switches all three:

ansible-playbook ansible/playbooks/k0s/setup-cluster.yaml -e use_thoth_artifacts=true

When true, each fetch routes through fetch-via-oras.yaml (oras pull from thoth); nodes get oras installed as a pre-task. No automatic fallback — a version missing from thoth errors loudly: push it first (make mirror-oci-artifacts ONLY=<artifact>) or run without the flag. Binaries are byte-identical to the GitHub-fetched ones; switching back is just omitting the flag.

Observability

The watcher play emits a prom textfile metric via playbooks/oci-mirrors/templates/oci-mirror-metrics.prom.j2 → /var/lib/node_exporter/textfile_collector/oci-mirror.prom on thoth. An alert on time() - oci_mirror_last_success > <threshold> is the natural hook — sized to however often you actually run reconciles, since there’s no timer.

Troubleshooting

Symptom	Likely cause	Diagnostic
`oras push` TLS error	Zot cert expired / SAN mismatch	`systemctl status step-cert-renewer` on thoth; SAN list in `group_vars/oci_mirrors.yaml`
`gh api releases` 401/403	PAT expired	check `/etc/oci-mirror/github.token`; rotate per runbook above
`gpg --verify` / `cosign verify-blob` fails	missing/stale keyring or pubkey, or upstream rotated	cross-check upstream’s published fingerprint; refresh per README
“sha256 mismatch”	upstream rebuilt the asset, or a bad pin	`git log` the pin; re-download + recompute yourself
consumer `oras pull` times out	thoth or atomsk down	`incus list atomsk:thoth`; `systemctl status zot` on thoth
consumer “manifest unknown”	version not mirrored yet	`make mirror-oci-artifacts ONLY=<artifact>`
interleaved runs	flock failed	check `/var/run/oci-mirror.lock` creation

internal-dns — the oci.rrchnm.internal / thoth.rrchnm.internal names.
macos-internal-access — trusting the heimdall CA that signs thoth’s cert.
internal-cert-authority.md (operator note, not published) — the CA design itself.
Operator changelog — 2026-05-24 in-cluster Zot removal; 2026-07-01 architecture rationale; the on-host-cron retirement.