Skip to content
RRCHNM Infra

Release-watcher re-hosting

Status: planned, blocked on ansible control server provisioning. Date 2026-05-24.

The OCI release-mirror watcher (ansible/playbooks/mirror-oci-artifacts.yaml) used to run as a daily systemd timer on thoth itself. That execution path was removed in commit 6c2096d (infra: drop on-thoth oci-mirror- watcher cron — operator-on-demand only); the watcher now runs only when an operator invokes make mirror-oci-artifacts from the workspace.

Reason: the on-thoth path was a workaround for not having a persistent ansible control plane. The ephemeral workspace container can’t host cron, so we apt-installed ansible-core on thoth and shipped a self- contained playbook tree to /etc/oci-mirror/playbooks/. That works but adds 30 MB of ansible runtime per scheduled-task target and forces a SOPS-decrypt-on-workspace → materialize-plaintext-on-target pattern because the on-target ansible can’t read SOPS files.

When a real ansible control server lands, the watcher should move there. This doc captures what to bring back, what to simplify, and what to verify.

What the prior on-thoth path did

Provisioned by setup-oci-mirrors.yaml Play 4 (“Release-watcher cron”, deleted in 6c2096d):

  1. apt install ansible-core on thoth (~30 MB).
  2. Created on-host dirs: /etc/oci-mirror/{playbooks,vars,group_vars, oci-mirrors/{tasks,templates}}.
  3. Copied the curated playbook subset to those dirs (no host_vars, no ansible.cfg, no SOPS dependency at runtime).
  4. Rendered + enabled oci-mirror-watcher.{service,timer} — OnCalendar=daily, RandomizedDelaySec=30min. Service ran ansible-playbook -i 'thoth,' -c local --tags push /etc/oci-mirror/playbooks/mirror-oci-artifacts.yaml.

The watcher itself (mirror-oci-artifacts.yaml) is unchanged by this removal — it still works for hosts: thoth invocations driven from the workspace, which is the only execution path today.

Operator-side state still in place on thoth (Play 3 / --tags watcher provisions these):

  • /etc/oci-mirror/github.token (0400 root, the SOPS-encrypted PAT from host_vars/thoth.sops.yaml materialized to disk)
  • /etc/oci-mirror/keyrings/oras.gpg (committed PGP keyrings)
  • /etc/oci-mirror/cosign-pubkeys/k0s.pub (committed cosign pubkeys)
  • /var/lib/node_exporter/textfile_collector/ (parent dir for the reconcile metrics file)

These all still get used when the watcher runs from the workspace via SSH (gpg + cosign + oras commands execute on thoth as the hosts: thoth target). They should keep getting provisioned — neither path makes them moot.

When this becomes blocked-no-longer

Once there’s at least one persistent ansible control server with:

  • A git checkout of this repo (or a vendored playbook tree).
  • The operator’s SOPS age key available so community.sops can decrypt host_vars at run time.
  • SSH access to thoth via the same operator-side automation key already used from the workspace.
  • A cron/systemd-timer scheduler (OnCalendar=daily + randomized delay), or a higher-level “scheduled-task” abstraction if the control server runs something like AWX / a scheduler service.

…then the watcher (and other scheduled plays as they emerge) can land there.

1. Cron unit on the control server

Daily timer. The unit invokes the workspace-equivalent command:

ANSIBLE_CONFIG=ansible/ansible.cfg \
SOPS_AGE_KEY_FILE=<operator age key> \
ansible-playbook -i ansible/inventory.yaml \
                 --tags push \
                 ansible/playbooks/mirror-oci-artifacts.yaml

The control server SSHes to thoth, all gpg/cosign/oras commands run there, the oras push is over the network (not loopback) but small extra latency on push. Pulls FROM upstream are network either way.

Use OnCalendar=daily + RandomizedDelaySec=30min to match the prior cadence. Lock file handling is already in mirror-oci-artifacts.yaml (flock -n /var/run/oci-mirror.lock) so overlapping fires no-op.

2. What to simplify vs. keep

Once the control server has SOPS, the PAT can come straight from the auto-decrypted oci_mirror_github_pat host_vars variable instead of being materialized to a 0400 file on thoth + slurped back via SSH. Two related cleanups in mirror-oci-artifacts.yaml:

  • Remove the slurp + set_fact: github_token pattern (currently in the play’s pre_tasks — was a workaround for the on-thoth path that couldn’t read SOPS). Use oci_mirror_github_pat directly in oci-mirrors/tasks/mirror-one-artifact.yaml’s Authorization: "Bearer ..." headers.
  • Remove the “Materialize GitHub PAT” task from Play 3 of setup-oci-mirrors.yaml + delete /etc/oci-mirror/github.token from thoth + drop oci_mirror_github_token_path from group_vars/oci_mirrors.yaml. The PAT becomes SOPS-only.

Keep:

  • Play 3’s keyring + cosign-pubkey staging onto thoth — gpg / cosign verify commands run on thoth as hosts: thoth, they need the files locally.
  • become: true on the watcher play — still needed for the metrics textfile write to /var/lib/node_exporter/....
  • The whole verify dispatch (mirror-one-version.yaml) — bytes flow upstream → thoth, signatures verified along the way; nothing about that depends on where the control plane lives.

3. Observability hand-off

The reconcile already writes Prometheus textfile metrics to /var/lib/node_exporter/textfile_collector/oci-mirror.prom on thoth. Once node_exporter is wired into the observability stack (it isn’t yet, as of 2026-05), Grafana will scrape these. Don’t move the textfile path to the control server — keep it on thoth so it co-locates with the registry it describes.

The systemd unit on the control server should have OnFailure=<some-alerting-handler> once the alerting topology exists. Today the watcher exit-code-non-zero just means the next operator make mirror-oci-artifacts will surface the same error.

4. Migration order

When ready to flip:

  1. Stand up the control server, repo checkout, SOPS access, SSH access to thoth.
  2. Add playbooks/control-server/setup.yaml or equivalent that renders + enables the systemd timer for the watcher on the control server. Don’t put this on the oci_mirrors group — it belongs in a control-server inventory group.
  3. Verify the control-server’s first reconcile produces the same results as a workspace-driven run (no-ops if nothing new upstream).
  4. Optionally apply the PAT-simplification cleanup (§2) in the same PR — operator can decide.

Trade-offs to weigh at re-implementation time

  • HA control servers. One control server = single point of failure for scheduled work. Two control servers = need either a scheduler aware of replication (so the same job doesn’t fire on both) or accept dual fires + rely on the watcher’s flock. Probably the cheaper path is one control server + restore-from-backup procedure for cold-recovery.
  • Blast radius. The control server has SSH access to thoth AND whatever else gets scheduled there over time. Constrain via per-target SSH keys (one key for the watcher → thoth root, separate keys for any other scheduled play → its targets).
  • Cron co-location with workload. The watcher specifically wants to run “close to” thoth (the only thing it pushes to). Other scheduled work might want to run close to other targets. If those start to diverge, consider per-site control servers (one in AQ114, one in RH461) rather than a single fleet-wide one. Lower blast radius per host, higher operational complexity.

Pointers

  • Commit that removed the on-thoth path: 6c2096d (infra: drop on-thoth oci-mirror-watcher cron).
  • Commit history of the watcher itself: a174339 (the foundational bugfix run), f944ccf (dead cosign-signing scaffolding removal).
  • Watcher playbook: ansible/playbooks/mirror-oci-artifacts.yaml.
  • Per-version verify dispatch: ansible/playbooks/oci-mirrors/ tasks/mirror-one-version.yaml.
  • Setup prereqs play (still needed): Play 3 of ansible/playbooks/ setup-oci-mirrors.yaml.
  • Trust material READMEs: ansible/playbooks/oci-mirrors/ keyrings/README.md + ansible/playbooks/oci-mirrors/ cosign-pubkeys/README.md.