apt-cacher stale metadata

Mitigated 2026-07-13 (migrated 2026-09-05)

The fix landed as commit d38c992c: roles/apt_cacher_ng now purges cached volatile index metadata on every run — and with the broadened pattern (InRelease/Release/Packages/Sources/by-hash), not just the narrow one described below. The stronger server-side always-revalidate config (hardening option 1) remains open. Kept for the diagnostic signature: an apt sqv “Message has been manipulated” error against one suite usually means stale cacher metadata, and the purge one-liner below is the 30-second fix.

Status: MITIGATED 2026-07-13 — roles/apt_cacher_ng/tasks/main.yaml now purges the volatile index metadata on every run (option 1b below), so make up starts each cacher with fresh, consistent indices. A server-side config rule (option 1a) is still open as a “stronger / always-fresh” alternative if the purge proves insufficient. · Filed: 2026-07-13 · Severity: medium

What happened

During a make up rebuild, the oci-mirrors stage failed on thoth with the ansible apt module’s Failed to update apt cache after 5 retries (empty detail). The real error, from incus exec atomsk:thoth -- apt-get update:

Err:3 http://deb.debian.org/debian trixie-updates InRelease
  Sub-process /usr/bin/sqv returned an error code (1):
  Verifying signature: Message has been manipulated

Only trixie-updates failed; trixie, trixie-security, and smallstep verified fine. apt-cacher-ng (ho-oh@hyperion / lugia@kyojin) had cached an InRelease for trixie-updates whose referenced by-hash indices had since moved, so its signature no longer matched — Debian trixie’s new sqv (sequoia) verifier rejects it as “manipulated”.

Likely trigger: thoth is kept across make down (oci_mirrors ∈ DOWN_KEEP), so it re-ran apt update against a cacher whose trixie-updates metadata was stale/inconsistent, while the freshly-rebuilt hosts (garage, etc.) happened to populate/verify clean.

Immediate fix (applied, manual)

Delete the stale release metadata on both cachers (keeps .deb packages), then re-fetch:

incus exec hyperion:ho-oh -- find /var/cache/apt-cacher-ng -path '*trixie-updates*' \( -name InRelease -o -name Release -o -name 'Release.gpg' \) -delete
incus exec kyojin:lugia   -- find /var/cache/apt-cacher-ng -path '*trixie-updates*' \( -name InRelease -o -name Release -o -name 'Release.gpg' \) -delete
incus exec atomsk:thoth   -- apt-get update      # verifies clean

If by-hash indices are also stale, broaden to .*/(InRelease|Release|Release\.gpg|Packages.*|.*by-hash.*)$ and systemctl restart apt-cacher-ng.

Hardening options (pick one; roles/apt_cacher_ng)

1. Server-side revalidation (preferred). Make apt-cacher-ng always revalidate volatile index files against upstream so a stale InRelease/by-hash can’t be served. In templates/acng.conf.j2: verify/tune VfilePattern/PfilePattern cover InRelease|Release|Packages|by-hash, and ensure the expiration maintenance job actually runs (the ReportPage expire action / a cron) rather than relying on ExTreshold: 30 passive drop. Consider a much shorter TTL for the metadata files specifically.
2. Client-side belt-and-suspenders. In the init apt-update task, add a rescue: that, on a sqv/“Message has been manipulated”/signature-verification failure, clears the cacher’s stale metadata (or retries with -o Acquire::http::No-Cache=true) and retries — so a wedged cache degrades gracefully instead of halting the whole make up.
3. Ops runbook. At minimum, document the one-liner above in the apt-cacher healthcheck / runbook so the next occurrence is a 30-second fix, not a diagnosis.

References

• Role: ansible/roles/apt_cacher_ng/ (templates/acng.conf.j2, defaults → acng_expire_threshold_days).
• Apt-cacher hosts: ho-oh (hyperion), lugia (kyojin) — fleet.yaml.
• Related: Debian trixie switched apt signature verification to sqv (sequoia), which is stricter about InRelease/by-hash consistency than the old gpgv path.