Skip to content
RRCHNM Infra

Alloy bare-metal rollout

Priority: mid. Bare-metal VMs are currently invisible to the observability stack — Loki only sees k0s pod logs via the in-cluster Alloy DaemonSet, so a caddy/coraza/lamp/dns/backup incident has no centralized log path. Hygiene-with-teeth: not actively breaking, but turns “what happened on host X at 03:00?” from a multi-host SSH+grep into a single Grafana query.

Depends on: none (Loki + the in-cluster Alloy DaemonSet are already running per k0s/observability/alloy.yaml).

Blocks: the caddy fleet rebuild (CHANGELOG 2026-07-01) — that plan’s Coraza / WAF audit log pipeline assumes systemd-journal → Alloy → Loki on the caddies. The rest of the caddy rebuild can proceed without it (logs land in journald locally), but the WAF observability promise is empty until this lands.

Background: AGENTS.md chat 2026-05-25 (Coraza WAF / ratelimit observability scoping discovered that Promtail was an incorrect assumption — the stack is Alloy, and it’s k0s-DaemonSet-only).

Current state (2026-05-25)

In-cluster (k0s): grafana/alloy:v1.15.1 runs as a DaemonSet via k0s/observability/alloy.yaml. Config in the alloy-config ConfigMap. Scrapes pod logs + scrapes Alloy’s own Prometheus metrics. Ships to in-cluster Loki.

Bare-metal: nothing. Every VM outside k0s_cluster writes logs to its local systemd journal and that’s where they stop. No log shipping role exists in ansible/roles/.

Affected fleets (all currently log-isolated):

GroupHosts
caddy_proxieskyogre, groudon, articuno, zapdos, moltres
caddy_kv_storesrayquaza (pending — see CHANGELOG 2026-07-01)
docker_serverseren, mikasa, armin
lamp_backendsomeka-s-recroom, omeka-classic-recroom, drupal-recroom, wordpress-recroom
package_proxieslugia, ho-oh
dns_resolversmesprit, azelf, uxie
dns_authoritativesarceus
step_ca_serversheimdall
backup_serversbackup2
standalone (not grouped)logi

Proposed change

New role ansible/roles/alloy_agent/ — Alloy installed as a systemd service, single config file /etc/alloy/config.alloy rendered per-host, scrapes the local systemd journal and remote-writes to the in-cluster Loki endpoint.

Universal application via init role: the alloy_agent role is imported at the end of ansible/roles/init/tasks/main.yaml, so every fleet that includes the init+nftables universal baseline (= every setup-*.yaml playbook) gets log shipping for free. No per-fleet wiring needed. Toggle is one var in group_vars/all.yaml (alloy_agent_enabled: false default). Flipping to true activates everywhere on the next setup run.

Role shape:

ansible/roles/alloy_agent/
├── defaults/main.yaml          # alloy version pin, loki endpoint, journal filters
├── handlers/main.yaml          # restart alloy
├── tasks/main.yaml             # install + config + enable (gated on alloy_agent_enabled)
└── templates/
    └── config.alloy.j2         # journal source + loki sink

Config sketch:

loki.write "default" {
  endpoint {
    url = "http://loki.observability.svc.rrchnm.internal:3100/loki/api/v1/push"
  }
}


loki.source.journal "host" {
  forward_to = [loki.write.default.receiver]
  labels = {
    host  = "{{ inventory_hostname }}",
    site  = "{{ rrchnm_site }}",
    group = "{{ group_names | difference(['all']) | first }}",
  }
}

Apply: import_role: name=alloy_agent lives at the end of roles/init/tasks/main.yaml, so every setup-*.yaml’s universal-baseline play (the one with roles: [init, nftables]) installs Alloy by default. --tags alloy slices the Alloy install on its own. Toggle: alloy_agent_enabled: false in group_vars/all.yaml keeps it dormant; flipping to true activates everywhere on the next setup run. Reversible by flipping back + apt-get purge alloy (or removing the binary at /usr/local/bin/alloy since the role installs from upstream tarball rather than apt).

Impact / what this buys

  • One-place log search across the fleet. Grafana → Loki query by host / group / site instead of ansible all -m shell -a 'journalctl --since=...'.
  • Closes the WAF observability gap. The caddy rebuild (CHANGELOG 2026-07-01) ships Coraza in DetectionOnly mode — the audit JSON is useless until it’s queryable. This is the pipe.
  • Single tool, single config language. Alloy is already in use in-cluster; we don’t pull in Promtail / Vector / fluent-bit as a parallel stack.
  • Unblocks future structured-log dashboards. Per-fleet panels (lamp PHP error rates, caddy 5xx, borg run outcomes) become trivial Loki queries once the source is there.

Non-goals / what this does NOT buy

  • Metrics scraping from bare-metal hosts. Alloy CAN scrape Prometheus exporters too, but adding node_exporter + scrape jobs is a separate todo. This one is journal → Loki only.
  • Application-level log enrichment. Apps that write to files (not journald) — e.g. Caddy’s access logs if configured to file — need separate loki.source.file blocks. Day-1 just covers journald.
  • Centralized log retention policy. Loki retention is whatever the in-cluster Loki is already configured for; this todo doesn’t tune it.

Risk

  • Blast radius: per-host. A bad Alloy config breaks log shipping on that host but doesn’t affect the app workload. The role uses validate: on the config template + state: started only after validation passes.
  • Reversibility: one-command — apt-get purge alloy && systemctl daemon-reload. No state to clean up beyond /etc/alloy/.
  • Apply window: any time. The Loki endpoint already exists; adding senders has no impact on existing in-cluster log flow.
  • Egress firewall: each non-k8s host’s nftables needs an output rule allowing :3100 to the Loki service IP (or the gateway IP fronting it). Add to roles/nftables/templates/nftables.conf.j2’s egress section, gated on alloy_agent_enabled. Without it, Alloy starts but silently fails to ship.

Why this priority

Mid: nothing’s broken, but the cost of an incident on a bare-metal host without centralized logs is real and recurring (every postmortem becomes “and we had to ssh to each host to grep journald”). Not top because no active outage depends on it, and the in-cluster log path already covers the k0s workloads which are most of the operational surface.

Bump to top when:

  • A real incident on a bare-metal host costs >30 min of triage to “where are the logs?” — that’s the moment the priority math flips.
  • WAF blocking mode is enabled per the caddy rebuild (CHANGELOG 2026-07-01) Phase 6 — at that point Coraza audit logs become the only signal for tuning and false-positive triage, and they need to be queryable, not journald-local.

Otherwise: ship alongside the caddy rebuild for the caddy_proxies + caddy_kv_stores subset (so the WAF promise is honored). Per-fleet sweeps from the original bands 2 + 3 are no longer needed — the role’s now part of init, so flipping the toggle activates everywhere at once.

Sequencing

Band 1 — role + universal application

  • A1 — Create ansible/roles/alloy_agent/ (defaults, tasks, template, handler). Pin Alloy version (matches in-cluster DaemonSet, v1.15.1). Loki endpoint var lives in alloy_agent_loki_endpoint (role defaults — placeholder pointing at http://loki.rrchnm.internal/loki/api/v1/push until a real reachable URL is wired). Done as part of the caddy-rebuild plumbing pass 2026-05-25 (CHANGELOG 2026-05-26).
  • A2 — Add nftables egress allow for the Loki endpoint, gated on alloy_agent_enabled. Edit ansible/roles/nftables/templates/nftables.conf.j2. Still TODO — role exists but Alloy can’t actually ship until this lands AND the Loki endpoint is exposed.
  • A3 — Wire role universally via import_role: name=alloy_agent at the end of ansible/roles/init/tasks/main.yaml. Every fleet that includes the init+nftables universal baseline gets it automatically. Toggle: alloy_agent_enabled in group_vars/all.yaml (default false). --tags alloy slices the install on its own. Done 2026-05-25 (after initial per-fleet wiring; refactored to universal via init).
  • A4 — Validate: Grafana → Loki query {group="caddy_proxies"} returns events; test one Coraza-style log line per caddy reaches Loki within 5s. Blocked on A2 + Loki bare-metal endpoint.

Per-fleet bands — collapsed

Bands 2 + 3 (one PR per fleet to wire the role in) are obsolete now that A3 lives in the init role. Flipping alloy_agent_enabled: true in group_vars/all.yaml activates every fleet that runs the universal baseline (which is every setup-*.yaml) on its next setup run.

Per-fleet exceptions, if needed, are one-line overrides:

  • Disable per-fleet: alloy_agent_enabled: false in group_vars/<group>.yaml.
  • Disable per-host: alloy_agent_enabled: false in host_vars/<host>.yaml.
  • Per-fleet journal scoping: alloy_agent_journal_units override in group_vars/<group>.yaml (ship only caddy.service, etc.).

Deferred

  • A12 — Prometheus exporter scraping from bare-metal via Alloy. Separate todo when metrics needs arise.
  • A13 — File-source ingestion for apps that write to files instead of journald (Caddy access logs if/when we redirect them to file).

Out-of-scope (flag, don’t fix here)

  • Loki retention tuning. Increasing volume from bare-metal hosts may push retention pressure; if it does, file a separate todo against k0s/observability/.
  • Centralized journald upstream protocol. systemd-journal-upload would be an alternative to Alloy, but Loki doesn’t natively accept journald protocol — would need a sidecar translator. Not worth it; Alloy is already the standard here.
  • Don’t deploy Alloy via the cluster Alloy DaemonSet’s ConfigMap. That’s k0s-scoped and host-network-bound. Bare-metal Alloy is a separate systemd unit with its own config file; don’t try to share ConfigMap state across the boundary.
  • Don’t ship raw /var/log/* files when journald already captures the same data. Double-counts log lines and inflates Loki ingestion cost. Stick to loki.source.journal unless an app bypasses journald entirely.

Open questions

  • Loki endpoint URL from bare-metal. In-cluster Alloy uses http://loki.observability.svc/... (cluster-internal DNS). Bare-metal needs a name that resolves and an IP that routes — either the internal Gateway IP fronting Loki (per ansible/vars/k0s-gateways.yaml) or a dedicated NodePort. Confirm which exists / which we prefer before A1. The Gateway path is cleaner if Loki already has one; otherwise add it as part of A1.
  • Alloy version pin. Match the in-cluster v1.15.1? Or pin independently and let the DaemonSet drift forward? Lean toward matching to keep config syntax compatible; revisit if upstream forces a break.
  • Per-fleet label conventions. First-cut label set in the config sketch above is host + site + group. Confirm group is the first inventory group that isn’t all (matches how the k0s Alloy labels pods). If a host is in multiple meaningful groups (e.g. docker_rootful ⊂ docker_servers), pick the most specific.

Last updated: 2026-05-25