Internal SMTP server (hermes)

A self-contained internal Postfix relay on the fleet that splits two streams of mail at the recipient: anything addressed to *@rrchnm.internal (system notifications, cron output, monitoring alerts) is delivered locally to a Maildir on hermes itself; anything addressed elsewhere (real human inboxes, off-site mailing lists) is forwarded out via the existing external SMTP relay. Removes the dependency on the external relay for the noisy-internal-notification stream while keeping it on the path for the small fraction of mail that actually needs to leave the building.

Status (2026-05-17)

Planning. Hermes is a comment-only IP/MAC reservation in inventory today (.72 on AQ114, planned postfix server, public IPv4 mapping pre-allocated to 129.174.134.64:25). No TF resource, no Ansible role, no host_vars yet. This doc captures the design decision tree from the chat and the phased path to standing it up.

Component	Reality	Plan reference
Hermes host	Comment-only reservation in `ansible/inventory.yaml` (`.72` AQ114, `129.174.134.64:25` public IP slot)	Phase 1 — provision
Local-mail delivery	n/a	Phase 2 — Postfix + Maildir for `sysadmin@rrchnm.internal`
External-relay forwarding	The whole-fleet behavior today goes through the external relay unconditionally	Phase 2 — split via `transport_maps`
TUI access	n/a	Phase 3 — NeoMutt on hermes (ssh-in workflow)
IMAP access	n/a	Phase 4 (optional) — Dovecot for IMAP from workstation
Retention policy	n/a	Phase 2 — `find + cron` on the Maildir

Why we want this

Most fleet mail is noise that never needs to leave the building. Cron output, k0s/Tetragon alerts, backup-job summaries, apt-cron security notices — by volume, the vast majority of mail generated on the fleet is system notifications targeted at the sysadmin. Today every one of those messages takes a round trip to an external SMTP provider, gets relayed back to the sysadmin’s work inbox, and sits there mixed in with normal email.
External-relay dependency for purely internal traffic is fragile. If the external relay throttles, blocks, or goes down, the whole notification stream stalls — even for messages that conceptually never had to leave the LAN. Splitting at the recipient cap means a relay outage only affects the small “outbound to public recipients” stream.
Sysadmin notifications deserve their own retention/searchability. Once they live in a Maildir on hermes, retention is find -mtime and search is grep -r (or NeoMutt’s ~b/~s filters, or any IMAP client if Dovecot is layered on). No more sifting work email for [CRON] subject lines.
The IP slot is already reserved. Hermes is in the inventory comment block at AQ114 .72 with a public IPv4 mapping pre-allocated. The design intent has been there for a while — this doc is the implementation plan.

Two mail roles, kept distinct

Role	What it does	Hermes behavior
Local destination	Accepts mail for a domain and writes it to local mailboxes	`mydestination` includes `rrchnm.internal`; `home_mailbox = Maildir/`
Relay	Forwards mail it isn’t authoritative for to an upstream	`relayhost = [external.smtp.com]:587`

A single Postfix daemon does both. transport_maps + mydestination together let it make a clean per-domain decision at receive time: *@rrchnm.internal → local delivery agent → Maildir, anything else → relay.

Software choices

Postfix + Maildir (mandatory)

Software	Role	Why
Postfix	SMTP daemon (accept, route, deliver)	Already the team standard; mature; the `transport_maps` + `virtual_alias_maps` primitives are exactly the shape this design needs
Maildir	On-disk format for local delivery	Built into Postfix’s local agent — not a separate package. One file per email under `~sysadmin/Maildir/{new,cur,tmp}/`. Easy to parse, easy to retain, no locking pathologies vs. mbox

Reading the mail — pick one or both

Option	Stack	Tradeoff
NeoMutt (TUI)	Just NeoMutt on hermes, pointed at the Maildir	Zero extra services. SSH in to read. Best for the “I rarely read it but want it searchable when I do” use case.
Dovecot (IMAP)	Dovecot listening on 127.0.0.1 (or LAN-only) + any IMAP client on workstation	Adds one daemon. Lets workstation mail clients (Thunderbird, aerc, mutt-via-IMAP) read hermes without an SSH session. Required if scripts/tools want a standard API rather than parsing Maildir files directly.
Direct Maildir parsing	Python `mailbox.Maildir` or equivalent	No daemon at all. Good for one-off scripts. Don’t run it concurrently with NeoMutt against the same Maildir or you’ll race on the rename-from-`new`-to-`cur` step.

Recommended starting point: Postfix + Maildir + NeoMutt only. Add Dovecot in a later phase if the SSH-in workflow gets annoying or if a workstation script wants IMAP.

Web UI deliberately not in scope

Roundcube / Snappymail are tempting but pull in the full LAMP stack (PHP, MariaDB, Caddy fronting) — for a single-mailbox sysadmin notifications inbox the maintenance burden vastly exceeds the value. Reconsider only if multiple humans need to read the same inbox.

Architecture: one hermes, no HA pair

Unlike the DNS resolvers (azelf/mesprit/uxie, three-instance) and the apt-cacher-ng pair (lugia/ho-oh, per-site), hermes is single-instance.

Reasoning:

Mail is store-and-forward by design. If hermes is down, sending hosts queue locally on their own Postfix (the existing relay-to-external pattern already requires every fleet host to run an outgoing-only Postfix, and queue retention there handles the gap).
A second instance would either need shared Maildir storage (NFS, with locking concerns) or split-brain inbox semantics (mail to sysadmin@ might land on either node). Neither is worth it for a notifications inbox.
The external relay is still on the path for outbound mail, so “real” mail still has the relay’s own redundancy.

If hermes goes down: outbound public mail stops; internal notifications queue on the sending hosts; once hermes comes back the queues drain. Acceptable for the SLA of “sysadmin alerts.”

Placement

Option	Pros	Cons
AQ114 / theia (preferred)	Matches the existing inventory reservation at `.72`. Sits on the more reliable site (AQ114 reliability bias — same rationale as the k0s primary controller on hyperion). Theia already hosts azelf + moltres so a third small container is well within budget.	Cross-site for RH461-originated mail, but it’s SMTP queueing — latency is irrelevant.
AQ114 / hyperion	Same site benefits; spreads load away from theia	Hyperion’s 860 QVOs already carry the etcd-fsync pain (see `disk-performance.md`); a Maildir write workload on top would be additional fsync pressure, though much lower volume than etcd.
RH461 / kyojin	Same-site as the bulk of fleet hosts (k0s, docker, lamp, caddies, dns, backup2)	RH461 is the less-reliable site; defeats the point of consolidating fleet-wide notifications somewhere stable.

Going with theia. Matches the pre-existing inventory reservation, sits on the reliable site, no fsync-sensitive neighbors. If theia becomes saturated we move it; the design isn’t placement-sensitive.

Sizing

Resource	Value	Rationale
Type	Container (not VM)	Lightweight workload — SMTP + filesystem writes; no kernel modules needed
CPU	1 vCPU	More than enough for the fleet’s notification volume
Memory	512 MiB	Postfix + NeoMutt working set is tiny; budget for Dovecot later if added
Disk	20 GiB on `theia_instances`	Maildir + Postfix queue + logs. Retention policy keeps growth bounded
Network	macvlan to host’s `eno1`	Same pattern as the DNS containers — direct LAN access at `10.112.12.72`

Postfix configuration

Final config files from the chat, ready to template into an Ansible role.

`/etc/postfix/main.cf`

# Identity
myhostname = hermes.rrchnm.internal
mydomain   = rrchnm.internal

# Accept mail for these locally; rrchnm.internal handled via transport_maps below
mydestination = hermes.rrchnm.internal, localhost

# Local delivery format
home_mailbox = Maildir/

# External relay for everything not locally destined
relayhost = [your.external.smtp.com]:587

# Routing tables
transport_maps     = hash:/etc/postfix/transport
virtual_alias_maps = hash:/etc/postfix/virtual

# Accept SMTP from fleet only (tighten to actual sender subnets)
mynetworks       = 127.0.0.0/8, 10.112.12.64/26, 10.112.113.128/25
inet_interfaces  = all
inet_protocols   = ipv4

mynetworks should be tightened to the actual fleet-facing subnets (AQ114 /26 + RH461 /25) — never 0.0.0.0/0. Hermes is reachable on its public IPv4 mapping at 129.174.134.64:25 per the inventory reservation, so the campus-edge ACL plus mynetworks are the two layers keeping it from being an open relay.

`/etc/postfix/transport`

rrchnm.internal    local:

Then postmap /etc/postfix/transport.

`/etc/postfix/virtual`

# Catch-all: every *@rrchnm.internal address routes to the sysadmin Maildir
@rrchnm.internal    sysadmin

Then postmap /etc/postfix/virtual.

Local user

useradd -m -s /usr/sbin/nologin sysadmin

Mail will land in /home/sysadmin/Maildir/{new,cur,tmp}/.

Mail flow

any fleet host
    │
    ▼
hermes (Postfix on 10.112.12.72:25)
    │
    ├── To: *@rrchnm.internal ────► /home/sysadmin/Maildir/  (Maildir, NeoMutt-readable)
    │
    └── To: someone@gmail.com ────► [your.external.smtp.com]:587 ────► internet

Smoke tests

# Should land in /home/sysadmin/Maildir/new/
echo "test body" | sendmail sysadmin@rrchnm.internal

# Should go out via the external relay
echo "test body" | sendmail you@gmail.com

# Verify routing table
postmap -q rrchnm.internal hash:/etc/postfix/transport      # → "local:"
postmap -q anything-else.com hash:/etc/postfix/transport    # → (no match → falls through to relayhost)

Retention policy

Maildir has no built-in retention — it’s plain files on disk. Implement via find + cron:

# /etc/cron.daily/maildir-retention  (chmod +x)
# Keep unread notifications 90 days, read ones 30 days, tmp 1 day.
find /home/sysadmin/Maildir/new -type f -mtime +90 -delete
find /home/sysadmin/Maildir/cur -type f -mtime +30 -delete
find /home/sysadmin/Maildir/tmp -type f -mtime +1  -delete

Sensible defaults: unread mail is kept longer so nothing important silently disappears before being seen.

If Dovecot gets added later, doveadm expunge -u sysadmin mailbox INBOX savedbefore 30d is a cleaner alternative (understands flags / read state semantically), but the find approach is fine for the initial deployment.

Reading the mail (Phase 3)

NeoMutt on hermes

# ~/.config/neomutt/neomuttrc on hermes (owned by sysadmin, but read via root or as sysadmin via su)
set mbox_type = Maildir
set folder    = /home/sysadmin/Maildir
set spoolfile = /home/sysadmin/Maildir

Workflow: ssh hermes -- sudo -u sysadmin neomutt.

Dovecot (Phase 4, optional)

Minimal /etc/dovecot/dovecot.conf:

protocols = imap
listen    = 10.112.12.72        # LAN-only; never 0.0.0.0 without TLS
mail_location = maildir:~/Maildir

ssl = no                        # internal-only; revisit if exposing past hermes
auth_mechanisms = plain
passdb { driver = pam }
userdb { driver = passwd }

Then any IMAP client (Thunderbird, aerc, mutt-via-IMAP, Python imaplib) reads from 10.112.12.72:143 as sysadmin.

Phased implementation

Phase 1: provision hermes (~30min)

Reserve / confirm IPs. .72 on AQ114 is already in the inventory comment block; promote it from comment to a real inventory entry under an smtp_servers group:
```
smtp_servers:
  hosts:
    hermes: { incus_remote: theia, ansible_host: 10.112.12.72, mac_addr: "<random-LAA-unicast>" }
```
MAC: fully randomized locally-administered + unicast, per the AQ114 convention.
Add TF resource. New file opentofu/smtp.tf defining one incus_instance.smtp_server for hermes, container type on theia_instances, macvlan NIC on eno1, ctr_debian image. Sizing from the table above.
make infra to bring it up. cloud-init handles baseline.
Public IP mapping. The inventory comment block reserves 129.174.134.64:25 — coordinate with network admin to confirm the existing NAT/firewall mapping. For initial bring-up, leave the public mapping in place but rely on mynetworks + smtpd_recipient_restrictions to prevent open-relay behavior. (Future: tighten further once the fleet-internal traffic is verified.)

Phase 2: Postfix role + local delivery (~1h)

New Ansible role: ansible/roles/postfix/ — installs postfix, templates main.cf/transport/virtual from the snippets above, runs postmap on the two map files, creates the sysadmin system user, lays down the maildir-retention cron.
New playbook: ansible/playbooks/setup-smtp-servers.yaml — Play 0 baseline (init + nftables against smtp_servers), then applies the postfix role. Match the shape of setup-caddy-servers.yaml (own-baseline-inline wrapper).
nftables: open 25/tcp to fleet subnets only; explicit ACCEPT from 10.112.12.64/26 + 10.112.113.128/25, default DROP for everything else on :25.
External relay credentials: if the external relay requires SASL auth, that goes in host_vars/hermes.sops.yaml with encrypted_regex covering smtp_sasl_password. Match the existing host_vars/<host>.sops.yaml pattern.
Makefile target: make smtp → setup-smtp-servers.yaml. Add to the make up pipeline at the same parallel band as caddy/lamp/backup/docker (Phase 3 of the up recipe — no DNS dependency since hermes resolves its relayhost via the campus DNS forwarder already configured fleet-wide).
Smoke tests: run the sendmail round-trip tests from the chat against hermes from one fleet host.

Phase 3: switch fleet senders to point at hermes (~30min)

Today every fleet host’s Postfix has relayhost = [external.smtp.com]:587. Flip them to relayhost = [hermes.rrchnm.internal]:25 (or [10.112.12.72]:25). Once flipped:

Mail to sysadmin@rrchnm.internal lands in hermes’s Maildir.
Mail to public recipients hits hermes, which forwards to the external relay.
Fleet hosts no longer need credentials for the external relay — only hermes does.

This is a per-host change handled by the existing baseline init role (or wherever the current relayhost setting lives). Audit first to find where the existing relayhost is set fleet-wide.

Phase 4 (optional, defer until needed): Dovecot for IMAP

Add dovecot-imapd to the postfix role (or split into a dovecot role), template the minimal config, open 143/tcp to the workstation subnet only via nftables. Document the workstation IMAP client setup in the same role’s README.

Open questions / decisions deferred

Public IP exposure. Hermes has 129.174.134.64:25 pre-allocated. Do we actually need port 25 inbound from the internet, or can hermes be LAN-only? If LAN-only, the public IP slot can be released (or held for future use). Lean: LAN-only initially — there’s no use case for accepting public-sourced mail today.
Sender auth for the external relay. Depends on which external relay (current setup uses one; this doc doesn’t presume which). Auth + TLS config in main.cf will need filling in once the target relay is named.
DKIM / SPF on outbound mail. If hermes is forwarding fleet-generated mail to the external relay, the existing SPF/DKIM posture of rrchnm.org outbound is unchanged — the external relay still does the final hop signing. If we ever wanted hermes to be the authoritative outbound MTA (direct internet delivery), that’s a much bigger lift with DNS + key management; not in scope.
Per-recipient routing inside rrchnm.internal. Catch-all to sysadmin covers the initial use case. If subdomains start to mean things (e.g. cron@, monitoring@, backup@ all going to different unix users for filtering), that’s a /etc/postfix/virtual extension and a useradd away; no architectural change.

apt-cacher-ng deployment (lugia on kyojin + ho-oh on hyperion; setup-package-proxies.yaml, package-proxies Make target) — closest peer in shape: a small internal service, per-site or single-site, integrated into make up, fronts an external dependency to reduce its blast radius. Hermes is single-site rather than per-site (mail’s store-and-forward semantics tolerate latency in a way apt downloads don’t). The pre-deployment plan doc (plan-apt-cacher-ng.md) was retired 2026-05-29.
internal-dns.md — defines rrchnm.internal as an authoritative zone on azelf + mesprit + uxie. Hermes’s myhostname = hermes.rrchnm.internal resolves against that zone; nothing extra needed beyond an A record for hermes in the rendered zone file (which auto-derives from inventory.yaml’s ansible_host once hermes is a real inventory entry rather than a comment).
infra-firewall.md — the nftables rules added in Phase 2 follow the existing pattern of explicit per-service ALLOWs from fleet subnets.