HAProxy reverse proxy

This document describes the HAProxy reverse-proxy fleet that fronts the CHNM web sites. Two frontends (HTTP redirect on :80, TLS passthrough on :443), per-site backends, per-source rate limiting with bandwidth throttling, custom error pages, and a custom log format that exposes per-source rate state. Configured by the in-tree haproxy Ansible role (ansible/roles/haproxy/), driven by ansible/playbooks/setup-haproxy-servers.yaml; the separate keepalived role provides VRRP failover. (Ported from the out-of-tree vibe/setup-haproxy-servers.yaml + keepalived/ into infra/ansible — see plan-haproxy-incusos.md in operator notes.)

Scope

This covers the web-site HAProxy fleet — atlas, prometheus, epimetheus today, with agni/rudra joining when built (they replace the retiring XCP-ng LBs; see fleet.yaml). The separate k0s public-edge HAProxy tier (janus/portunus/terminus) is a thin L4 passthrough to the k0s public Cilium Gateway and is not covered here.

Installed from the upstream Debian backport (bookworm-backports-3.0 from haproxy.debian.net) so we have HAProxy 3.0 features available — notably the bwlim-in filter used for throttle enforcement.

Architecture

Layer	Purpose
frontend http (:80, mode http)	HTTP→HTTPS redirect routing; per-site IP-whitelist enforcement; HTTP rate-limit measurement.
frontend https (:443, mode tcp)	TLS passthrough — HAProxy does not terminate TLS. Inspects the TLS hello and routes by SNI to the right per-site backend. Connection and bytes rate-limit measurement.
listen stats (:8181)	Admin stats page at http://<bind_ip>:8181/haproxy?stats.
backend http-<fqdn>	One per managed site. Mode http, forwards to the site’s HTTP origin.
backend https-<fqdn>	One per managed site. Mode tcp (passthrough). Optional send-proxy-v2 and websocket tunneling per site config.

The package’s /etc/haproxy/haproxy.cfg owns global and defaults; the playbook only writes into /etc/haproxy/conf.d/. EXTRAOPTS="-f /etc/haproxy/conf.d" in /etc/default/haproxy causes HAProxy to load every .cfg in that directory after the package config, so error-file directives and global log targets are inherited.

Rate limiting

Stick tables

Per-frontend stick-tables, keyed on source IPv4. Live-rate semantics — actions fire while the rate is currently above threshold and clear automatically when it drops. No sticky-ban / 7-day flag pattern.

Frontend	Table contents	Expire
http	http_req_rate(5m), gpc0, bytes_in_rate(10m), gpc1	1h inactivity
https	conn_rate(5m), gpc0, bytes_in_rate(10m), gpc1	1h inactivity

expire 1h governs table memory only, not enforcement behavior. Size is 100k entries.

Current thresholds

Derived from p99 / p99.9 of a ~21h log sample (May 2026). Anchored above the legitimate-traffic ceiling, soft-enforced via per-stream bandwidth limit (10 KB/s) so a false-positive degrades the experience instead of dropping it.

Frontend	Field	Threshold	Source for the value
HTTPS	conn_rate(5m)	100	Just above p99 (88). Max observed 302 (DotBot crawl).
HTTPS	bytes_in_rate(10m)	100 KB	At p99 (~100 KB). Max observed 262 KB.
HTTP	req_rate(5m)	300	Between p90 (280) and p95 (425). Catches the obvious bots.
HTTP	bytes_in_rate(10m)	80 KB	Above p99 (~80 KB). Tracked but not enforced — only the counter increments.

acl ip_whitelist src 10.0.0.0/8 exempts internal RFC1918 traffic from every enforcement rule. Internal monitors (10.112.*) that would otherwise trip the threshold are unaffected.

Enforcement

A named bwlim-in filter is declared on each frontend:

filter bwlim-in throttled default-limit 10k default-period 1s

The active enforcement lines apply that filter via set-bandwidth-limit when the relevant ACL is true:

# HTTPS (mode tcp)
tcp-request content set-bandwidth-limit throttled if exceeds_conn_limit !ip_whitelist
tcp-request content set-bandwidth-limit throttled if exceeds_bytes_limit !ip_whitelist

# HTTP (mode http)
http-request set-bandwidth-limit throttled if { sc_http_req_rate(0) gt 300 } !ip_whitelist

Effect: an abusive source’s connections are capped to 10 KB/s per stream. The connection still goes through, just so slowly that scraping/probing becomes uneconomical. Legitimate users sharing an IP (NAT, mobile carrier) get slow-but-functional service rather than being locked out.

Reject as an alternative

Each enforcement section also contains commented-out reject (HTTPS) and deny 429 (HTTP) lines. An operator can switch from throttle to hard reject without re-running the playbook:

1. Edit /etc/haproxy/conf.d/frontends.cfg on the host.
2. Comment the throttle line(s), uncomment the corresponding reject/deny line(s).
3. Validate: haproxy -c -f /etc/haproxy/haproxy.cfg -f /etc/haproxy/conf.d
4. Reload: systemctl reload haproxy

Same pattern for the custom log-format: option httplog / option tcplog are rendered as commented alternatives.

The next playbook run will rewrite the file back to the defaults (throttle on, custom log-format on). If a switch should be permanent, change it in the playbook template.

Custom log format

Each frontend uses a custom log-format that takes the standard httplog/tcplog fields and appends per-source rate-limit state.

Fields appended

Field	Frontend	Meaning
rl_req_rate	HTTP	HTTP requests from src IP in last 5m (threshold 300)
rl_conn_rate	HTTPS	TCP connections from src IP in last 5m (threshold 100)
rl_gpc0	both	counter, increments each event while the corresponding rate exceeds threshold
rl_bytes_rate	both	bytes received from src IP in last 10m (HTTP threshold 80 KB, HTTPS 100 KB)
rl_gpc1	both	counter, increments each event while bytes_rate exceeds threshold

rl_gpc0 and rl_gpc1 are bounded only by the table-entry lifetime. Any non-zero value on these fields means the source crossed a threshold at least once during the entry’s lifetime — useful for quickly listing IPs that ever tripped a rate limit (vs. those that just happened to be busy this minute).

Example log lines

# HTTPS (mode tcp)
1.2.3.4:54321 [21/May/2026:14:30:00] https https-foo.example.org/server1 1/1/29966 2856 -- 13/13/4/4/0 0/0 rl_conn_rate=3 rl_gpc0=0 rl_bytes_rate=8192 rl_gpc1=0

# HTTP (mode http)
1.2.3.4:54321 [21/May/2026:14:30:00.123] http http-foo.example.org/redirect 0/0/1/3/4 308 136 - - ---- 23/2/0/0/0 0/0 "GET / HTTP/1.1" rl_req_rate=42 rl_gpc0=0 rl_bytes_rate=1024 rl_gpc1=0

Logging chain

Stage	Where
HAProxy emits to syslog	log /dev/log local0 in /etc/haproxy/haproxy.cfg global, inherited via defaults (no level filter — passes all severities).
rsyslog routes	/etc/rsyslog.d/49-haproxy.conf: :programname, startswith, "haproxy" → /var/log/haproxy.log; stop.
Single write per event	The frontends do not add their own log directives. Earlier configs did, producing 3× duplication; removed.

There is no in-frontend log-level “verbosity dial” — <level> on the log directive is a filter, not a verbosity knob. HAProxy emits at fixed severities (access at info, errors at err, etc.). If more visibility is needed per-frontend, the levers are: http-request capture (more fields), option log-separate-errors, set-log-level <severity> if <cond>, or a separate file target via a frontend log directive.

Custom error pages

Generated from roles/common/templates/error_page.html.j2 (a self-contained HTML with inline CSS) and written to /etc/haproxy/errors/<code>.http. Codes rendered: 400, 403, 408, 429, 500, 502, 503, 504.

Each file is a complete HTTP response: status line + headers + blank line + body. Includes Content-Type: text/html; charset=utf-8, Cache-Control: no-cache, no-store, must-revalidate, Connection: close, and (for 429) Retry-After.

The errorfile directives in the package’s defaults block bind these to status codes. Anywhere HAProxy emits one of those codes (per-site IP-whitelist 403, the future rate-limit 429 if switched to deny, backend connect failures 502/503/504), the configured page is returned.

Template variables come from the http_status_messages group var: each entry has code, reason, headline, subline, and optional retry_after.

Termination state codes

Every access-log line ends with a termination-state field. 2 chars for TCP-mode (the https frontend), 4 chars for HTTP-mode (the http frontend).

Char position	Meaning
1st	Who closed / what category. Uppercase = active close. Lowercase = timeout. C=client, S=server, P=proxy, R=resource exhaustion, I=internal err, D=backend down, K=killed by operator.
2nd	Session state at close. R=reading request, Q=in queue, C=connecting to backend, H=waiting on response headers, D=in data phase, L=last flush, T=tarpit, -=normal completion.
3rd (HTTP)	Persistence cookie state — - unless stick cookies used.
4th (HTTP)	Persistence cookie operation — same.

Common states observed in production

State	Reading
--	Normal close, TCP mode. Session completed cleanly.
----	Normal close, HTTP mode (4-char form of --).
cD	Client timeout in data phase. Client opened TCP + TLS, then sat idle past timeout client 30s. Classic TLS scanner/probe pattern. Public TLS endpoints typically show 15-25% of this.
CD	Client active close during data transfer. User killed the request mid-flight (closed tab, network drop). Normal at low %.
SC / SC--	Server-side abort during connect attempt. HAProxy tried to open a connection to the backend and failed — refused, down, RST. Worth investigating if concentrated on one backend.
sD	Server timeout in data phase. Backend went silent past timeout server 30s. Very rare = healthy backends.
SD	Server actively reset during data transfer. Backend crashed or RST mid-response.
PR--	Proxy rejected during request reading. HAProxy itself denied the request before forwarding. In our config this is the per-site http-request deny deny_status 403 if !ip_whitelist lines firing — i.e., the IP-whitelist enforcement working.
cH	Client timeout while waiting for server headers. Slow backend + impatient client / proxy.
cR	Client timeout while reading request. Stalled HTTP request.
SH	Server abort while sending headers (typically backend crash).
LR	Logged after request — informational, not a termination per se.

A high cD ratio is the strongest scanner-noise signal on the HTTPS frontend. Throttle catches sustained ones; the rest are background.

Operational reference

Updating a single site (incremental — no full re-apply)

Per-site routing is map-driven, so adding/updating a normal site does NOT re-render frontends.cfg. Edit the catalog (vars/websites-{static,hugo}.yaml), then run the config-only path — it skips pre-flight, the apt install, and keepalived (those are other tags), and renders only conf.d/* + the maps, then hot-reloads (seamless: workers drain, no dropped connections):

# canary one node first, then drop --limit to roll to the rest
ansible-playbook playbooks/setup-haproxy-servers.yaml --tags config --limit epimetheus
ansible-playbook playbooks/setup-haproxy-servers.yaml --tags config

The nodes apply in parallel (no serial — three built today: atlas, prometheus, epimetheus), and the config play skips fact-gathering. What changes for a new site: one line in maps/sites.map (or sites-suffix.map for a wildcard) + its conf.d/<id>.cfg backend. frontends.cfg only changes for enforce_ip_allowlist / acme_pin_first override sites.

Emergency zero-reload add (Ansible stays source of truth — re-render reconciles): add a route to the live process via the admin socket:

echo "add map /etc/haproxy/maps/sites.map newsite.dev.chnm.gmu.edu newsite.dev.chnm.gmu.edu" \
  | sudo socat - /run/haproxy/admin.sock

Validate config

haproxy -c -f /etc/haproxy/haproxy.cfg -f /etc/haproxy/conf.d

The Ansible playbook runs this as a task before restarting; a syntax error fails the play rather than the running service.

Inspect stick-tables live

echo "show table https" | sudo socat - /run/haproxy/admin.sock
echo "show table http"  | sudo socat - /run/haproxy/admin.sock

Each row shows per-IP rates and counters — gold standard for “is rate-limiting working?”

Analyze logs

ansible/roles/haproxy/files/analyze-haproxy-log.py parses the custom log format and produces: rate distributions (p50 / p75 / p90 / p95 / p99 / p99.9 / p99.99 / max), top-N source IPs by max rate, top-N /24 prefixes (catches distributed scrapers per-IP limits miss), and the termination-state breakdown.

/var/log/haproxy.log

sudo /path/to/analyze-haproxy-log.py

# include rotated/gzipped logs for multi-day window
sudo /path/to/analyze-haproxy-log.py /var/log/haproxy.log /var/log/haproxy.log.*.gz

# wider top tables
sudo /path/to/analyze-haproxy-log.py --top 50

Threshold tuning workflow

1. Let logs accumulate at least 24-72h (need diurnal + weekday/weekend cycles).
2. Run analyze-haproxy-log.py against the full window (including rotated/gzipped).
3. In the RATE DISTRIBUTIONS section, read off p99 and p99.9 for the rate you’re tuning.
4. Pick a threshold above p99 (catches the top ~1% — outliers) or above p99.9 (catches top ~0.1% — only clearly abusive).
5. Edit thresholds in ansible/group_vars/haproxy_servers.yaml (haproxy_https_conn_rate_limit, haproxy_https_bytes_rate_limit, haproxy_http_req_rate_limit, haproxy_http_bytes_rate_limit). The role re-renders the ACL/action lines + the doc comments from these.
6. Re-render + reload: ansible-playbook playbooks/setup-haproxy-servers.yaml --tags config.
7. After 24h of enforcement, re-run the analyzer; look at the THRESHOLD TRIGGERS section — any legitimate IPs in there are false positives and the threshold needs to rise.

Distributed crawlers (Applebot, Baidubot, AWS scraper farms)

Per-IP rate limiting can’t catch these — they fan out across hundreds of IPs at low per-IP rates. The analyzer’s TOP /24 PREFIXES table surfaces them. Options:

• Verify legitimate search-engine crawlers (Applebot, Bingbot, etc.) via forward-confirmed reverse DNS and whitelist by allowlist if needed.
• Block ASNs at the firewall layer for clearly hostile distributed scrapers.
• Add a per-/24 stick-table (would need a second table keyed src,/24 — not currently configured).

Files

Path	Owner	Purpose
ansible/playbooks/setup-haproxy-servers.yaml	repo	Orchestrator. pre-flight → init+nftables → haproxy role → keepalived role.
ansible/roles/haproxy/	repo	Install (3.0 backport) + render frontends/stats/per-site cfg + errorfiles + F5-return route. Templates: frontends.cfg.j2, stats.cfg.j2, site.cfg.j2.
ansible/roles/keepalived/	repo	VRRP (one vrrp_instance per VIP). Reusable; nginx variant dropped in the port.
ansible/group_vars/haproxy_servers.yaml	repo	VIP/VRRP model, rate-limit thresholds, fwmark/F5-return gateway, stats bind. vrrp_auth_password in the sibling .sops.yaml.
ansible/group_vars/all.yaml (http_status_messages)	repo	Shared error-page catalog (caddy + haproxy).
ansible/roles/caddy/templates/error-page.html.j2	repo	Shared (daemon-neutral) error-page body; haproxy wraps it in an HTTP/1.0 head.
ansible/roles/haproxy/files/analyze-haproxy-log.py	repo	Log-distribution analyzer.
/etc/haproxy/haproxy.cfg	package	Global + defaults + errorfile directives. Not managed by the role.
/etc/haproxy/conf.d/frontends.cfg	role (rendered)	frontend http, frontend https, filters, rate-limit ACLs. Static — per-site routing is map-driven (below); only enforce_ip_allowlist/acme_pin_first overrides emit per-site lines.
/etc/haproxy/conf.d/stats.cfg	role (rendered)	listen stats block.
/etc/haproxy/conf.d/<fqdn>.cfg	role (rendered)	One per HAProxy-fronted site; backend http-<fqdn> + backend https-<fqdn>.
/etc/haproxy/maps/sites.map	role (rendered)	Exact SNI/Host → backend-id. One line per site (+ aliases). Looked up by both frontends. Outside conf.d so the -f conf.d load doesn’t parse it as config.
/etc/haproxy/maps/sites-suffix.map	role (rendered)	Wildcard suffix → backend-id (map_end). One line per *. site.
/etc/haproxy/errors/<code>.http	role (rendered)	Per-code error response files.
/etc/rsyslog.d/49-haproxy.conf	package	Routes haproxy programname to /var/log/haproxy.log.
/var/log/haproxy.log	runtime	Single-write access log with rl_* fields.
/run/haproxy/admin.sock	runtime	Unix socket for show table, disable server, etc.

Why TLS passthrough (not termination)

The https frontend is mode tcp and routes by SNI. This means:

• No certificates live on the proxy. Each backend handles its own TLS termination.
• Renewals and certificate management stay on the origin hosts, not the proxy fleet.
• HAProxy can’t see the HTTP request (path, method, headers) on :443 — only the TLS hello and bytes. Rate limiting on HTTPS is therefore connection/byte-based, not request-based. The http frontend on :80 can do request-rate limiting because it sees plaintext HTTP.

If we ever want WAF-style request inspection on HTTPS, that’s the trade-off to revisit — would need TLS termination at the proxy and a certificate distribution story.