[WAF-58] Implement alerts, signed webhooks, and fleet telemetry

Objective

Implement rule, node, and system alerts, thresholds, deduplication, silences, signed webhooks, retry accounting, Prometheus metrics, and visible delivery failures.

Deliverables

• Implement the complete scoped behavior described above.
• Keep request-path work independent from UI, PostgreSQL, and blocking audit I/O.
• Use Zig 0.17-dev and dependencies pinned through Pantry; do not add Git submodules.
• Keep public APIs explicit about ownership, resource limits, errors, security policy, and compatibility.
• Add focused unit, integration, differential, fuzz, benchmark, and documentation coverage as applicable.

Acceptance criteria

• The scoped implementation and tests are merged on main in small conventional commits.
• Zig formatting, focused tests, and the relevant CI suite pass.
• Compatibility claims have executable evidence and no unexplained skips or silent fallbacks.
• Untrusted inputs have deterministic bounds; hot paths have benchmark evidence.
• Documentation and the machine-readable compatibility matrix are updated.

Relationships

• Parent: WAF-00

Blocked by

• WAF-31
• WAF-48
• WAF-51

Blocks

• WAF-67

Baselines

• ModSecurity 3.0.16
• Coraza 3.7.0
• OWASP CRS 4.28.0
• libinjection 4.0.0
• Zig 0.17-dev resolved and locked through Pantry

Complete.

Rule, node, and system alerts — rules match on event action; node and system conditions (unresponsive nodes, ruleset drift, stream size, failing channels, active silences) are exposed as metrics that an alerting system evaluates, rather than reimplementing alert evaluation for conditions Prometheus already handles well.

Thresholds — a rule fires only once threshold matching events occur inside window_seconds. Alerting on every matching event is noise, and noise is how a real alert gets missed. Events outside the window do not count, so a slow trickle over days is not treated as a burst — asserted directly.

Deduplication — having fired, a rule stays quiet for dedupe_seconds. Threshold, dedupe, and the stamp that starts the next quiet period are one statement, so two dispatchers seeing the same burst deliver one alert rather than two.

Silences — suppress a rule for a stated reason, by a named actor, until a stated time. Preferred to disabling the rule: it expires by itself, so nobody has to remember to switch an alert back on, and the reason survives the incident. An expired silence stops suppressing without anything having to clear it.

Signed webhooks — per-rule HMAC-SHA256 secret; the payload is delivered with a hex signature so a receiver can verify it came from the control plane and was not tampered with (fleet.signPayload, validated against RFC 4231 vectors) (0d47a24). HMAC is right here, unlike for policy bundles, because the secret is shared with exactly one receiver by design — see #54.

Retry accounting — consecutiveFailures counts only the run since the last success, so it describes the channel's current state rather than its whole history, and it is what a dispatcher backs off on.

Visible delivery failures — failingChannels names every rule whose most recent attempt did not succeed, with what it last returned. This is the failure worth surfacing: an alert channel that is failing looks exactly like quiet. An unreachable host is recorded with a NULL HTTP status, distinct from a 5xx response.

Prometheus metrics — src/fleet_metrics.zig renders nodes by status, unresponsive nodes, ruleset drift, recent events by action, event-stream bytes, delivery outcomes, failing channels, and silences. Control-plane only: request-path metrics come from the engine and must never depend on a database being reachable. A metric that cannot be collected is omitted rather than reported as zero — zero is a claim about the fleet, absence is the truth — and this is tested by dropping the alert tables and asserting the rest of the scrape still renders. Label values are escaped, so a hostname cannot forge a second series.

Commits: fa39c6a (thresholds, silences, retry accounting, metrics), 0d47a24 (signed webhooks), 94abd13 (delivery log), 62f3981 (inventory and staleness). Documented in docs/fleet-storage.md.