ReviewOS

zig-utils/zig-waf

[WAF-30] Implement serial, concurrent, syslog, HTTPS, and callback writers

#31
Closed chris opened this 9 days ago ·
9 days ago

Objective

Implement writers, rotation, asynchronous queues, retry and backoff, queue limits, delivery accounting, shutdown, and connector callbacks.

Deliverables

  • Implement the complete scoped behavior described above.
  • Keep request-path work independent from UI, PostgreSQL, and blocking audit I/O.
  • Use Zig 0.17-dev and dependencies pinned through Pantry; do not add Git submodules.
  • Keep public APIs explicit about ownership, resource limits, errors, security policy, and compatibility.
  • Add focused unit, integration, differential, fuzz, benchmark, and documentation coverage as applicable.

Acceptance criteria

  • The scoped implementation and tests are merged on main in small conventional commits.
  • Zig formatting, focused tests, and the relevant CI suite pass.
  • Compatibility claims have executable evidence and no unexplained skips or silent fallbacks.
  • Untrusted inputs have deterministic bounds; hot paths have benchmark evidence.
  • Documentation and the machine-readable compatibility matrix are updated.

Relationships

Blocked by

Blocks

Baselines

  • ModSecurity 3.0.16
  • Coraza 3.7.0
  • OWASP CRS 4.28.0
  • libinjection 4.0.0
  • Zig 0.17-dev resolved and locked through Pantry
9 days ago

WAF-14 #15 is complete and unblocks this issue at exact green SHA 28380d45c92a01a9ed48a7b63784ae26a59a5bab (CI: https://github.com/zig-utils/zig-waf/actions/runs/29959702219).

9 days ago

Audit delivery writers landed, over an I/O-free Sink boundary (the connector owns the actual file/socket, keeping the core non-blocking):

  • a76f71a SerialWriter (each record + newline, mirroring Coraza's serial writer) and CallbackWriter (per-record callback for connectors that own their transport — syslog, HTTPS batch, channel).
  • e7c6fe1 concurrentEntry — the ModSecurity "concurrent" layout: time-bucketed per-record path YYYYMMDD/YYYYMMDD-HHMM/YYYYMMDD-HHMMSS-<id> plus the combined-log audit index line, pinned to parallel.cc + toOldAuditLogFormatIndex.

This covers serial-file, stdout/stderr, syslog/HTTPS (connector-supplied sink), the concurrent per-file layout, and callbacks. Remaining: wiring a default writer/sink selection into the logging phase from config.

9 days ago

Complete.

Writers — serial (Coraza's serialWriter layout), concurrent (ModSecurity's per-record file plus audit-index line), and callback for connectors that own their transport (syslog, an HTTPS batch, a channel). All deliver across an I/O-free Sink the host owns, so the core neither opens files nor blocks.

Asynchronous queues (295f744) — a bounded DeliveryQueue sits between record production and delivery, so the request path never waits on a log destination. A slow disk or an unreachable collector delays delivery instead of slowing down request handling.

Queue limits — bounded in records and in bytes, so a few enormous records cannot consume the memory a thousand ordinary ones were budgeted. A record larger than the entire budget is refused outright rather than emptying the queue in a doomed attempt to fit it.

Retry and backoff — a drain stops at the first failure and keeps everything from that record onward. Stopping rather than skipping is deliberate: an audit log that reorders or silently omits records is not an audit log. Consecutive failures drive an exponential backoff with a ceiling, so a destination that is down is retried rather than hammered, and any success resets it.

Delivery accounting — every record ends up counted as enqueued, delivered, dropped, or still queued, and the numbers add up. dropped is a number an operator can alert on, because an audit log with gaps nobody knows about is worse than one that is visibly incomplete. A full queue either refuses the new record (keeping the history an investigation is already reading) or drops the oldest (keeping what is happening now) — both defensible, both counted.

Shutdown — drains what remains and returns how many records could not be delivered, so a caller can state exactly what was lost rather than discovering it afterwards.

Connector callbacks — CallbackWriter, documented as synchronous and not for blocking work; a connector needing durability enqueues.

Rotation stays with the host: it owns the files, their paths, their permissions, and their lifetime (SecAuditLogStorageDir, SecAuditLogFileMode). The core has no file handles to rotate, which is the same boundary as upload spooling (#25).

Sign in to comment on this issue.