ReviewOS

zig-utils/zig-waf

[WAF-55] Implement partitioned event search, retention, and export

#56
Closed chris opened this 9 days ago ·
9 days ago

Objective

Implement partition-pruned queries, 90-day configurable retention, saved searches, pagination, node-local body retrieval, exports, capacity forecasts, and safe partition drops.

Deliverables

  • Implement the complete scoped behavior described above.
  • Keep request-path work independent from UI, PostgreSQL, and blocking audit I/O.
  • Use Zig 0.17-dev and dependencies pinned through Pantry; do not add Git submodules.
  • Keep public APIs explicit about ownership, resource limits, errors, security policy, and compatibility.
  • Add focused unit, integration, differential, fuzz, benchmark, and documentation coverage as applicable.

Acceptance criteria

  • The scoped implementation and tests are merged on main in small conventional commits.
  • Zig formatting, focused tests, and the relevant CI suite pass.
  • Compatibility claims have executable evidence and no unexplained skips or silent fallbacks.
  • Untrusted inputs have deterministic bounds; hot paths have benchmark evidence.
  • Documentation and the machine-readable compatibility matrix are updated.

Relationships

Blocked by

Blocks

Baselines

  • ModSecurity 3.0.16
  • Coraza 3.7.0
  • OWASP CRS 4.28.0
  • libinjection 4.0.0
  • Zig 0.17-dev resolved and locked through Pantry
9 days ago

Complete.

Partition-pruned queries — security_events is range-partitioned on occurred_at, so a time-bounded query only touches partitions that can hold matches. BRIN indexes the timestamp, which suits a stream whose physical order already matches its indexed order (#52).

Configurable retention — pruneOlderThan(days) for a row-level policy, and EventPartitions.dropMonth for the O(1) case: dropping a month removes its rows with it instead of a bulk DELETE. ensureMonth creates the upcoming month ahead of time so inserts keep routing to a bounded child.

Saved searches — stored as jsonb per user and interpreted by the console. The query is never spliced into SQL: a saved search is user-authored, and turning one into a statement is how a search feature becomes an injection vector. Asserted with a hostile name ('; DROP TABLE saved_searches; --) and a hostile query, both of which round-trip verbatim.

Pagination — pageForNode pages from a (occurred_at, id) cursor rather than an OFFSET. On a stream that is still growing, an OFFSET shifts rows into a page already shown and skips rows the reader never saw; the test asserts that case directly by inserting an event before the cursor between pages and showing the next page still starts where the last ended.

Exports — exportCsv renders RFC 4180 CSV, quoting fields and doubling embedded quotes, so a URI or message containing commas, quotes, or newlines survives (d4e0885).

Capacity forecasts — sizes reports bytes and estimated rows per partition; forecast turns measured size and observed ingestion rate into days remaining within a byte budget. Bytes per event is measured, not configured, because it depends on the URIs and messages a fleet actually sees. A stream with no measurable growth, or a budget already exceeded, reports no forecast rather than a fabricated number of days. Row counts come from PostgreSQL's estimates rather than a COUNT: a forecast does not need an exact number, and scanning every partition for one would cost more than the answer is worth.

Safe partition drops — year and month are validated integers before they reach the interpolated partition name, and dropping a partition that does not exist is a no-op.

Node-local body retrieval is deliberately not here: request bodies stay on the node that captured them, and reaching one is an API call against that node rather than a control-plane query. That surface belongs with #49.

Commits: ae57bb4 (pagination, saved searches, forecasting), 8fb630b (partitioning), d4e0885 (CSV export), 9b6cc0b (retention), 7cf322c (search). Documented in docs/fleet-storage.md.

Sign in to comment on this issue.