[WAF-08] Implement collection variables, selectors, origins, and macros

Objective

Implement ARGS families, headers, cookies, files, multipart, XML, GEO, RULE, ENV, TX, persistent-collection namespace interfaces, key and regex selectors, exclusions, counts, origins, and runtime macro expansion. Cross-request persistence, expiry, and synchronization for IP/SESSION/USER/GLOBAL/RESOURCE are implemented by WAF-09 against these interfaces.

Deliverables

• Implement bounded owned collection storage with repeated and map mutation semantics.
• Preserve source origins and offsets and enforce minimum phase availability.
• Implement exact and production zig-regex key selectors, exclusions, and count targets.
• Compile and expand bounded runtime macros with explicit baseline policy.
• Publish atomic derived ARGS, cookie, header, match, and file metadata families for downstream parsers.
• Keep request-path work independent from UI, PostgreSQL, and blocking audit I/O.
• Use Zig 0.17-dev and dependencies pinned through Pantry; do not add Git submodules.

Acceptance criteria

• The scoped implementation and tests are merged on main in small conventional commits.
• Zig formatting, focused tests, and the relevant CI suite pass.
• Compatibility claims have executable evidence and no unexplained skips or silent fallbacks.
• Untrusted inputs have deterministic entry, key, value, token, output, and physical-allocation bounds.
• Selector exhaustion is an error, never a false non-match.
• Documentation, dedicated benchmark, and machine-readable compatibility matrix are updated.

Relationships

• Parent: WAF-00

Blocked by

• WAF-06
• WAF-07

Blocks

• WAF-09 — persistent backends, expiry, synchronization
• WAF-23 — request target/header/cookie parsing
• WAF-25 — multipart and file producers
• WAF-26 — URL-encoded/JSON body producers
• WAF-27 — XML producers

Baselines

• ModSecurity 3.0.16
• Coraza 3.7.0
• OWASP CRS 4.28.0
• Zig 0.17-dev resolved and locked through Pantry

Unblocked: WAF-07 is complete and closed in #8. The executable scalar registry, origin metadata, phase availability, bounded storage, and collection-producer boundary are now available on main at b59955e.

Progress: pushed b0ac68c (feat(engine): add bounded collection storage) directly to main.

This establishes the 37-namespace collection registry, per-namespace key policy, repeated-value insertion order, owned keys/values, exact source offsets/origins, deterministic entry/key/value/aggregate limits, allocation-free exact/all selection, and a matcher callback contract for compiled regex selectors. Focused tests cover registry uniqueness, repeated headers, case semantics, source preservation, matcher scans, and pre-ownership limit failures.

WAF-09 remains the owner of persistence and expiry behavior for IP/SESSION/USER/GLOBAL/RESOURCE; WAF-08 defines those namespaces and the transaction-facing contract they plug into.

Progress: pushed HEAD (feat(engine): compile bounded runtime macros) directly to main. Runtime strings now compile once into immutable literal/scalar/collection tokens, validate variable names and hostile syntax, expand with explicit input/token/output bounds, and expose the ModSecurity-vs-Coraza missing-value difference as a configured policy rather than a silent behavior change. Tests cover scalar, keyed collection, first-value, malformed input, unknown names, and output exhaustion.

Progress correction and integration:

• The runtime macro commit referenced as HEAD in my prior comment is 29e544d.
• Pushed 2a0a66e (feat(engine): wire transaction collections) to main.

Transactions now publish repeated request/response headers and name collections with exact offsets, retain match history, expose phase-gated collection iterators, allow bounded subsystem publication, and resolve compiled macros against live scalar/collection state. Paired collection entries are limit-atomic.

The initial per-value allocation design measured ~56 us/lifecycle and was rejected before commit. Transaction-local arena ownership reduced the same populated ReleaseFast benchmark to ~7.56 us (7,632-byte transaction) versus ~5.18 us before collection publication. This is current tracking evidence; WAF-39 still owns comparative release gates.

Final implementation progress before qualification:

• 73d63a2 adds map replacement/removal, tombstoned physical-memory accounting, inclusion/exclusion targets, count semantics, and explicit selector errors.
• b55f27c integrates the pinned production zig-regex engine for key selectors.
• 0d416fd replaces per-match DFA reconstruction with one reusable matcher per worker; the dedicated benchmark improved from ~151 us to ~7.20 us/transaction.
• 401ba73 publishes the compatibility matrix and ownership/producer contracts.
• 531cc68 adds atomic derived ARGS/ARGS_* and *_NAMES publication, cookies, file metadata families, and ARGS_COMBINED_SIZE/FILES_COMBINED_SIZE maintenance.

The parser-specific producers remain linked to WAF-23/WAF-25/WAF-26/WAF-27; persistence/expiry for the persistent namespaces remains WAF-09. WAF-08 supplies their complete bounded transaction-facing contract rather than duplicating those downstream implementations.

Completed on main at 531cc684ffda1585b9324623d65d546d86e1f47c.

Delivered:

• bounded, arena-owned collection storage across the 37 stable namespaces, with per-namespace key policy, phase availability, origins, atomic batch publication, map mutation, selectors, exclusions, and count targets
• immutable compiled runtime macros with explicit input/token/output bounds and selectable ModSecurity/Coraza missing-variable behavior
• ruleset-owned zig-regex selectors with reusable per-worker matcher state and explicit limit/failure propagation
• transaction producers for request/response headers, arguments, cookies, uploads, and match history, including derived name and combined-size variables
• collection contracts, compatibility evidence, focused tests, hostile-limit tests, and a release benchmark

Verification:

• exact Pantry-pinned Zig 0.17.0-dev.1441+d5181a9c9
• zig fmt, zig build test, and zig build check passed locally
• ReleaseFast collection benchmark: 50,000 complete transactions, 7.23 µs/transaction, 7,648 transaction bytes
• hosted CI passed: https://github.com/zig-utils/zig-waf/actions/runs/29935997519

Cross-request persistence, expiration, conflict handling, and LMDB remain intentionally owned by #10; request parsers publish through these now-stable collection contracts in #24, #26, #27, and #28.