[WAF-09] Implement TX and persistent IP, SESSION, USER, and GLOBAL collections

Objective

Implement bounded transaction-local TX mutation and cross-request IP, SESSION, USER, GLOBAL, and RESOURCE collections with ModSecurity 3.0.16 and Coraza 3.7.0 semantics. Persistence must be concurrency-safe, expiration-aware, and explicit about backend failure policy; ordinary requests that do not initialize a persistent collection must pay no persistence cost.

Compatibility contracts

• TX remains transaction-owned and is never written to a persistence backend.
• initcol binds a persistent namespace to a caller-supplied key, loads it at most once per transaction, lazily removes expired variables, and exposes the stored values through the normal collection API.
• setuid, setsid, and setrsc bind the USER, SESSION, and RESOURCE keys through the same initialization path.
• setvar supports bounded assignment, deletion, and signed numeric addition/subtraction with checked overflow and compatible missing/non-numeric handling.
• expirevar stores an absolute per-variable expiration derived from a bounded TTL. Expired values are invisible on load and removable by bounded maintenance cleanup.
• Only dirty persistent collections are committed. A commit atomically applies the transaction mutation log against an expected record revision.
• Revision conflicts use a bounded retry/rebase policy so concurrent increments do not silently lose updates.
• Backend unavailability, timeout, conflict exhaustion, corruption, and capacity errors remain distinct. The configured fail-open or fail-closed policy is observable and never silently changes ruleset behavior.

Deliverables

• A versioned backend callback contract with explicit ownership for load results, atomic commits, cleanup, feature discovery, and typed errors.
• A bounded transaction persistence session that tracks namespace keys, loaded revisions, dirty mutations, expiration metadata, and retry limits.
• A thread-safe in-memory backend for tests, embedded deployments, and concurrency qualification.
• A production LMDB backend linked from Pantry-managed openldap.org/liblmdb; no system-library fallback and no Git submodule.
• Builder/runtime configuration for backend ownership, failure policy, key/value/record limits, retry limits, deadlines, and cleanup budgets.
• Engine APIs required by the future SecLang action compiler for initcol, setuid, setsid, setrsc, setvar, and expirevar.
• Focused unit, threaded integration, restart/persistence, expiration, corruption, fault-injection, and performance coverage.
• Documentation and machine-readable compatibility evidence.

Interfaces

The backend receives a composite (namespace, collection_key) key. load returns an allocator-owned snapshot with a monotonic revision. commit receives the expected revision and an ordered mutation batch and must either apply all changes once or return a conflict without partial publication. cleanup removes no more than the configured record/variable budget.

The transaction layer owns loaded values and mutation data. Backend callbacks may block only when a persistent collection is explicitly initialized or flushed; they are never called by the default request lifecycle. Callers choose fail-open or fail-closed behavior at build time.

Resource limits

• Maximum initialized namespaces per transaction.
• Maximum collection-key, variable-name, and variable-value bytes.
• Maximum values and serialized bytes per persistent record.
• Maximum mutations, retry attempts, cleanup work, and backend wait duration.
• Checked timestamp and signed arithmetic with deterministic error reporting.

Acceptance criteria

• TX assignment, deletion, and checked arithmetic match the pinned compatibility fixtures.
• Every persistent namespace round-trips through the in-memory and LMDB backends.
• Expired variables are never exposed, and cleanup remains bounded.
• Concurrent increments preserve the exact total under real threads; conflict exhaustion follows the configured policy.
• LMDB restart tests prove committed records survive process/backend recreation and atomic failures leave the prior revision readable.
• Corrupt or oversized persisted data is rejected without unbounded allocation.
• Requests that never initialize persistence make zero backend calls.
• Exact Pantry-pinned Zig formatting, tests, compile checks, and the persistence benchmark pass locally and in hosted CI.
• Implementation lands on main in small conventional commits with issue evidence and no unexplained compatibility skip.

Test matrix

• Fresh/missing and existing collection initialization.
• Case-insensitive persistent variable keys and independent namespace/collection keys.
• Assignment, deletion, positive/negative delta, zero, invalid numeric text, and overflow.
• TTL boundary, clock rollback, lazy expiration, refresh, delete, and bounded cleanup.
• Concurrent distinct records and heavily contended single-record increments.
• Conflict retry success/exhaustion and backend timeout/unavailable/corrupt/capacity faults under both failure policies.
• LMDB open, close, restart, map-full, reader pressure, malformed bytes, and crash-safe atomic transaction behavior.
• No-persistence lifecycle and initialized-collection latency/allocation benchmarks.

Relationships

• Parent: WAF-00

Blocked by

• WAF-08 — complete

Blocks

• WAF-12 — persistent directives and collection configuration
• WAF-15 — ctl and action execution using collection mutation
• WAF-34 — SecRules Test Set persistence cases
• WAF-35 — ModSecurity differential fixtures
• WAF-36 — Coraza differential fixtures

Baselines and upstream references

• ModSecurity 3.0.16
• Coraza 3.7.0
• ModSecurity initcol
• ModSecurity expirevar
• LMDB
• Zig 0.17.0-dev.1441+d5181a9c9, resolved and locked through Pantry

WAF-08 (#9) is complete and its collection contract is now available on main at 531cc68. Hosted qualification is green: https://github.com/zig-utils/zig-waf/actions/runs/29935997519

This removes the WAF-08 dependency for this issue. Preserve the published ownership, phase-gating, origin, bounded-storage, selector/error, and derived-family contracts when implementing this scope.

First implementation slice landed in 011776f (feat(engine): add revisioned persistence backend).

It adds:

• versioned backend callback table with explicit snapshot ownership and typed failure modes
• bounded IP/SESSION/USER/GLOBAL/RESOURCE namespace and mutation contracts
• optimistic expected-revision atomic commits
• expiry-aware loads and bounded cleanup
• thread-safe in-memory backend that builds replacements before publication, so validation/allocation/conflict failures cannot expose partial mutation batches
• focused atomicity, conflict, expiry, cleanup, and capacity tests

Exact pinned Zig build test and build check passed before push. Engine transaction sessions, conflict-safe arithmetic, LMDB, concurrency qualification, docs, benchmarks, and hosted release qualification remain open in this issue.

Conflict-safe transaction persistence landed in main as feat(engine): add persistent transaction sessions.

The session performs no backend I/O until explicit initialization, loads each namespace/key once, owns bounded mutation logs, flushes only dirty bindings, and retries optimistic conflicts by reloading the current revision. Ordered set, remove, add, and expire mutations are supported; numeric additions treat missing/non-numeric prior data as zero to match the pinned ModSecurity set_var.cc behavior while using checked i64 arithmetic instead of native integer overflow.

A focused two-session stale-revision test proves concurrent deltas compose to the exact stored total. Exact pinned Zig tests and compile checks passed before push.

Engine integration landed as feat(engine): wire persistent collection actions.

Delivered in this slice:

• immutable builder configuration for backend callbacks, resource limits, and fail-open/fail-closed policy
• zero-I/O transaction session creation and explicit phase-gated collection initialization
• transaction-local publication of loaded persistent values with a distinct persistent origin
• action-facing set, checked add/subtract, remove, and expire APIs
• dirty-only flush during processLogging
• observable backend failure classification (unavailable, timeout, conflict exhaustion, corruption, capacity) under both policies
• rollback of queued mutations or bindings if transaction-local publication fails
• pinned ModSecurity-compatible numeric prefix parsing with checked i64 overflow
• end-to-end engine tests for load-once behavior, mutation ordering, reload, expiry boundary, absent backend, and failure policy

Exact pinned Zig tests and compile checks passed before push. LMDB, threaded stress, housekeeping metadata, documentation, and benchmarks remain.

Pantry LMDB is now pinned on main in deps.yaml and pantry.lock as openldap.org/liblmdb@0.9.35 (chore: pin Pantry LMDB runtime). The installed artifact supplies lmdb.h, liblmdb.a, the shared library, pkg-config metadata, and maintenance tools. No system-library fallback or submodule was added. Exact pinned Zig tests and compile checks passed before push.

The real Pantry-linked LMDB backend landed as feat(engine): add LMDB persistence backend.

Key properties:

• Zig 0.17 translate-C binding generated from Pantry lmdb.h; static link to the pinned liblmdb.a
• versioned, length-delimited binary record format with strict truncation, trailing-byte, enum, field, count, and total-size validation before bounded allocation
• namespace-separated LMDB keys
• atomic write transactions that load, validate expected revision, apply the full mutation batch, and publish once
• LMDB error classification for capacity/corruption/unavailability
• bounded cursor cleanup that removes expired variables and advances record revisions
• restart/reopen durability, stale-revision non-publication, expiry cleanup, hostile-codec, and four-thread contended increment tests
• CI now installs exact Pantry 0.10.47 and openldap.org/liblmdb@0.9.35 before compiling

The complete focused suite passed three consecutive times locally with the exact pinned Zig compiler, followed by build check.

Compatibility hardening landed as fix(engine): preserve persistent expiry semantics.

Pinned ModSecurity source review showed that value replacement preserves an existing expiry and expirevar may create expiry-only state before a value exists. The shared record model, in-memory backend, LMDB codec, and mutation engine now preserve those semantics while keeping expiry-only entries invisible to rule reads.

This commit also adds checked TX setvar arithmetic coverage, LMDB corrupt-record rejection, map-full atomicity (prior revision remains readable), and a portable namespace test correction identified from hosted Linux CI logs. The focused suite passed twice locally plus compile checks before push.

Contention hardening landed as perf(engine): back off persistence conflicts. Hosted Linux and a local stress run both exposed bounded optimistic retry starvation under four contending LMDB writers. Conflict rebasing now uses a small bounded spin followed by scheduler yield; this preserves the configured retry ceiling while allowing the current writer to finish. The complete suite, including the exact 200-increment result, passed 20 consecutive local runs before push.

WAF-09 documentation and benchmark evidence landed as docs: publish persistent collection contracts.

The new contract document covers ownership, load-once lifecycle, TX and persistent mutation semantics, expiry-only state, optimistic atomicity, failure policy, LMDB deployment/format, connector logging responsibility, and the boundary with PostgreSQL fleet storage. The executable compatibility matrix now links WAF-09 to source, tests, docs, and benchmark evidence.

ReleaseFast local result (20,000 transactions):

• no persistence initialized: 4.955 µs/transaction
• initialized in-memory load + increment + flush: 13.415 µs/transaction
• transaction size: 7,824 bytes
• final counter: exactly 20,000

The default path performs no backend calls; the benchmark is now a hosted CI step. Exact pinned tests and compile checks passed before push.

Pinned action parity correction landed as fix(engine): publish persistent identity scalars. setSessionCollection, setUserCollection, and setResourceCollection now bind their persistent namespaces and publish SESSIONID, USERID, and scalar RESOURCE, matching ModSecurity set_sid.cc, set_uid.cc, and set_rsc.cc. This also corrects the canonical scalar registry from 77 to 78 names and adds end-to-end coverage.

Exact tests, compile checks, and the ReleaseFast persistence benchmark passed before push (4.506 µs no-persistence path; 12.484 µs initialized in-memory path; exact 20,000 counter).

Threaded in-memory qualification landed as test(engine): stress in-memory persistence. Four real threads repeatedly load and increment one shared revisioned record; the exact final value is 400 with zero lost updates. The full suite passed 20 consecutive runs before push, covering both in-memory and LMDB contention paths.

Backend capability discovery landed as feat(engine): discover persistence capabilities. The ABI now advertises optimistic revisions, atomic batches, per-variable expiry, bounded cleanup, durable storage, and hard-deadline support. Builders can require an exact feature set and fail configuration before publishing a WAF.

The in-memory backend advertises the four core semantics; LMDB additionally advertises durability and intentionally does not claim a hard deadline that the upstream locking API cannot guarantee. A focused builder rejection test prevents silent capability fallback. Exact tests and compile checks passed before push.

Completed on main through 0c7c533 and qualified by hosted CI: https://github.com/zig-utils/zig-waf/actions/runs/29939075574

Delivered:

• bounded TX assignment/deletion/checked numeric mutation with pinned ModSecurity prefix parsing
• persistent IP/SESSION/USER/GLOBAL/RESOURCE load-once transaction sessions
• setuid/setsid/setrsc key binding plus USERID/SESSIONID/RESOURCE scalars
• assignment, deletion, atomic delta, expiry, expiry-only state, lazy visibility, and bounded cleanup
• expected-revision atomic mutation batches with bounded retry/backoff and observable exhaustion
• explicit fail-open/fail-closed classifications and backend capability discovery/rejection
• thread-safe atomic in-memory backend
• Pantry-pinned LMDB 0.9.35, Zig translate-C binding, bounded versioned binary format, static link, atomic transactions, durable reopen, cleanup, corruption and map-full handling
• real-thread exact-total stress for both backends, restart, stale-revision, expiry, capacity, malformed data, and no-backend/no-I/O tests
• ownership/deployment documentation and machine-readable compatibility evidence

ReleaseFast evidence at the issue boundary (20,000 transactions): 4.955 µs/transaction with persistence unused, 13.415 µs/transaction for in-memory load + increment + flush, 7,824-byte transaction state, exact final counter 20,000. A later scalar-binding correction increased transaction state to 7,920 bytes and measured 4.506/12.484 µs respectively.

LMDB does not silently claim a hard deadline unsupported by upstream locking. Capability discovery reports that absence, and configurations requiring hard deadlines fail before WAF publication. Timeout-capable backends use the same typed failure policy.

Hosted Linux passed format, 59+ unit/integration/C smoke tests, compile, ownership/lifecycle/scalar/collection/persistence benchmarks, locked sibling verification, no-submodule enforcement, Pantry 0.10.47 installation, and exact LMDB 0.9.35 installation.

Post-close reliability fix pushed as HEAD (fix(engine): reload revisions after conflict backoff).

Hosted runs exposed rare RetryLimitExceeded failures in both in-memory and LMDB contention tests. The retry path previously loaded the newest revision and then backed off, allowing active writers to stale that revision before the next commit. Backoff now occurs before reload, so each retry commits from the freshest revision.

Evidence: 20 consecutive exact pinned Zig 0.17-dev full test-suite runs passed locally, followed by compile and diff checks. The configured retry bound and explicit exhaustion error remain unchanged.

Stress-harness semantics tightened in HEAD (test(engine): retry dirty sessions after exhaustion).

RetryLimitExceeded is an explicit, non-destructive result: the session remains dirty and may be flushed again. The concurrent workers now distinguish that documented transient outcome from allocator/backend failures and make at most eight outer attempts; no increment is reapplied or duplicated. Twenty-five consecutive full-suite runs passed after the change, covering both in-memory and LMDB contention.