[WAF-11] Compile immutable, cache-efficient execution plans

Objective

Compile fully parsed SecLang documents into an immutable, thread-safe, cache-efficient execution plan. WAF-11 owns structural compilation, deterministic ordering, phase indexes, default snapshots, chain graphs, string/literal deduplication, safe publication, and plan introspection. WAF-12 and the action/operator issues own the eventual complete semantic registry and execution behavior.

No request may interpret source text, traverse parser AST ownership, allocate a rule graph, access PostgreSQL/UI state, or perform blocking logging work.

Compilation and ownership contract

• plan.Compiler consumes one or more completely parsed seclang.Document values plus their stable source registry.
• Compilation is transactional: it returns one owned Plan or an error diagnostic; no partial plan is publishable.
• The plan owns every string, selector, operator parameter, action value, macro program, literal, index, and graph edge required after parser inputs are destroyed.
• All published plan memory is immutable. Per-request mutable operator state belongs to a worker/request scratch object, never the shared plan.
• Use compact typed IDs and contiguous slices instead of pointer-rich per-rule allocations.
• Source ordering is deterministic across documents and directives; rule/action/target order is preserved exactly.
• Unknown generic directives remain available as plan metadata for WAF-12 validation. Reduced builds must not silently drop them.

Structural IR

The public plan exposes stable typed identifiers and bounded introspection for:

• Interned strings and source references.
• Directives and rule source order.
• Five phase-specific rule indexes.
• Rules with target spans, operator descriptor, transformation pipeline, action range, metadata, effective phase, and chain position.
• Target collection/modifier/selector descriptors.
• Operator name/parameter/negation plus optional literal-prefilter descriptor.
• Actions split into transformations, metadata/non-disruptive, disruptive/flow, and unknown descriptors without prematurely claiming semantic support.
• SecDefaultAction snapshots applied only to following rules in the compatible scope.
• Chain graphs with explicit head/member/next relationships and validation of unterminated or structurally invalid chains.
• Markers and generic directives addressable for later skip/update/removal compilation.

Required structural semantics

• Parse numeric id and phase values strictly and reject overflow/trailing bytes.
• Reject duplicate explicit rule IDs with primary and secondary source spans.
• Resolve effective phase from explicit rule actions, then compatible defaults, then phase 2.
• Preserve default action ordering while allowing explicit rule actions to replace/append according to later semantic classification; WAF-12 may refine directive-specific validity without rebuilding source order.
• A chain action links the immediately following SecRule. Chains cannot cross non-rule directives or document boundaries and cannot end without a member.
• Only chain heads are entered in a phase index; every member records its head and next member.
• All members of a chain must compile into the head's effective phase or fail structurally.
• Compile action t: entries into an ordered transformation pipeline, including t:none reset behavior, while leaving transformation availability to WAF-16.
• Precompute safe literal candidates for exact/prefix/contains/phrase operators and provably literal regexes. Absence of a prefilter must never change matching behavior.
• Intern equal strings and literal blobs once per component and expose measured deduplication.

Structural sharing and publication

• A compiler may reuse immutable components from a prior plan only when their canonical content identity matches exactly.
• Shared components use explicit thread-safe lifetime accounting; destroying one generation cannot invalidate another.
• Plan fingerprints include normalized structural content and compiler ABI/version, not allocator addresses or filesystem enumeration order.
• Waf.Builder accepts a compiled plan and transfers or retains ownership explicitly.
• Runtime reload publishes only a completely built Waf/plan generation. Existing transactions remain pinned to their original generation; new transactions observe the replacement.
• Publication/retirement has no use-after-free window and no request-path plan mutation.
• Compilation and reload failures leave the active generation unchanged.

Resource and security limits

Distinct errors cover maximum:

• documents, directives, rules, rules per phase, targets, actions, transformations, chain members, generic directives, markers, and source references;
• interned string count, individual string/literal bytes, aggregate owned plan bytes, prefilter count/bytes, and graph edges;
• numeric parsing, typed-ID overflow, duplicate IDs, malformed phase/default/chain structure, and unsupported compiler ABI.

All size arithmetic is checked. No recursion is proportional to rule/action/chain count. Malformed inputs produce source-anchored diagnostics. Allocation failure unwinds every owned component.

Deliverables

• src/plan.zig: immutable IR, compiler, limits, diagnostics, fingerprints, and introspection.
• Engine builder/runtime integration with explicit plan ownership and generation pinning.
• Focused fixtures for defaults, phases, targets, actions, transformations, macros, chains, markers, generic directives, duplicates, and malformed structures.
• CRS 4.28 structural compilation gate after the parser corpus gate.
• Allocation-failure injection, deterministic fuzz/mutation entry point, real-thread sharing/reload stress, and repeated compile/deinit leak checks.
• ReleaseFast benchmarks for small and CRS plans reporting compile time, rules/directives per second, owned bytes, bytes/rule, deduplication, phase traversal cost, and reload pause.
• Documentation and machine-readable evidence linked to source tests.

Acceptance criteria

• All 58 pinned ModSecurity/Coraza/CRS configuration files remain parseable and deployable CRS configuration compiles structurally with zero unexplained exclusions.
• Every compiled source span resolves after caller parser inputs are destroyed.
• Deterministic plan fingerprints and indexes are identical for identical ordered input.
• Duplicate IDs, invalid phases, numeric overflow, dangling chains, cross-boundary chains, phase disagreement, and all configured limits fail explicitly.
• Five phase indexes contain exactly their eligible chain heads in source order.
• Default snapshots and t:none behavior match pinned baseline fixtures.
• Literal prefilters are conservative and differential tests prove they never introduce false negatives.
• At least two simultaneous generations share eligible immutable components and can be destroyed in either order under real threads.
• Active-generation publication is atomic at the API boundary; failed builds/reloads do not disturb the current generation.
• Exact Pantry-pinned Zig formatting, tests, compile checks, corpus compile gate, fuzzing, and benchmarks pass locally and in hosted CI.
• Implementation lands directly on main as small conventional commits with issue-linked evidence.

Relationships

• Parent: WAF-00

Blocked by

• WAF-10 — complete

Blocks

• WAF-12 — complete stable directive validation
• WAF-14 — metadata/non-disruptive actions
• WAF-15 — disruptive/flow/runtime actions
• WAF-16 — transformations
• WAF-17 — numeric/string/regex operators
• WAF-18 — phrase/dataset/IP operators
• WAF-19 — injection operators
• WAF-20 — validation/external operators
• WAF-21 — Lua/exec/mutation actions
• WAF-22 — plugin registries
• WAF-33 — explain/test/benchmark tools
• WAF-53 — signed policy bundle rollout

Pinned baselines

• ModSecurity 3.0.16
• Coraza 3.7.0
• OWASP CRS 4.28.0
• Zig 0.17.0-dev.1441+d5181a9c9, locked through Pantry

Dependency WAF-10 #11 is complete at e72709c with hosted green evidence: https://github.com/zig-utils/zig-waf/actions/runs/29942330646. This issue is now unblocked on the bounded, owned SecLang frontend.

Implemented and pushed HEAD (feat(engine): add immutable plan storage).

The first WAF-11 slice adds:

• compact typed IDs and contiguous directive/argument/rule/target/action arrays;
• a deduplicated string table stored as ranges into one byte blob;
• plan-owned source bytes, line indexes, paths, and include origins;
• source slice/location lookup that remains valid after parser ownership is destroyed;
• five phase index slices (rules default structurally to phase 2 until the next defaults/phase commit);
• checked typed-index, source, string, aggregate storage, and object-count limits;
• transactional compiler cleanup and explicit plan ownership/deinit.

Tests destroy the parsed source immediately after compilation and continue resolving plan strings, source slices, locations, targets, operators, and phase entries. Focused limit and deduplication tests plus exact pinned Zig tests/checks are green.

Implemented and pushed HEAD (feat(engine): compile phases defaults and chains).

Structural compilation now includes:

• strict 64-bit rule IDs with overflow and duplicate rejection;
• explicit/default/inherited phases with five ordered head-only indexes;
• owned SecDefaultAction snapshots;
• immediate chain graphs with head/next/position metadata;
• rejection of dangling, cross-directive, cross-document, and phase-disagreeing chains;
• ordered default plus explicit transformation pipelines with compatible t:none reset behavior;
• action ranges on SecRule, SecAction, and SecDefaultAction directives;
• distinct count limits for defaults and transformations.

Focused fixtures cover default phase inheritance, explicit phase override, chain-member inheritance, head-only indexes, ID presence, transformation reset, duplicate/overflow IDs, invalid phase, dangling chains, directive boundary crossing, and chain phase mismatch. Exact pinned Zig tests/checks pass.

Pushed two compatibility increments:

• e18c179 accepts the pinned baseline's named phase:request alias (and compatible response/logging aliases) with strict numeric fallback.
• HEAD adds a repeatable structural plan corpus mode and build gate.

Exact local result across all five pinned roots:

parser corpus roots=5 files=58 bytes=803184 directives=925 plans=58 rules=761 plan_owned_bytes=2243969

This uncovered and fixed the named-phase incompatibility before CI gating. There are zero corpus compilation exclusions.

Implemented source-anchored structural compiler diagnostics in 507a925 (feat(engine): diagnose structural plan failures). Added stable WAF-PLAN-* codes and a transactional compileOutcome API. Duplicate IDs now report the later action as primary and the original definition as related; invalid IDs/phases/transformations and dangling or phase-inconsistent chains report precise owned source spans. Legacy typed-error compilation remains available. Exact Zig 0.17-dev unit tests and compile checks pass.

Compiled conservative literal prefilters in dc2e37b (perf(engine): compile conservative literal prefilters). Plans now carry bounded exact/prefix/suffix/contains/phrase-set descriptors, deduplicated literal IDs, explicit prefilter count/byte limits, and a rejection-only prefilterMayMatch API. Negated operators and ambiguous phrase/regex syntax deliberately receive no prefilter; only provably literal regexes are eligible. Focused tests cover every descriptor, unsafe absence cases, phrase any-match behavior, and resource-limit failures. Exact Zig unit and compile checks pass.

Added the remaining addressable structural metadata in 164a45c (feat(engine): index structural plan metadata): deterministic marker and generic-directive indexes, stable action-family classification (transformation, metadata, non-disruptive, disruptive, flow, unknown), and distinct bounds for source references, rules per phase, chain members, graph edges, markers, and generic directives. Tests prove every new bound fails with its own typed error and unknown actions remain preserved. Exact Zig unit and compile checks pass.

Added canonical compiler-versioned plan fingerprints in 35e4bd6 (feat(engine): fingerprint compiled plans). The BLAKE3 identity is domain-separated by compiler ABI and hashes typed semantic fields plus resolved string bytes field-by-field. It excludes allocator addresses, struct padding, filesystem paths, and source offsets. Tests prove identical ordered inputs at unrelated paths produce identical fingerprints while semantic changes do not. Exact Zig unit and compile checks pass.

Implemented thread-safe immutable payload sharing in bea5061 (feat(engine): share immutable plan generations). Each plan generation is now a lightweight handle over an atomically reference-counted immutable component. compileWithPrevious uses the public fingerprint only as a fast reject, then performs field-wise equality over every owned structural/source/index field before reuse. Tests cover both destruction orders, non-equivalent rejection, and concurrent real-thread reads while the sibling generation retires. Exact clean-cache Zig unit and compile checks pass.

Bound compiled plans to immutable WAF generations in 3a6f51a (feat(engine): bind plans to waf generations). Builder.setRetainedPlan and buildTransferringPlan make ownership explicit; failed validation preserves caller ownership. WAF teardown releases its plan only after transactions drain, transactions expose their pinned plan, and runtime reload keeps old/new plan fingerprints isolated across generations. Feature discovery now reports compiled-plan support. Exact Zig unit and compile checks pass.

Added exhaustive allocation-failure and repeated lifetime checks in ed5767f (test(engine): exhaust plan allocation failures). The test first measures every allocator call in a representative defaults/chains/prefilters/markers plan, injects OOM at each allocation index, and asserts allocated bytes equal freed bytes. A 250-generation compile/deinit loop adds leak and teardown coverage. Exact Zig tests pass.

Added the reusable structural plan fuzz oracle and deterministic runner in 806dda9 (test(engine): fuzz structural plan compilation). The oracle accepts parser/semantic diagnostics only with valid spans, then checks compiler ABI/fingerprint, source spans, typed ranges, ordered phase indexes, chain invariants, and conservative prefilter invariants for every successful plan. The pinned 10,000-case mutation run passes at seed 11936128518282651045; unit and compile checks also pass.

Added the ReleaseFast structural-plan benchmark in 7883907 (perf(engine): benchmark compiled plan lifecycle). It reports compile time and plans/directives/rules per second; owned bytes and bytes/rule; unique/interned/referenced/deduplicated string bytes; phase-index traversal throughput; identical-generation reuse compile time; and isolated publication totals/average over 1,000 reloads. Local pinned-Zig sample: 137,295 plans/s, 411,886 rules/s, 2,587 owned bytes, 109 deduplicated string bytes, 2.708B indexed rules/s traversal, 8,584 ns reuse compile, and 4 ns average publication.

Closed the request-time macro parsing gap in 2275bb6 (feat(engine): compile immutable macro programs). Operator/action %{…} values now reference bounded, deduplicated immutable programs containing literal/scalar/collection tokens and pre-split names/keys; unresolved names remain structural for WAF-12 validation. Dynamic operator parameters cannot receive unsafe literal prefilters. Added distinct program/token limits, source-anchored malformed-macro diagnostics, fingerprint/sharing coverage, fuzz invariants, and compiler ABI v2. Exact unit/check/fuzz/ReleaseFast benchmark gates pass.

Published the WAF-11 contract and machine-readable evidence in 2290616, added hosted plan fuzz/benchmark gates in e5dd843, recorded green hosted measurements in 33825c6, and added a pinned 10,000-input differential mutation test proving every actual exact/prefix/suffix/contains/phrase/literal-regex match survives its advisory prefilter in 19b64bc. Full boundary run 29945716010 is green; the exact final-SHA run is now in progress at https://github.com/zig-utils/zig-waf/actions/runs/29946143469.

WAF-11 is complete at 19b64bc6f449653aa423b6fa41882477d7029f79 with exact hosted CI green: https://github.com/zig-utils/zig-waf/actions/runs/29946143469

Delivered: owned compact typed IR; five ordered phase indexes; defaults, chains, IDs, transformations, action families, markers and generic directives; conservative prefilters with 10,000-case differential mutation safety; immutable macro programs; stable source-anchored diagnostics; compiler-ABI-v2 BLAKE3 fingerprints; field-verified atomic component sharing; explicit Builder retain/transfer ownership; WAF/transaction generation pinning; allocation-failure injection; real-thread lifetime stress; deterministic plan fuzzing; CRS ReleaseFast metrics; docs and embedded evidence.

Pinned corpus result: 58 plans, 925 directives, 761 rules, 2,419,595 plan-owned bytes, zero unexplained exclusions. WAF-12 owns complete directive-specific semantic validation and may now build on this boundary.