[WAF-16] Implement all stable transformations and aliases

Objective

Implement the complete stable ModSecurity 3.0.16 / Coraza 3.7.0 transformation union as typed, immutable execution plans with byte-exact behavior, case-insensitive aliases, t:none reset semantics, ordered pipelines, multi-match checkpoints, bounded scratch storage, and executable differential evidence. Transformation execution must remain independent from UI, PostgreSQL, audit delivery, and all blocking I/O.

Stable union

Encoding, decoding, and representation

• base64Decode, base64DecodeExt, base64Encode
• hexDecode, hexEncode, sqlHexDecode
• urlDecode, urlDecodeUni, urlEncode
• htmlEntityDecode, cssDecode, jsDecode, escapeSeqDecode
• utf8toUnicode
• length

Canonicalization and filtering

• lowercase, uppercase
• trim, trimLeft, trimRight
• compressWhitespace, removeWhitespace
• removeNulls, replaceNulls
• removeComments, removeCommentsChar, replaceComments
• cmdLine
• normalisePath, normalisePathWin
• Accept normalizePath and normalizePathWin as case-insensitive aliases that compile to the canonical kinds.
• parityEven7bit, parityOdd7bit, parityZero7bit

Compatibility digests and pipeline control

• md5, sha1; document that these are byte-compatible SecLang transformations, not approved password/signature primitives.
• none as a compile-time pipeline reset only; it must never become a runtime transform step.

Typed plan and registry

• Replace free-form transformation names in published plans with a stable TransformationKind enum and canonical feature-discovery names.
• Validate every configured transformation before ruleset publication. Unknown names and unavailable reduced-build features fail with the stable plan diagnostic; no runtime lookup fallback.
• Preserve declared order after default inheritance, t:none, and SecRuleUpdateActionById overlays.
• Preserve chain-member pipelines independently while retaining WAF-15 chain-head action/intervention semantics.
• Keep aliases case-insensitive at configuration time and canonical in fingerprints, serialization, evidence, and benchmarks.
• Define explicit native/plugin registration boundaries without weakening immutable-plan ownership; plugin execution itself remains linked to WAF-22.

Execution API and ownership

• Add a reusable per-worker/per-transaction transformation executor with bounded scratch buffers and no global mutable state.
• Return explicit bytes, changed/not-changed state, and ownership/lifetime information so unchanged transforms may safely borrow input and changed transforms cannot outlive scratch storage.
• Preserve embedded NULs and arbitrary byte strings; never assume UTF-8 unless a specific transformation requires UTF-8/code-point decoding.
• Use overflow-checked output sizing and deterministic limits for input bytes, output bytes, pipeline steps, cumulative expanded bytes, and cache storage.
• Make allocation, invalid input, limit exhaustion, and plugin failure distinguishable. Compatibility transforms with tolerant malformed-input semantics must return their specified bytes rather than being conflated with engine failure.
• Ensure partial pipeline failure publishes no operator match, capture, multi-match checkpoint, action, or intervention state.

Byte semantics

• Match the pinned upstream handling of malformed/incomplete percent, base64, hex, entity, CSS, JavaScript, C escape, SQL hex, and UTF-8 sequences.
• Match upstream path normalization for repeated separators, dot segments, root traversal, trailing separators, and Windows slash conversion.
• Match exact comment state machines, including unterminated comments and comment-marker edge cases.
• Match ASCII-only case and whitespace definitions where upstream is byte-oriented; do not introduce locale-sensitive behavior.
• Match digest output representation and exact length semantics.
• Define and test expansion bounds for URL encoding and utf8toUnicode.

Pipeline, multiMatch, and cache semantics

• Execute transformations in published plan order and propagate changed-buffer state without redundant copies.
• Implement WAF-15 multiMatch checkpoints at the original value and after each transformation that changed the value, without duplicate evaluation for unchanged steps.
• Retain the correct transformed value and source/origin metadata for operator evaluation, captures, rule evidence, and replay.
• Implement bounded transaction/worker transformation caching where SecCacheTransformations enables it; cache keys include transformation kind/pipeline identity and exact input bytes.
• Keep cache misses correct under collisions and disable caching explicitly when limits are exhausted rather than bypassing transformations.
• Never share mutable scratch, digest state, or cache entries across request workers without synchronization.

Evidence and qualification

• Build a generated inventory from the pinned ModSecurity scanner/parser and Coraza registry; fail CI if the stable union drifts.
• Import or adapt pinned Coraza transformation JSON fixtures and relevant ModSecurity regression fixtures with retained Apache/BSD notices.
• Add differential vectors for every stable kind and alias, including empty, unchanged, malformed, binary, maximum-bound, and expansion cases.
• Add pipeline/default/overlay/chain/multiMatch integration tests and exhaustive allocation-failure rollback tests.
• Add deterministic fuzzing for every decoder/state machine plus random ordered pipelines under output/work limits.
• Add ReleaseFast benchmarks for no-op/borrowed steps, common CRS pipelines, decoder-heavy inputs, path normalization, digest transforms, multiMatch, and cache hit/miss behavior.
• Record throughput, p50/p95/p99, allocations, input/output bytes, and cache behavior in an executable evidence manifest.
• Update feature discovery, the compatibility matrix, architecture documentation, and operator-facing transformation documentation only after executable evidence passes.

Acceptance criteria

• Every stable union name above resolves case-insensitively to a typed implementation or canonical alias and has positive, malformed, boundary, and differential evidence.
• All pinned Coraza transformation fixture cases and relevant ModSecurity/CRS transformation cases pass with no unexplained mismatches.
• t:none, inherited defaults, overlays, chains, and multiMatch match the pinned behavior.
• Unknown or unavailable transformations reject the ruleset before publication; no silent no-op or runtime string dispatch exists.
• Untrusted work and output growth have deterministic limits; allocation/limit failures leave match/action state unchanged.
• Common unchanged transforms avoid output allocation, and measured hot pipelines have recorded ReleaseFast gates suitable for later WAF-39 thresholds.
• Formatting, focused tests, inventory/differential corpora, deterministic fuzzing, benchmarks, and hosted CI pass on the exact closing SHA.
• Work lands directly on main in small conventional commits with issue/commit links, no submodules, and no unrelated checkout changes.

Relationships

• Parent: WAF-00 #1

Blocked by

• WAF-11 #12 — immutable execution plans (complete)
• WAF-14 #15 — atomic match/action state (complete)
• WAF-15 #16 — chain/multiMatch execution metadata (complete)

Blocks / feeds

• WAF-17 #18 — numeric/string/regex operators
• WAF-19 #20 — SQLi/XSS operators
• WAF-22 #23 — native Zig and C plugin registries
• WAF-33 #34 — validate/test/explain/replay tooling
• WAF-34 #35 — SecRules Test Set
• WAF-35 #36 — ModSecurity differential regression
• WAF-36 #37 — Coraza differential and HTTP E2E
• WAF-37 #38 — CRS FTW compatibility
• WAF-39 #40 — request-path performance gates

Pinned baselines

• ModSecurity 3.0.16 at 7ea9fef
• Coraza 3.7.0 at 27069d0
• OWASP CRS 4.28.0 at 55b09f5
• Zig 0.17.0-dev.1441+d5181a9 through Pantry

Dependency update: WAF-11 #12 is complete at 19b64bc with exact hosted CI green. This issue can now consume the immutable typed IR, stable diagnostics, prefilters/macro programs, fingerprints, shared payloads, and explicit WAF plan ownership contract.

First WAF-16 slice landed on main as 4f67721.

Delivered:

• typed 35-kind stable ModSecurity/Coraza transformation inventory;
• compile-time enum/spec ordering invariant and case-insensitive resolution tests;
• canonical normalizePath/normalizePathWin aliases;
• none retained strictly as a compile-time reset;
• unknown transformations rejected before plan publication;
• transformation action values and published pipelines canonicalized, with macros excluded;
• plan ABI advanced to 7 and all executable evidence pins updated.

Local Pantry Zig 0.17 qualification:

• full suite: 221/221 pass;
• pinned five-root structural corpus: 58 files, 925 directives, 761 rules.

Awaiting hosted CI on the exact SHA while execution semantics proceed in small family commits.

Bounded execution slice landed on main as 01d6ef6.

Implemented with byte-preserving, binary-safe semantics:

• reusable two-slot executor scratch with explicit borrowed/executor lifetime tags;
• separate upstream changed semantics (including length, parity, and single-whitespace compression edge behavior);
• input/output/pipeline/cumulative limit contracts and explicit errors;
• zero-copy unchanged ASCII case and trim paths;
• lowercase/uppercase, trim variants, whitespace compression/removal, NUL removal/replacement, all three parity transforms, and length;
• exhaustive scratch allocation-failure ownership coverage.

Focused transformation suite: 5/5 pass on Pantry Zig 0.17. The preceding typed-plan slice passed 221/221 plus the pinned five-root plan corpus; hosted CI is running on the new exact SHA.

Base64 family landed on main as 18dcb2e. The executor now distinguishes strict ModSecurity decoding, Coraza partial decoding, ModSecurity fully forgiving extended decoding, and Coraza whitespace/dot-forgiving extended decoding; it also preserves embedded NUL output, optional padding, profile-specific empty encode flags, bounded sizing, and binary-safe standard encoding. Focused 8/8 and full 224-test suites pass locally.

The complete hex family landed on main: hexDecode/hexEncode in 8810a06 and pinned ModSecurity-compatible sqlHexDecode in ac2a44b. SQL hex decoding recognizes case-insensitive 0x prefixes only when followed by a complete byte, consumes complete hexadecimal runs, preserves malformed prefixes byte-for-byte, supports embedded NUL/non-UTF-8 output, and stays inside the executor bounds. Focused 9/9 and the full repository test suite pass locally.

The complete URL family landed on main: byte-oriented urlDecode/urlEncode began in 8810a06, and urlDecodeUni plus profile exactness landed in c0c9ca1. The executor now handles standard %xx, IIS %uXXXX, +, malformed/incomplete escapes, embedded binary output, full-width ASCII folding, and an immutable optional ModSecurity Unicode map whose backing storage is explicitly borrowed. Coraza malformed-percent changed flags remain distinct from ModSecurity without forcing a copied result. Focused 10/10 and the full repository suite pass locally.

Pinned CSS decoding landed on main as 33925fc. It bounds escapes to six hex digits, consumes exactly one following ASCII whitespace byte, preserves embedded binary output, applies the pinned full-width folding rules for four/five/six-digit forms, handles continuation-to-newline and terminal backslash cases, and preserves the ModSecurity/Coraza changed-flag divergence for non-hex escaped literals. Focused 11/11 and the full suite pass locally. The grouped HTML/CSS/JS/C-escape acceptance item remains open until its other three members land.

ANSI/C escapeSeqDecode landed on main as bca9377. The binary-safe state machine covers simple escapes, exact two-digit hex, one-to-three-digit octal with byte truncation, malformed/unknown escape removal, and terminal backslash preservation. Focused 12/12 and the full suite pass locally; the grouped decoder item remains open for JS and HTML entity decoding.

JavaScript escape decoding landed on main as ebef6e1. It covers lower-byte \\uHHHH, \\xHH, bounded octal with the pinned three-digit overflow rule, full-width ASCII folding, simple/unknown escaped literals, malformed prefixes, embedded binary, and terminal backslash semantics. Focused 13/13 and the full suite pass locally. HTML entity decoding remains before the grouped decoder item can close.

The grouped HTML/CSS/JavaScript/C-escape family is complete. A reproducible Go 1.25 HTML5 inventory (2,229 names, including paired code points) with pinned SHA-256 and retained BSD notice landed in 891fed4; profiled entity execution landed in 78691f3. ModSecurity mode preserves its five case-insensitive prefix entities, low-byte numeric conversion, semicolon tolerance, and length-based changed flag. Coraza mode preserves full case-sensitive HTML5 names, legacy no-semicolon prefix fallback, paired Unicode output, Windows-1252 numeric replacements, and invalid-code-point replacement. Together with CSS 33925fc, C escapes bca9377, and JS escapes ebef6e1, focused 16/16 and the full suite pass locally.

utf8toUnicode landed on main as 5638f2e. Coraza mode uses strict RFC-compatible UTF-8 decoding, emits lowercase %u forms, and converts every malformed byte to %ufffd; ModSecurity mode preserves the pinned permissive two/three/four-byte state machine, overlong and surrogate artifacts, invalid/truncated lead handling, interior-NUL behavior, and changed flags. Both paths preflight exact output size with checked arithmetic, so a six-byte limit admits an actual six-byte escape and rejects true expansion beyond it. Focused 18/18 and the full suite pass locally.

All three comment transforms landed on main as 27ee5c3. removeComments preserves the pinned block/HTML shared close-state machine, line-comment truncation, unterminated-space sentinel, and terminal closed-comment NUL sentinel; removeCommentsChar removes marker tokens without removing their content; replaceComments replaces complete and unterminated C-style blocks with one space. Marker precedence, embedded NULs, closer-only input, and changed flags are covered. Focused 19/19 and the full suite pass locally.

cmdLine landed on main as 5b78517. It removes quote/backslash/caret bytes, collapses command separators and whitespace, removes pending space before / and (, lowercases ASCII only, preserves arbitrary non-ASCII bytes, and keeps the pinned changed-state distinction: ModSecurity reports only length change while Coraza reports any triggering input byte. Focused 20/20 and the full suite pass locally.

Path canonicalization landed on main as a525d21. normalisePath and normalisePathWin now collapse separators, eliminate self segments, resolve backreferences without crossing absolute roots, preserve unresolved relative parents, retain embedded binary bytes, and handle trailing separators. Profile tests cover Coraza’s filepath.Clean wrapper behavior (including root doubling and mapped-Windows changed flags) separately from ModSecurity. The previously landed typed resolver in 4f67721 accepts both American-spelling aliases case-insensitively and serializes only canonical British names. Focused 21/21 and the full suite pass locally.

The stable built-in union is now compile-time exhaustive. Raw binary md5 and sha1 compatibility transforms landed in d944835, including empty and binary upstream vectors and exact 16/20-byte output bounds. The operator contract and explicit prohibition on using these broken hashes for passwords, signatures, tokens, certificates, bundle integrity, or new security designs landed in adbe9ac. Focused 22/22 and the full suite pass locally.

Ordered execution landed in two chunks: bounded scratch/checkpoint staging in 50fa5f3, and immutable-plan/transaction integration in a855565. Pipelines execute in published order without losing executor-buffer provenance; cumulative work and step counts are bounded; multiMatch stages the original value plus every upstream-reported changed step (including changed-but-byte-equal cases); chain members retain independent t:none-resolved pipelines; transformed values retain original name/source metadata. Any transform, allocation, or limit failure returns no checkpoints and cannot publish match, capture, action, intent, or intervention state. Focused 24/24, exhaustive allocation-failure coverage, member/head integration tests, and the full repository suite pass locally.

SecCacheTransformations execution landed on main as 1cc784f, with the operator contract in a0e93e0. The transaction-local LRU keys the full typed pipeline, exact input bytes, and multiMatch; hash candidates require complete kind/byte equality. Cached outputs retain staged checkpoints. Entry count and logical storage bytes are bounded, LRU eviction is measured, and an individually oversized entry or cache-only allocation failure clears the cache and enters an explicit terminal cache status while the transformation result still succeeds. Ruleset SecCacheTransformations On and builder override behavior are wired before transaction creation. Focused 26/26 cache/pipeline tests and the full repository suite pass locally.

Pinned Coraza 3.7.0 transformation evidence landed on main in two reviewable commits:

• e5d8f08 retains all 35 upstream JSON files byte-for-byte, the upstream Apache-2.0 license, source revision, and SHA-256 manifest.
• af825de adds the dedicated Zig 0.17 build-test boundary, verifies all 35 fixture digests, reproduces Coraza fixture byte decoding, and executes all 355 cases under the Coraza profile.

zig build test --summary all passes with the new 2/2 corpus/digest tests alongside the cached 248-test core suite and C ABI smoke test. No fixture was removed, edited, or silently skipped.

Upstream transformation drift is now executable and hosted-CI enforced:

• da66ee8 exposes the two compatibility aliases as immutable typed registry data with compile-time collision checks.
• c99ef7d scans the pinned ModSecurity lexer spellings, verifies all 36 parser token bindings, scans Coraza registrations, and compares both upstream inventories against the typed registry.
• 3a80680 makes the exact pinned-source comparison a required hosted CI step.

Current evidence: ModSecurity 38 accepted names / 36 parser tokens, Coraza 34 registered names, zig-waf 35 canonical kinds + 2 aliases + none, with zero mismatches. The full local suite remains green at 250/250 tests.

Bounded sizing and the WAF-22 handoff are now explicit:

• fc36644 replaces conservative Base64/URL sizing with safe real-output bounds, adds checked two-pass sizing for Coraza HTML5 entities, and proves tight success/failure limits without partial writes.
• 090dc06 reserves a distinct PluginFailure, exposes stable configuration/allocation/input/output/work/plugin policy classes, and fixes builder-only registry freezing, typed slots, generation binding, hot-reload lifetime, callback borrowing/copying, and reduced-build rejection as the required WAF-22 boundary.

Focused transformation coverage is now 27/27; the full repository suite was green at 250/250 before the additive failure-policy test.

The final WAF-16 qualification slices landed on main, closing every remaining byte-semantics and evidence item:

• d036577 — ReleaseFast bench-transformations harness covering borrowed no-op, decoder-heavy, path normalization, digest, CRS pipeline, multiMatch, and cache hit/miss paths.
• cf0e98e — per-kind differential vectors (tests/transformation_differential.zig) asserting byte output and changed across empty, unchanged, malformed, binary, and expansion categories, a completeness check that every stable kind and both aliases are exercised, and a maximum-bound invariant proving deterministic, allocation-bounded output for every kind under both profiles.
• 0d56997 — compatibility matrix engine.transformations entry plus operator and architecture documentation of the qualification evidence.
• 040f283 — p50/p95/p99 per-operation latency for the CRS pipeline and cache-miss paths, alongside allocation counts, output byte sizes, and cache hit/miss/eviction counters in the executable manifest.

Evidence recap:

• 355-case retained Coraza 3.7.0 corpus (SHA-256 pinned) plus the new per-kind differential vectors pass under both profiles.
• Byte semantics for malformed percent/base64/hex/entity/CSS/JS/C-escape/SQL-hex/UTF-8, path normalization, comment state machines, ASCII case/whitespace, and digest output/length are covered by the focused suite, retained corpus, and differential vectors.
• Deterministic fuzzing exercises every decoder and random ordered pipelines under output/work limits.
• The manifest records borrowed no-op and warmed steady-state CRS pipeline at zero per-operation allocations, and bounded LRU eviction.

Integration coverage for t:none resets, phase defaults, SecRuleUpdateActionById overlays, chain-member pipelines, multiMatch checkpoints, and exhaustive allocation-failure rollback lives in src/plan.zig and src/transformations.zig.

zig build test is green locally (256 tests) with zig fmt --check clean; hosted CI runs the inventory, corpus, differential, and fuzz steps on the exact closing SHA. Plugin transformation execution remains scoped to WAF-22 #23, and request-path percentile release gates to WAF-39 #40.

Complete, with the inventory check as the evidence that the union is actually the union.

zig build test-transformation-inventory compares the typed registry against the pinned upstream sources themselves — ModSecurity 3.0.16's seclang-scanner.ll and seclang-parser.yy, and Coraza 3.7.0's transformations.go — rather than against a list someone transcribed. Run just now:

transformation inventory modsecurity_names=38 modsecurity_tokens=36 coraza_names=34 \
  canonical=35 aliases=2 reset=1 mismatches=0

Zero mismatches: every stable transformation either has a canonical implementation, is an alias of one, or is none.

• Typed, immutable plans — transformations.Kind is an enum, specs is index-locked to it by a comptime assertion, so a registry that drifts out of enum order fails to compile rather than mis-dispatching at runtime.
• Case-insensitive aliases — resolution is eqlIgnoreCase over canonical names then aliases, and comptime checks reject an alias that duplicates a canonical name or another alias.
• t:none reset semantics — resolve returns .reset for none, distinct from any builtin, so a reset clears the pipeline rather than being one more step in it.
• Ordered pipelines and multi-match checkpoints — PipelineResult and Checkpoint record the intermediate value at each step, which is what t: chains with multi-match need in order to test every stage.
• Bounded scratch storage — Storage (borrowed / two executor buffers / cache) with Limits, so a chain of expanding transformations has a deterministic ceiling and cannot allocate unboundedly. CacheStats reports limit_exhausted and allocation_failed distinctly, so exhaustion is visible rather than silent.
• Byte-exact behavior — Result.changed follows upstream semantics rather than byte equality (length and the parity transforms report a change; a single compressed tab becoming a space reports none), which is where a naive implementation diverges.
• Executable differential evidence — tests/transformation_differential.zig (exact-output vectors per kind and alias, plus per-kind maximum-bound and expansion invariants) and tests/transformation_evidence.zig, both in the default zig build test.
• Independent of UI, PostgreSQL, and blocking I/O — src/transformations.zig imports only std; the database layer is a separate build.

Documented in docs/transformations.md.