[WAF-10] Implement the SecLang lexer, parser, diagnostics, and includes

Objective

Implement a bounded, source-preserving SecLang frontend compatible with the stable syntax accepted by ModSecurity 3.0.16 and Coraza 3.7.0. The frontend must lex logical lines, parse directives and rule structure, resolve local include trees safely, reject trailing or malformed input, and emit actionable diagnostics without performing semantic directive compilation owned by WAF-11/WAF-12.

Compatibility contracts

• Accept UTF-8 configuration bytes while treating SecLang structural punctuation and whitespace according to the pinned baselines.
• Normalize LF and CRLF line endings; preserve byte offsets and physical line/column locations.
• Join backslash continuations only where baseline syntax permits and retain the complete logical-to-physical source map.
• Recognize comments only outside quoted/escaped content.
• Support single-quoted, double-quoted, and unquoted directive arguments with compatible escaping and empty-value behavior.
• Parse directive names case-insensitively while preserving original spelling and source slices for formatting/diagnostics.
• Parse SecRule targets, target negation/count markers, selectors, operators, operator negation, quoted parameters, and the complete comma-delimited action field without losing escapes or macro text.
• Parse SecAction, SecDefaultAction, marker, update, removal, engine, body, audit, debug, collection, and other directives into a generic lossless directive AST; WAF-12 owns semantic support validation.
• Reject non-whitespace trailing bytes, unterminated quotes, dangling escapes/continuations, missing rule operands, empty directive names, malformed action separators, and invalid include forms.
• Expand local Include and IncludeOptional trees deterministically. Authenticated remote rules remain WAF-13.

Deliverables

• src/seclang/source.zig: owned source registry, stable source IDs, byte/line lookup, and include ancestry.
• src/seclang/lexer.zig: bounded logical-line and token lexer with quote/escape/comment state.
• src/seclang/parser.zig: lossless directive and rule AST plus strict full-input parser.
• src/seclang/include.zig: root-confined local resolver with deterministic glob ordering and cycle detection.
• Structured diagnostics with code, severity, primary span, optional secondary/include spans, and bounded human rendering.
• Public parse APIs for strings, files, and include trees with explicit allocator ownership.
• Focused baseline fixtures, malformed-input tables, include integration tests, fuzz entry points, and parser throughput/allocation benchmarks.
• Documentation and machine-readable compatibility evidence.

Interfaces

Parser.parseSource consumes one owned source and returns an owned Document containing directives and diagnostics. IncludeResolver.parseTree receives an explicit configuration root and entry path, registers each canonical source once, and returns documents in deterministic expansion order. AST nodes reference stable SourceId plus half-open byte spans; no node borrows caller buffers.

The parser produces syntax, not a partially validated WAF. Semantic compilation must consume a completely parsed document set and may publish only if every required directive and feature validates.

Resource and security limits

• Maximum bytes per source, logical line, token, quoted value, and diagnostic.
• Maximum tokens/directives/diagnostics per source.
• Maximum include depth, source count, aggregate bytes, and glob matches.
• Canonical root confinement; reject absolute paths when policy forbids them, .. escapes, symlink escapes, non-regular files, cycles, and duplicate canonical sources.
• Deterministic lexical glob ordering independent of filesystem enumeration.
• No recursive parser call proportional to attacker-controlled rule/action count.
• Failure occurs before unbounded ownership; resource-limit errors remain distinct from syntax and I/O errors.

Acceptance criteria

• Representative pinned ModSecurity and Coraza configuration fixtures produce the expected lossless AST and spans.
• Every CRS 4.28 configuration file is lexed and parsed; unsupported semantics may be reported only by later compilation, not misparsed here.
• Line continuation, quoting, escaping, comments, macros, regex delimiters, action commas, and target selectors have differential fixtures.
• Include ordering, optional-missing behavior, cycle detection, canonical deduplication, and root/symlink escape rejection are executable.
• Malformed and trailing input always produces a bounded diagnostic or typed limit error; no silent truncation or partial-success publication.
• Parser fuzzing covers arbitrary bytes, deep quoting/escaping, long logical lines, and include graphs.
• ReleaseFast benchmark reports bytes/second, directives/second, allocations, and peak owned bytes for small configs and CRS.
• Exact Pantry-pinned Zig formatting, tests, compile checks, and parser benchmark pass locally and in hosted CI.
• Implementation lands on main in small conventional commits with linked evidence.

Test matrix

• LF/CRLF/final-line-without-newline and continuation boundaries.
• Empty, single, double, and mixed quoting; escaped quotes, backslashes, commas, hashes, and whitespace.
• Comments at line start/after tokens versus hashes inside operators, regexes, macro text, and quotes.
• SecRule with one/many/negated/count/selector targets, negated operators, empty/nonempty parameters, and zero/many actions.
• Directives with zero/one/many arguments and original-case preservation.
• Syntax diagnostics at first/middle/final byte with exact line/column and include ancestry.
• Include file, sorted glob, optional missing, duplicate, cycle, depth/source/byte/glob limit, absolute path policy, traversal, and symlink escape.
• Allocation failure injection and repeated parse/deinit leak checks.

Relationships

• Parent: WAF-00

Blocked by

• WAF-03 — complete

Blocks

• WAF-11 — immutable semantic execution plans
• WAF-12 — stable directive union validation
• WAF-13 — updates and authenticated remote rules
• WAF-33 — validate/format/test/explain CLI
• WAF-34 — SecRules Test Set
• WAF-37 — CRS FTW compatibility
• WAF-38 — parser and include fuzzing

Baselines and upstream references

• ModSecurity 3.0.16
• Coraza 3.7.0
• OWASP CRS 4.28.0
• Zig 0.17.0-dev.1441+d5181a9c9, resolved and locked through Pantry

WAF-03 (#4) is complete, so this issue is unblocked. The implementation contract has been expanded from a placeholder into concrete source ownership, lossless syntax, include-security, diagnostics, limits, differential fixtures, fuzzing, and benchmark acceptance criteria. Work starts only against this tracked scope.

The first WAF-10 slice landed as feat(parser): add owned SecLang sources.

It adds stable 32-bit source IDs, copied UTF-8 path/config bytes, half-open spans, physical line starts, byte-accurate 1-based locations, include ancestry, and deterministic per-source/aggregate/path/line/source-count limits. Invalid UTF-8, ancestry, IDs, and spans remain distinct typed errors, and rejected input is validated before ownership. Focused ownership, CRLF location, ancestry, invalid UTF-8, limit, and span tests pass with the exact pinned compiler.

Logical-line normalization landed as feat(parser): normalize SecLang logical lines.

The iterator handles LF/CRLF physical lines, applies the pinned scanner continuation rule (strip horizontal whitespace before backslash and indentation after newline), retains logical-to-physical segments and whole physical spans, and reports bounded line/segment and dangling-continuation errors. Returned lines own both text and source maps with allocation-failure-safe cleanup. Focused mapping, CRLF, multi-continuation, limit, and EOF tests plus full compile checks passed.

Quoted argument tokenization landed as feat(parser): tokenize quoted SecLang arguments.

Tokens preserve raw logical slices, quote kind, logical offsets, and mapped physical spans. The state machine handles single/double/unquoted arguments, escaped bytes, empty quoted values, inline hashes inside active tokens, and comments beginning between tokens. Unterminated quotes, dangling escapes, token size, and token count are explicit bounded errors. Focused quote/escape/hash/comment and hostile-limit tests plus full compile checks passed.

Continuation boundary parity landed as fix(parser): preserve continuation token boundaries. At top level, whitespace consumed by the pinned continuation pattern now leaves one synthetic token separator, preventing ARGS from merging with the next quoted operand. Inside active quotes it remains stripped, matching scanner behavior. Source mapping clamps the synthetic byte to the preceding physical boundary. Focused multi-line coverage and full checks pass.

Lossless directive-shape parsing landed as feat(parser): parse SecLang directive shapes.

parseSource owns directive names and arguments, preserves quote kind and physical spans, classifies SecRule/SecAction/SecDefaultAction/SecMarker/Include/IncludeOptional case-insensitively, and exposes target/operator/action accessors. It ignores only fully tokenized comments/blank lines and strictly rejects quoted/invalid names, missing operands/action lists/paths, and extra rule/action/marker/include arguments. Directive and aggregate argument counts are bounded. Focused continuation/comment/shape/trailing/limit tests and compile checks passed.

Implemented and pushed 7011016 (feat(parser): structure SecRule syntax).

Evidence:

• Added bounded parsers for target lists, target modifiers (!/&), selectors, explicit/negated operators, implicit @rx, and action lists.
• Action splitting is quote- and escape-aware; target splitting preserves escaped pipes.
• Malformed/empty elements, dangling escapes, unterminated nested quotes, and configured count limits fail explicitly.
• Exact pinned Zig 0.17-dev zig build test, zig build check, formatting, and git diff --check pass.

Next: attach these structured nodes to the owned directive AST, then add source-anchored diagnostic rendering.

Implemented and pushed HEAD (feat(parser): attach structured rule nodes).

The owned directive AST now contains parsed target, operator, and action nodes for SecRule, SecAction, and SecDefaultAction. Parser limits include syntax element bounds, and malformed structured syntax is propagated rather than deferred to execution-plan compilation.

Focused evidence: exact pinned Zig 0.17-dev tests/checks, formatting, malformed target propagation, action-limit propagation, and git diff --check all pass.

Implemented and pushed two diagnostic increments:

• 1ad8bdc — stable SecLang diagnostic codes, severity, primary/secondary spans, include ancestry, exact line/column excerpts, and hard-bounded human rendering.
• HEAD — parser-owned diagnosed outcomes with source-anchored failures while preserving the typed parseSource API.

Successful documents now expose a diagnostics collection for later non-fatal messages. Failed parses can be consumed without string matching through parseSourceOutcome; allocation failure and invalid source identity remain typed infrastructure errors. Tests cover errors at the first, middle, and final source positions plus include ancestry and a strict rendered-byte cap.

Exact pinned Zig 0.17-dev tests passed twice, along with compile checks, formatting, and git diff --check.

Implemented and pushed HEAD (feat(parser): resolve confined include trees).

The new resolver:

• owns the source registry and parsed documents as a single tree;
• resolves every existing path canonically and confines it to an explicit configuration root;
• rejects forbidden absolute paths, lexical traversal, symlink escape, non-regular files, cycles, depth overflow, source/byte overflow, oversized/invalid patterns, and glob-match overflow distinctly;
• expands Include/IncludeOptional in source order, sorts glob results lexically, and deduplicates canonical sources;
• supports bounded *, ?, and bracket-class matching without allowing * to cross path separators;
• records include ancestry on every registered child source.

Safe-allocator integration tests cover sorted globs, canonical duplicate inclusion, optional missing files, cycles, depth/match limits, traversal, absolute-path policy, symlink escapes, wildcard classes, and separator boundaries. Exact pinned Zig 0.17-dev tests/checks, formatting, and diff checks pass.

Implemented and pushed HEAD (feat(parser): expose owned parse APIs).

Public frontend entry points now cover:

• owned in-memory bytes with typed errors;
• owned in-memory diagnosed outcomes;
• a single file plus its relative include tree;
• an explicit root plus entry include tree.

All successful and diagnosed outcomes own their source registry, spans, AST, and diagnostic context; tests mutate the caller buffer after parsing to prove no borrowed input survives. File tests prove nested includes resolve relative to the including file. Exact pinned Zig tests/checks and diff checks pass.

Implemented and pushed HEAD (test(crs): add pinned parser corpus gate).

Local pinned-baseline result against CRS v4.28.0 commit 55b09f5acfd16413e7b31041100711ceb7adc89c:

parser corpus roots=3 files=33 bytes=771763 directives=794

The three roots are the shipped crs-setup.conf.example, plugins/, and rules/ trees. The repository's util/crs-rules-check/examples/ files are deliberately malformed validator fixtures and are therefore not described as deployable CRS configuration; running the tool over the entire checkout correctly rejects those malformed examples with bounded diagnostics.

The corpus runner accepts repeatable file/directory roots, recursively selects only .conf/.conf.example, prints a source diagnostic on first failure, and records the exact CRS commit in dependencies.lock.json.

Implemented and pushed HEAD (perf(parser): benchmark throughput and ownership).

The ReleaseFast benchmark now reports bytes/second, directives/second, allocations/parse, and peak live owned bytes while asserting that every parse/deinit returns tracked live ownership to zero.

Local exact-compiler results:

parser input_bytes=309 iterations=50000 directives=200000 bytes_per_second=6477004 directives_per_second=83844 allocations_per_parse=26 peak_owned_bytes=7963
parser input_bytes=98977 iterations=1356 directives=100344 bytes_per_second=15966218 directives_per_second=11937 allocations_per_parse=3319 peak_owned_bytes=536193

The second workload is CRS 4.28's REQUEST-942-APPLICATION-ATTACK-SQLI.conf. These numbers establish a reproducible baseline rather than a release threshold; optimization work can now compare the same pinned workload.

Implemented and pushed HEAD (test(parser): add deterministic fuzz oracle).

The reusable fuzzOne oracle accepts arbitrary bytes, treats invalid UTF-8 as a typed rejected input, validates every successful AST span against the owned registry, validates failed-parse spans, renders diagnostics under a 1 KiB cap, and checks that owned source bytes equal the input.

The deterministic driver covers four streams: arbitrary bytes, ASCII structural noise, mutations of a valid chained rule, and deep quote/escape/newline patterns. A 10,000-case run with seed 6840335614489015467 is green and exactly reproducible.

Two final test increments are pushed:

• f81901b injects failure at every allocation point in an owned rule parse and proves allocated bytes equal freed bytes for every unwind path.
• HEAD pins ModSecurity 3.0.16 and Coraza 3.7.0 exact commits, verifies them in CI, and adds all of their .conf/.conf.example files to the corpus gate.

Combined exact local result across ModSecurity, Coraza, and deployable CRS configuration:

parser corpus roots=5 files=58 bytes=803184 directives=925

There are no unexplained exclusions.

Published WAF-10 contracts and evidence in HEAD (docs: publish SecLang parser evidence).

Documentation now covers ownership, public APIs, lexical and structured syntax behavior, diagnostic codes/rendering, include threat model, all resource limits, the deliberate CRS validator-fixture exclusion, exact baseline commits/counts, reproduction commands, and the boundary between syntax and WAF-11/WAF-12 semantics.

The same facts are checked as embedded machine-readable JSON at src/compatibility/evidence/seclang-parser.json; unit tests validate its issue identity, corpus file count, and zero unexplained exclusions.

Completion evidence

WAF-10 is complete on main at e72709cb9661d0d70c600bb5dd58263d5621d610.

Hosted boundary: https://github.com/zig-utils/zig-waf/actions/runs/29942330646 — green.

Delivered:

• owned UTF-8 source registry, physical locations, continuation segment maps, and include ancestry;
• bounded LF/CRLF logical-line lexer with quote/escape/comment compatibility;
• lossless directive AST plus structured SecRule targets, operators, and actions;
• stable source-anchored diagnostics with secondary spans, ancestry, and bounded rendering;
• typed owned byte/file/tree APIs;
• root-confined deterministic includes with optional semantics, canonical deduplication, cycle detection, lexical and symlink escape rejection, and explicit limits;
• injected failure at every allocation point with complete unwind accounting;
• reusable fuzz oracle and 10,000-case deterministic hosted fuzz gate;
• ReleaseFast throughput/allocation/peak-owned benchmark;
• embedded machine-readable evidence and operator/developer documentation.

Pinned hosted corpus:

parser corpus roots=5 files=58 bytes=803184 directives=925

This covers every discovered .conf/.conf.example file in ModSecurity 3.0.16 and Coraza 3.7.0 plus all deployable CRS 4.28 setup/plugin/rules configurations. The 13 excluded CRS util/crs-rules-check/examples inputs are deliberately malformed validator fixtures, are rejected by this parser, and are explicitly documented; unexplained exclusions are zero.

Semantic directive/operator/action availability remains correctly deferred to WAF-11/WAF-12. No partial configuration is published by this syntax layer.