Frontend: optimize lexing, parsing, bytecode, and module compilation

Parent: #459 Depends on: #460, #461, #465, #479, #481

Goal

Optimize lexing, parsing, AST/bytecode generation, source evaluation, and module compilation for cold and dynamic code without hiding compilation outside timed boundaries.

Scope

• Add source-size/syntax-mix benchmarks for scripts, modules, functions/constructors, direct eval, dynamic import, JSON-like literals, classes/private fields, async/generators, Unicode identifiers/strings/templates, regex literals, and invalid syntax diagnostics.
• Attribute lex/token storage, AST allocation/layout, scope analysis, bytecode emission, constant/name tables, source maps/locations, debugger registration, optimizer metadata, and allocator/GC pressure.
• Reduce passes/allocations and improve locality; consider direct bytecode emission or compact transient IR only with exact diagnostics, early errors, source ranges, and tree-walker availability.
• Reuse immutable parse/compile artifacts only with exact source/options/realm/dynamic-scope keys and bounded lifecycle.
• Optimize streaming/module compilation incrementally while preserving failure atomicity and exact final bytes.

No-workaround rules

• No precompiled benchmark source, omitted source locations/debug metadata, delayed syntax error, reduced Unicode/Annex B/module behavior, hash-only cache collision risk, or third-party parser/compiler.

Acceptance

• #460 publishes parse/compile/evaluate cold and warm latency, CPU, allocations, peak RSS, bytecode/code bytes, and diagnostics for valid/invalid sources.
• Parser/compiler/tree-walker/VM forced differentials and full test262/module/Annex B/source-location/debugger gates pass.
• Streaming and OOM/fault paths are failure-atomic and bounded.
• Exact-parent evidence proves phase/allocation improvements without hiding work in cache warmup.
• Cold lifecycle #479 and tiering #481 consume the results without regressing #44.

Foundation

Current pipeline: src/lexer.zig → src/parser.zig/ast.zig → src/compiler.zig/bytecode.zig; module streaming work under #406–#410.

The frontend optimization work now has a fast semantic/rejection loop from tools(test): add focused frontend suite (#494).

The production-module artifact currently covers precedence, ASI, nested destructuring, templates, private fields, optional chaining/nullish coalescing, regexp literals, and five invalid-source diagnostics. All 12 probes pass in Debug and TSan. This is test-loop infrastructure for the optimization work; it does not satisfy the benchmark or exact-parent acceptance criteria, so #496 remains open.

Closed the strict-parameter frontend complexity slice as #520. Parser duplicate validation now uses an invocation-local exact hash index, and Context/eval/module parsing supplies freeable scratch backing so repeated parses do not retain bucket capacity. At 4,096 unique strict parameters, exact-parent full eval parse-and-compile improves from 12.288 ms to 1.554 ms and from 286.8M to 21.0M instructions. Implementation: https://github.com/zig-utils/zig-js/commit/a5bb209a and https://github.com/zig-utils/zig-js/commit/e11bec9c. Evidence: https://github.com/zig-utils/zig-js/commit/39c617e4.

#536 is complete on main: 4b5a705b added benchmark-first 1K/2K/4K ordinary-string rows plus an escaped control, and 23df967d made unescaped lexer tokens source-backed while lazily decoding escaped spans.

The fail-on-first-allocation witness proves the zero-copy path. Across exact-parent diagnostic 1K/2K/4K rows, retired instructions are 0.730x/0.718x/0.720x and median peak RSS is 0.804x/0.676x/0.665x; escaped decoding remains exact. Full units are 1,594 pass / 1 expected skip / 0 failures or leaks, focused TSan passes, and the affected test262 comparison remains 758/758. #496 remains open for the other frontend surfaces and its umbrella acceptance gates.

#537 is complete on main: 09b525bc added benchmark-first 1K/2K/4K ordinary-template rows plus escaped/tagged controls, and 718cb122 made plain cooked quasis source-backed while lazily batch-decoding escaped spans.

Ordinary exact-parent diagnostic instruction ratios are 0.816x/0.803x/0.800x with median peak-RSS ratios 0.952x/0.905x/0.871x. Escaped/tagged controls retain exact checksums. Source/normalized/decoded pointer-lifetime and OOM witnesses pass; full units are 1,596 pass / 1 expected skip / 0 failures or leaks; template test262 remains 84/84. #496 remains open for its broader frontend gates.

#538 is complete on main: 85dfd44a added benchmark-first 1K/2K/4K escaped-identifier rows with exact decoded-name validation, and 81bef094 removed the redundant raw-name rescan and per-byte append path.

Escaped exact-parent diagnostic instruction ratios are 0.968x/0.966x/0.966x; the ordinary 4K control is 0.989x. Pointer/allocation, private-name, OOM, forced execution, TSan, full-unit, and 350/350 identifier/reserved-word test262 coverage pass. #496 remains open for its broader frontend gates.

Frontend child #540 is complete in f888b717: private-name duplicate validation now uses a deterministic exact-byte O(N log N) grouping pass over scratch-backed indexes rather than a quadratic nested scan.

Seven-pair exact-parent medians at 1K/2K/4K private fields fell from 0.874/3.000/13.033 ms to 0.108/0.219/0.450 ms; retired-instruction ratios were 0.163x/0.089x/0.036x. Peak RSS stayed within 0.6%. The affected 2,962 test262 class-element cases and full 1,600-test unit inventory had 0 regressions.

This unblocks the source-slice allocation work in #539, which is now continuing against f888b717 as its exact parent.

Frontend child #539 is complete in 8cd6034f: unescaped private-name tokens now borrow their complete source slice; escaped names retain owned decoding and OOM semantics.

Stable exact-parent retired-instruction ratios at 1K/2K/4K plain fields were 0.907x/0.883x/0.870x, with peak-footprint ratios 0.868x/0.811x/0.794x. The escaped 4K control remained 1.0003x instructions with identical median memory. Pointer, zero-memory, decoded-byte, offset, test262, TSan, focused frontend, private runtime, and full unit gates all passed with 0 regressions.

Together with #540, this leaves private-heavy class parsing bounded and removes proportional arena copies for ordinary private names.

Child #544 is complete and closed.

4fdff17b replaces incremental numeric-separator normalization with one exact allocation while preserving a borrowed zero-allocation path for plain literals. c7e4f7f7 publishes the exact-parent report and raw evidence.

Across separated 1K/2K/4K parser workloads, the candidate removes exactly one logical allocation/resize request per literal, reduces cumulative parser allocation bytes by 11.6–14.2%, and reduces stable retired instructions by 5.4–5.6%. The stable 4K wall row is 0.887x parent. The unseparated 4K control preserves allocation requests and bytes exactly, with 1.005x wall and 0.9944x peak RSS. Full units, focused TSan, frontend/lexer tests, and exact-parent numeric/BigInt test262 coverage pass with 0 conformance flips.

Child #545 is complete and closed.

f91369c6 makes oversized unseparated decimal BigInts borrow their validated immutable source digits and separated forms reuse the exact normalization allocation. e1debe62 publishes the exact-parent report and raw evidence.

Across unseparated 1K/2K/4K workloads, one logical request is removed per parse and allocation bytes fall 45.2% / 58.1% / 71.5%. The separated same-value 4K control falls 39.0% while retaining its one normalization allocation. Wall medians stay within 2.1% and peak RSS does not regress. Full units, focused TSan/frontend/lexer, and exact-parent BigInt/numeric test262 coverage pass with 0 flips.

Completed child #546 with benchmark-first, implementation, and checked raw evidence on main:

• faa21251 benchmark rows
• 727d87a5 bounded radix conversion
• 552a77a5 exact-parent evidence

Oversized binary/octal/hexadecimal BigInt conversion now uses pre-sized base-1e9 scratch limbs and maximal radix chunks instead of byte-per-decimal-digit list growth. The 1K/2K/4K diagnostic rows retain full independent decimal-oracle validation and expose the remaining approximately quadratic growth. Candidate wall ratios are 0.018x/0.017x/0.016x; instruction ratios agree at 0.0229x/0.0184x/0.0164x. Full units, focused TSan, and exact-parent BigInt/numeric test262 comparisons pass without regressions.

Evidence: docs/.data/exact-parent-frontend-bigint-hex-2026-08-12.md

Completed correctness/security child #547 with benchmark-first, implementation, and checked raw evidence on main:

• b7bbea78 benchmark rows
• 085cb622 exact Unicode identifier classification
• 61396be6 exact-parent evidence

The lexer now retains its local ASCII fast path while raw and escaped non-ASCII identifiers consume zig-regex's generated Unicode 17 ID_Start/ID_Continue authority. Symbols, punctuation, unassigned code points, surrogate escapes, malformed UTF-8, and continue-only starts are rejected instead of changing parse boundaries. Exhaustive scalar agreement, full units, focused TSan, and exact-parent identifier/private-name test262 gates pass.

The diagnostic report records the correctness cost explicitly: raw-Unicode retired instructions rise 7.7–7.8% and stable 4K wall rises 9.3%, with unchanged allocations and near-linear growth. ASCII and escaped-ASCII controls remain wall-neutral. Evidence: docs/.data/exact-parent-frontend-unicode-identifiers-2026-08-12.md

Completed frontend child #548 on main with benchmark rows, one-exact-allocation escape decoding, and checked raw evidence.

Across 1K/2K/4K escaped-string rows, one allocation/resize request is removed per literal per parse; total parser requests fall 33.0-33.2%, allocated bytes 8.0-9.8%, and stable retired instructions 2.6-3.1%. The ordinary 4K control preserves allocation requests and bytes exactly. Wall and energy remain diagnostic due to dispersion. Lexer/OOM/diagnostic tests, focused Debug/TSan, full units, and exact string/directive/template test262 comparisons pass with 0 regressions.

Evidence: https://github.com/zig-utils/zig-js/blob/main/docs/.data/exact-parent-frontend-escaped-strings-2026-08-12.md

#496 remains open for its broader frontend matrix and acceptance gates.

Escaped template-quasi allocation work is complete in child #549:

• 67149a3f benchmark
• b3c8c3ba implementation
• fe6c1a4e exact-parent evidence

Valid escaped cooked quasis now use one exact allocation, plain quasis remain source-backed, and invalid tagged cooked quasis allocate no cooked buffer. Full units and exact template/tagged-template test262 comparisons are unchanged. See #549 for the complete gates and measurements.

Template raw-value normalization child #550 is complete:

• 48556702 benchmark
• ab8c6d9e implementation
• ec4c1d2d checked exact-parent evidence

Mixed CRLF/lone-CR template text now uses one exact normalized allocation; CR-free text remains source-backed. Full units and exact template/tagged-template test262 comparisons are clean. Stable instructions fall 7.9-8.3% across 1K/2K/4K, and retained parser allocation bytes fall 9.6-12.6%. See #550 for the complete gates and dispersion-qualified measurements.

Escaped private-name ownership child #551 is complete:

• d05d9942 benchmark
• 36414902 implementation
• 27d44d42 evidence

Each escaped private name now owns one complete decoded #-prefixed token instead of retaining an intermediate decoded identifier plus a formatted copy. Allocation requests fall 59.5-59.8%, bytes 13.0-15.5%, and stable instructions 10.0-10.1%. Full units, TSan, forced execution, and 2,962 exact class-element test262 cases pass unchanged. See #551 for full measurements and gates.

Exact tagged-template substitution storage child #552 is complete:

• efa336aa benchmark
• c3449e6b implementation
• be55f127 evidence

The lexer records top-level substitution count during its existing scan, and the parser now fills exact N+1/N+1/N cooked/raw/expression arrays instead of retaining three growth lists. Allocated bytes fall 12.2-18.3% and stable instructions 1.3-1.4%. Full units, TSan, exact template test262, and forced tree-walker/required-bytecode execution pass. See #552 for full evidence.

Frontend child #553 is complete on main: 1240045a froze the module rows, 78dbd220 removed the discarded second module token stream while preserving exact Annex B lexical-goal rejection, and 080f2302 published raw exact-parent evidence. Stable instructions fall 33.6-34.4%, allocation requests 20.6-20.7%, and bytes 31.5-47.9%; module/Annex B test262 remains 756/756 with zero flips. #496 stays open for its broader frontend matrix.

#554 is complete and closed. Benchmark-first, runtime, and evidence commits are 3bba3de9, 87eba6c2, and 2de66881. Repeated statement-location construction is now bounded by one exact line-start index plus binary searches; 1K/2K/4K retired instructions scale approximately linearly and fall to 0.0263x/0.0130x/0.0064x of the parent. Focused test262 remains 1130 -> 1130 with zero flips.

Completed frontend child #555 in 00a401d7, with exact-parent evidence in 59638193. Parse-time ordinary-function scope tracking removes overlapping arguments/eval source rescans: depth-1,024 retired instructions fall 94.85%, candidate depth growth is linear, and all six rows retain identical allocation counts/bytes. #555 is closed with 2,470 → 2,470 affected test262 passes.

Child #560 is complete in 8dc1c182. Parse-only RegExp compilation now reuses a resettable per-parse arena and releases state after each literal without delaying syntax/ReDoS validation. Exact-parent 4K allocation requests fall 536,269 → 4,148 and bytes 96,011,248 → 2,549,128; seven-pair medians improve across the 1K/2K/4K curve, and test262 has 0 flips/regressions.

Related runtime/frontend boundary child #607 is now open. Parser statement metadata covers only parseStatement roots, while the tree walker currently performs location-registry probes for every expression evaluation. The child will encode that exact publication boundary without widening AST nodes or weakening debugger/late-attach semantics.

Related parser/runtime boundary child #607 is complete in 506d8950. The parser now exposes the exact statement-root publication predicate, eliminating impossible evaluator registry probes without widening AST nodes. Full units are 1,724 pass / 1 skip; exact debugger test262 remains 2/2.

Opened #608 from the current-main parser-only Unicode identifier profile. IdentifierStart currently performs the same first-code-point classification/decode twice before IdentifierPart scanning. Existing 1K/2K/4K ASCII, Unicode, escaped, private, and escaped-private rows provide exact-parent coverage; the child preserves #538 ownership and error semantics.

Child #608 is complete via 4fe004e1. IdentifierStart is now decoded/classified once before IdentifierPart scanning. Formal Unicode 1K/2K/4K instruction ratios are 0.9654x/0.9658x/0.9657x with identical allocation counts/bytes; ASCII/escaped/private controls remain instruction-neutral to slightly better. All 3,230 affected test262 cases remain exact.

Opened #609 from the post-#608 parser-only profile. Reserved-binding checks still linearly walk 38 unconditional plus nine strict spellings for every ordinary name; the current 4K Unicode row samples 654 byte-equality leaves and 290 isForbiddenBindingName leaves. The child uses one immutable compile-time exact classification table and keeps every contextual/escaped grammar rule parser-owned.

Child #609 is complete via 0d5eb7a0. One immutable length-partitioned table replaces repeated reserved-word list walks. Formal instruction ratios are 0.8241x/0.8249x/0.8258x for Unicode 1K/2K/4K, 0.7689x for ASCII 4K, and 0.8210x for escaped 4K; private controls remain neutral and all allocations/checksums are exact. Affected test262 stays 2,325/2,325.

Child #610 is complete on main in 288136ba.

Statement coordinates now advance once at parse entry while retaining the existing completion-order debugger registry. Classic-for speculation transactionally rolls back its coordinate cursor and discarded ghost locations. Exact-parent instructions improve to 0.7327x on the causal Unicode row and 0.8835x–0.9107x across the repeated/mixed/nested location rows; the one-statement control is 0.9949x with allocation count and bytes exact. Test262 stays 1,028 -> 1,028; full units pass 1,726 plus one expected skip.

Child #611 is complete on main: benchmark contract 62a340ef, runtime fix bd24f7f2.

Functions with no direct LexicallyDeclaredNames now skip an impossible attacker-sized parameter/body conflict index. Exact-parent instructions improve to 0.8035x ASCII, 0.8462x Unicode, 0.8568x escaped, and 0.8031x nested-shadow; 11 allocation requests and 278,656 bytes disappear in the ordinary 4K rows. The direct lexical slow-path control stays exact in allocations/bytes and 1.0013x instructions. Test262 remains 2,526 -> 2,526; full units pass 1,728 plus one expected skip.

#612 is complete in 9861bf53. The lexer now consumes fully validated 8-byte ASCII IdentifierPart blocks while preserving the original byte for every escape, Unicode, malformed-input, and delimiter fallback.

Exact-parent 4K instruction ratios are 0.9143x for ASCII parameters, 0.9419x for Unicode identifiers, and 0.9626x for raw private names, with allocation counts/bytes unchanged. Escaped controls preserve exact checksums/allocations and document their short-prefix probe cost. Full units pass 1,729 with one expected skip; 8,851 identifier/class/numeric test262 cases remain exact with zero failures or flips.

#613 is complete in 09bcd444. Strict-parameter uniqueness now reserves its exact simple-name capacity once, while empty/singleton lists allocate nothing and duplicate/strict-binding checks remain exact.

Across exact-parent 1K/2K/4K growth, instruction ratios are 0.8167/0.8259/0.8315 for ASCII, 0.8957/0.8952/0.8965 escaped, and 0.8687/0.8791/0.8758 Unicode. Allocation requests fall by 8/9/10 and bytes by 34,872/69,712/139,368. Both direct and nested body controls improve without checksum changes. Full units pass 1,730 plus one expected skip, and 2,526 function-family test262 cases remain exact.

Opened #615 for the bounded TDZ-admission complexity slice: exact parse-plus-compile growth witnesses first, then a single exhaustive pending-name traversal with unchanged conservative TDZ semantics. This is independent of the user-owned queues/jobs work.

Frontend/compiler child #618 is implemented benchmark-first: 3fcd7f02 adds ten production parse+compile contracts and 14ca5cd4 reuses one repeated-body capture analysis across admission and lowering. All structural checksums and focused semantic/GC gates remain exact; full Test262 and guarded exact-parent evidence are tracked on #618.

Compiler child #619 is implemented benchmark-first in be27ecd4 and 8bbbf402. Deferred class bodies now scan one exact set of ordinary/non-Environment frame slots instead of rescanning the full member surface per slot, while eager computed keys and Environment-backed captures retain their existing admission semantics. Ten structural/allocation controls, focused compiler/frontend gates, and the full 1,733-pass / 1-skip unit inventory are green. Exact isolated performance and Test262 accounting remain tracked on #619 before closure.

Opened #621 for the remaining duplicate plain-function binding-inventory pass found after #615: shadowing and TDZ classification currently rebuild the same parameter/declaration map and traverse the same statement surface independently. The child is benchmark-first and preserves exact TDZ/opcode semantics.

Compiler child #621 landed benchmark-first in 3bc06b01 and 84efdae5. Plain-function admission now builds one binding inventory for shadow/TDZ decisions and proves no-lexical early exits exactly. Full units, compiler TSan, and 1,869 exact Test262 cases are green; guarded performance evidence remains on #621.

Opened frontend child #668: eval, realm evaluation, ShadowRealm.evaluate, and dynamic Function-family constructors currently pass physical StringCell bytes to the parser. The fix establishes one canonical WTF-8 ingress contract without changing lexer semantics or dynamic-code realm/error ordering.

Frontend child #668 is complete via 355329dc. Eval, realm evalScript, ShadowRealm.evaluate, and Function-family parsing now share a canonical WTF-8 ingress contract without changing lexer/parser semantics or realm/error ordering. Both storage layouts passed focused interpreter/VM coverage; Test262 620/620.

#718 landed in 4740df55, adding activation-local, precisely rooted class-heritage lowering across suspension and moving GC. This closes that slice only; the parent remains open for its broader GC/JIT audit.

Frontend/compiler child #621 is complete in 1a0cf6e2. Exact 1K/2K/4K evidence validates one binding inventory per plain-function compile: linear growth is preserved, retired instructions fall to about 0.94x across all widths, and allocation/byte counts strictly decrease. #496 remains open for the broader frontend optimization roadmap.

Linked security/frontend child #761 owns collision-resistant parser declaration-name membership. The design keeps immutable grammar lookup allocation-free, shares one lazy secure context across a parse root and nested template subparsers, and includes focused source-to-result/growth controls so hardening does not add a per-table entropy syscall.

Annex B frontend/runtime classification child #760 landed in 52a7c387. The invocation-local legacy-name index is securely seeded only on its first eligible candidate; candidate-free/strict/skipped scans remain table- and entropy-free. Exact-parent test/annexB/language stayed 845/845 passing.

Parser declaration/name membership hardening is complete in #761 via fix(parser): seed declaration name membership (#761): every parser source-string table now shares one lazy realm-derived secure context, with failure-atomic entropy/OOM behavior and no syntax regressions.

Compiler binding/capture membership hardening child #770 is complete in a3c39817, with six exact-parent 1K/2K/4K control/adversarial artifacts in 3d249965. Compiler/frontend/full/TSan gates and 9,554 relevant Test262 cases pass with zero flips.

Completed parser child #855 in 8afcf79: cover-grammar node identity remains O(1)-expected and exact, but no longer relies on allocator/ASLR placement as its collision defense. The implementation reuses one lazy parse-root seed across nested template parsers, adds no AST metadata or parse pass, and produced 0 exact-parent Test262 expression flips. Full evidence is on #855.

Completed frontend child #856 in f16ce8a: Annex B and repeated-catch AST identity plans retain O(1)-expected exact lookup without relying on arena/ASLR placement. The compiler reuses one existing lazy root seed, adds no AST fields/pass/runtime state, and produces byte-identical Chunk/Test262 results across seeds and exact parent.

New frontend allocation work is tracked in #913: 261981c reuses scratch token storage across template substitutions without skipping lex/parse work or moving AST payload lifetimes. Correctness gates passed (63 parser units; 1,131 exact-parent test262 cases). Performance publication remains pending #914, which tracks first-class direct-binary provenance so the current collector does not require dummy benchmark-overlay commits. Neither child nor this umbrella is being declared complete prematurely.

The publication previously pending under #913/#914 is now complete in be59a314. Both children are closed; #496 remains open for its broader acceptance gates.

The seven-pair growth report and embedded raw inputs compare exact parent bff189a with candidate 261981c on the Apple M2 Pro diagnostic host, Zig 0.17.0-dev.1441+d5181a9, real GC 0285ef8 and regex c60344d. At 1K/2K/4K substitutions, logical allocation requests per parse fall 2061→1038, 4109→2062, and 8205→4110; allocated bytes fall 354384→125232, 706640→248112, and 1411152→493872. Both parent and candidate retain linear instruction growth across the widths.

All six rows (three widths plus plain, escaped and tagged controls) preserve raw samples and exact checksums. The escaped control retains identical allocations/bytes and its noisy 1.104x wall ratio (22.09%/26.57% RSD) remains visible, not promoted into either a win or a stable regression.

This closes the scoped allocation/provenance evidence work, not the quiet-reference throughput/energy gate or the frontend umbrella. README's universal scorecard was not changed. The separate parser correctness series #920–#924 is landed; #925's module-boundary verification is running and is not claimed as performance evidence.