ReviewOS

zig-utils/zig-js

Frontend: tokenize modules once under the correct lexical goal

#553
Closed chris opened this 9 days ago ·
9 days ago

Parent: #496 Related: #461, #479

Problem

Parser.initWithScratchDiagnostic always tokenized source with Script HTML-comment handling enabled. Parser.parseModule then discarded that complete token stream and tokenized the same source again with HTML-like comments disabled. Every production module parse therefore paid for and retained an unnecessary Script-goal token list before performing the required Module-goal scan.

Scope

  • Freeze parser-only 1K/2K/4K static-module declaration/import/export rows with exact AST, source-range, cardinality, and checksum validation.
  • Freeze Script-vs-Module HTML-like-comment rejection controls and malformed-module diagnostic offsets.
  • Let production module entry points retain exactly one token stream while preserving Module lexical-goal rejection.
  • Preserve the existing Parser.init(...).parseModule() public path.
  • Keep source construction outside warmup/scored boundaries and lexing, parsing, AST construction, early errors, and validation inside.

No-workaround rules

  • No source pre-scan, cached/precompiled benchmark input, hidden warmup tokenization, omitted tokens/source locations, weakened Annex B behavior, delayed errors, or duplicate Script/Module parsers.
  • Do not accept HTML-like comments in modules or reject their Annex B Script forms.
  • Do not touch stacks, queues, jobs, Promise/microtask, or Worker lifecycle code.

Acceptance

  • Benchmark source, 1K/2K/4K identities, jobs, checksums, warmup, timed boundary, and existing Script/Module controls were committed before the runtime change.
  • Production module parsing creates one lexer and one retained token stream while preserving Module lexical-goal rules.
  • Exact AST/module-request/export/source-range checks and diagnostic offsets remain unchanged.
  • Failure paths are allocation-failure atomic and bounded; no discarded Script token arena remains.
  • Focused frontend Debug/TSan, full units, module/Annex B/source-location test262, and forced module execution pass without regressions.
  • Seven-pair exact-parent evidence reports wall, CPU, allocation requests/bytes, peak RSS, instructions/cycles/energy availability, thermal state, and dispersion without discarded samples.

Security boundary

Module source and comment forms are attacker-controlled. The fix removes redundant work without allowing lexical-goal confusion, hiding malformed input, changing diagnostic boundaries, or retaining attacker-proportional discarded storage.

Landed

9 days ago

Benchmark-first commit landed on main as 1240045a: bench(frontend): add module lexical-goal rows (#553).

The frozen parser-only rows use one wide import list, one matching local export list, and one source-location marker at 1K/2K/4K entries. Source construction remains outside warmup/scored boundaries; the current production Parser.init -> parseModule path, both tokenization passes, parsing, AST construction, early errors, and independent validation remain inside.

Every sample validates exact module specifier, imported/local/exported names and order, AST kinds, declaration/initializer, entry cardinality, marker source location, and checksum. At 200 jobs the frozen exact checksums are 132427000, 477207800, and 1795915000; all rows pass. No runtime fix is included in this commit.

The next commit will change only module-goal initialization and preserve the Script compatibility path and HTML-comment rejection controls.

9 days ago

Completed on main in three linked commits:

  • 1240045a freezes the 1K/2K/4K parser-only module rows and exact checksums.
  • 78dbd220 records the first Annex B HTML-like comment in the original Script tokenization and makes Module parsing reject that exact offset while retaining the same token stream.
  • 080f2302 publishes all raw samples and the summary report.

Seven order-balanced pairs at each size show stable retired-instruction reductions of 33.6-34.4%, allocation-request reductions of 20.6-20.7%, allocated-byte reductions of 31.5-47.9%, and peak-RSS reductions of 9.1-30.3%. The stable 1K wall row improves 35.2%; noisy 2K/4K wall medians remain explicitly diagnostic.

Validation is green: 1,623 full unit passes plus one expected skip, focused frontend Debug/TSan 12/12, and exact module-code/import/import.meta/Annex B test262 756 -> 756 with zero failures or flips. Script HTML comments remain accepted; Modules reject the first exact open/close comment offset without a second scan.

Sign in to comment on this issue.