ReviewOS

zig-utils/zig-js

Lexer: scan ASCII identifier continuations in blocks

#612
Closed chris opened this 9 days ago ·
9 days ago

Parent: #496 Related: #538, #547, #608, #609, #611

Problem

After #610/#611 removed source-location and impossible parameter-conflict work, Lexer.lexIdentNameWithOwnedPrefix is the largest leaf in the current ReleaseFast Unicode-identifier profile. Every raw continuation byte executes the escape check, ASCII/non-ASCII split, first-versus-part conditional, scalar identifier-part classification, and loop update. The frozen ASCII and Unicode 4K parameter names contain long ASCII suffixes after their already-validated start, so this byte-serial control work dominates even though no escape or delimiter occurs inside each suffix.

The first state is also redundant on the raw path after #608: a zero validated-start length denotes a leading \\u escape and can enter the owned decoder immediately; every non-zero raw start has already been consumed and all remaining scalars use IdentifierPart.

Proposed resolution

Split the leading-escape path from raw identifier continuation scanning. Consume complete 8-byte ASCII blocks only when every lane is an ECMAScript ASCII IdentifierPart; otherwise return to the exact scalar boundary so backslash escapes, non-ASCII UTF-8, invalid bytes, and delimiters follow their existing validation paths. Keep the scalar tail and Unicode 17 authority unchanged.

Acceptance

  • ASCII blocks stop at the first delimiter, backslash, non-ASCII lead byte, invalid ASCII punctuation, and EOF without overread or token-boundary drift.
  • Leading and embedded Unicode escapes retain decoded text, escape provenance, allocation behavior, and failure offsets.
  • Raw Unicode IdentifierStart/IdentifierContinue and malformed UTF-8 behavior remain governed by the existing Unicode 17 tables and decoder.
  • Numeric adjacency and private-name tokenization remain exact.
  • Exact-parent evidence covers ASCII, escaped, Unicode, raw private, and escaped-private 4K controls with unchanged checksums and bounded allocations.
  • Focused lexer/frontend Debug/TSan, full units, identifier/private/numeric test262, and forced tree-walker/required-bytecode witnesses remain exact.

Security boundary

Identifier length, byte alignment, escape placement, UTF-8 validity, and delimiter position are attacker-controlled. Block loads must prove the complete width is in bounds, may advance only when every lane is valid, and must fall back at the original byte when any lane fails.

No-workaround rules

No ASCII-only grammar, skipped Unicode validation, permissive malformed UTF-8, approximate delimiter search, source/benchmark recognition, token/AST widening, or job/Promise/microtask/Worker lifecycle changes.

9 days ago

Implemented and pushed to main in 9861bf53 (perf(lexer): scan identifier continuations in blocks (#612)).

The raw identifier path now:

  • routes the leading-escape sentinel directly to the owned decoder;
  • loads an 8-byte vector only after proving the complete width is in bounds;
  • advances only when every lane is [A-Za-z0-9_$];
  • leaves any mixed block untouched for the existing exact scalar escape, Unicode 17, malformed-byte, and delimiter paths;
  • keeps the allocation-free borrowed slice for raw ordinary and private names.

Added focused coverage around 7/8/9 and 15/16/17 continuation boundaries, all ASCII IdentifierPart classes, EOF/delimiters, Unicode handoff, leading and embedded escapes, raw and escaped private names, allocation ownership, and malformed UTF-8.

Exact-parent performance evidence compares parent bd24f7f2 with candidate 9861bf53: 15 order-balanced pairs per row, no discarded samples, ReleaseFast, one parser construction/parse/AST validation/arena teardown per boundary. All checksums matched.

4K controlwall ratioinstruction ratiocycle ratioallocationsbytes
ASCII parameters0.953x0.9143x0.9369x57 -> 572,534,568 -> 2,534,568
escaped identifiers0.983x1.0185x1.0056x8,249 -> 8,2494,197,304 -> 4,197,304
Unicode identifiers1.060x0.9419x0.9773x57 -> 572,534,568 -> 2,534,568
raw private names0.957x0.9626x0.9766x53 -> 532,584,328 -> 2,584,328
escaped private names0.953x1.0395x0.9553x8,245 -> 8,2454,210,200 -> 4,210,200

This is a diagnostic host, so wall/cycle noise is reported rather than promoted. The escaped controls intentionally contain raw prefixes too short to form a complete block; they retain exact allocation/checksum behavior and improve median wall time, while their instruction counts expose the failed-block probe cost rather than hiding it.

Correctness evidence:

  • lexer-related units: 22/22 passed;
  • focused block-boundary TSan: passed;
  • frontend: 12/12 passed;
  • full units: 1,729 passed, 1 expected skip, 0 failed, 0 leaked;
  • test262: identifiers 268 -> 268, class statements 4,367 -> 4,367, class expressions 4,059 -> 4,059, numeric literals 157 -> 157; 0 failures or flips across 8,851 cases;
  • four forced tree-walker/required-bytecode witnesses: passed in both tiers.

No jobs, stacks, queues, Promise, microtask, Worker, or lifecycle surfaces changed.

Sign in to comment on this issue.