ReviewOS

zig-utils/zig-js

Frontend: decode escaped identifiers in one span-based pass

#538
Closed chris opened this 9 days ago ·
9 days ago

Parent: #496 Related: #493, #460, #461, #536, #537

Problem

Lexer.lexIdentName scans every escaped identifier once to validate Unicode escapes and IdentifierStart/IdentifierPart, then decodeIdent rescans the entire raw identifier and appends every non-escaped byte individually into arena storage. Generated code using escaped bindings/properties, obfuscator output, conformance workloads, and adversarial wide parameter lists therefore pay two full scans plus avoidable per-byte growth/copy work.

Scope

  • Add parser-only 1K/2K/4K escaped-identifier growth rows with frozen source construction, decoded-name structural checks, warmup, and timed parse boundary.
  • Preserve the existing allocation-free source slice for unescaped ASCII/Unicode identifiers.
  • At the first escape, materialize decoded storage once, copy raw spans in batches, decode each Unicode escape during the validating scan, and remove the redundant second scan.
  • Preserve escaped-keyword distinction, IdentifierStart/IdentifierPart validation, Unicode scalar bounds, \\uXXXX/\\u{X} behavior, non-ASCII raw identifiers, diagnostics/source positions, private names, and source lifetime.

Security and correctness

  • Exact decoded bytes, not hashes, determine identifier identity.
  • Invalid escapes and invalid decoded start/continue characters must retain exact SyntaxError classification/location.
  • Decoding must remain OOM-propagating and failure-atomic; no global cache or attacker-predictable intern table may be introduced.
  • Do not special-case benchmark names, loosen Unicode validation, change escaped keyword semantics, or defer early errors.

Acceptance

  • Benchmark source and decoded-name structural checks land before lexer changes.
  • Pointer/allocation witnesses prove unescaped identifiers remain borrowed and escaped identifiers own exact decoded bytes.
  • 1K/2K/4K escaped rows show bounded near-linear work and materially lower instruction/allocation cost versus the exact parent; an ordinary control remains unchanged.
  • Lexer/parser/frontend suites, escaped keyword/private/property/binding cases, malformed Unicode/OOM diagnostics, forced execution paths, full units, and affected test262 identifier subtrees pass with no regressions.
9 days ago

Benchmark-first commit landed on main as 85dfd44a: bench(frontend): add escaped identifier growth rows (#538).

The parser-only runner now freezes strict 1,024/2,048/4,096 parameter lists spelling every name as paramet\\u0065rN. Source construction is outside warmup and timing. Each parse validates the function declaration, exact parameter count, decoded parameter prefix, and numeric suffix equal to the parameter index before summing all decoded name bytes.

Frozen one-job checksums:

  • escaped 1,024: 13,233
  • escaped 2,048: 27,569
  • escaped 4,096: 56,241

The existing unescaped strict-parameter rows remain the control. No lexer behavior or performance claim is included in this commit; the implementation will be compared against this exact parent.

9 days ago

Completed by 81bef094: perf(frontend): decode escaped identifiers in one pass (#538).

Ordinary identifiers stay on the original allocation-free source-slice loop. The first \\u escape now branches into a validating decode continuation that copies intervening raw source spans in batches and UTF-8-encodes each validated code point during that same scan. This removes the former full decodeIdent rescan and byte-by-byte ordinary append path without adding a cache, hash identity, or interning policy.

Coverage proves:

  • an ordinary mixed ASCII/Unicode identifier tokenizes with a zero-byte allocator and exact source-pointer identity;
  • an escaped identifier reports OutOfMemory with the same allocator, owns distinct decoded storage when allocation succeeds, preserves raw pos/end, and decodes both \\uXXXX and \\u{X} exactly;
  • escaped private names decode exactly;
  • forced escaped binding, property, and private-name execution returns 17 on parent and candidate;
  • escaped if remains an UnexpectedToken early error.

Seven order-balanced exact-parent diagnostic pairs, 100 parses per sample, identical frozen decoded-name checksums:

  • 1,024 escaped parameters: 842,565,360 -> 815,533,432 retired instructions (0.968x);
  • 2,048: 1,677,883,753 -> 1,621,251,723 (0.966x);
  • 4,096: 3,375,953,241 -> 3,261,946,145 (0.966x);
  • ordinary 4,096 control: 2,612,326,869 -> 2,584,618,682 (0.989x).

Candidate normalized escaped instructions per parameter vary by only 0.001x from 1,024 to 4,096, preserving bounded near-linear growth. RSS samples were noisy and are not used for a public claim.

Validation:

  • lexer units: 11/11
  • parser units: 30/30
  • focused frontend: 12/12
  • lexer TSan: 11/11
  • full sharded units: 1,597 passed, 1 expected skip, 0 failed/leaked
  • exact-parent identifier/reserved/future-reserved test262: 350 -> 350; flips 0

The benchmark-first requirement is 85dfd44a. All acceptance items are satisfied.

Sign in to comment on this issue.