ReviewOS

zig-utils/zig-js

Frontend: avoid copying unescaped string literals

#536
Closed chris opened this 9 days ago ·
9 days ago

Parent: #496 Related: #493, #460, #461

Problem

Lexer.lexString eagerly appends every byte of every quoted JavaScript string literal into arena-owned ArrayList storage, even when the literal has no escape and its exact value is already a stable slice of the Context-owned source. Wide JSON-like object literals, generated tables, bundles, and modules therefore pay avoidable allocation, copying, growth, and retained parse-arena bytes proportional to their unescaped string payload.

Scope

  • Add parser-only growth rows for ordinary and escaped quoted-string programs with frozen source construction, checksums, warmup, and timed parse boundary.
  • Return an exact zero-copy source slice for unescaped literals.
  • Lazily materialize decoded storage at the first escape and copy raw spans in batches rather than appending every byte.
  • Preserve quote removal, escape decoding, line continuations, JSON-superset U+2028/U+2029, legacy-octal flags, WTF-8 behavior, diagnostics, source locations, and source lifetime.

Security and correctness

  • No borrowed slice may outlive the owned source backing used by the AST/compiled artifact.
  • Malformed and unterminated literals must retain exact failure classification/location.
  • Escaped decoding must remain OOM-propagating and failure-atomic.
  • Do not special-case benchmark text, intern attacker-controlled literals globally, omit source ownership, or weaken validation.

Acceptance

  • Benchmark source and structural checks land before the runtime change.
  • Fail-on-first-allocation proves an unescaped literal tokenizes without allocation; escaped literals still allocate and decode exactly.
  • 1K/2K/4K ordinary rows show bounded near-linear work and materially lower allocation/instruction cost versus the exact parent; escaped control remains semantically exact.
  • Lexer/parser/frontend suites, forced execution paths, OOM diagnostics, full units, and affected test262 string/source parsing pass with no regressions.
9 days ago

Benchmark-first commit landed on main as 4b5a705b: bench(frontend): add quoted-string growth rows (#536).

The parser-only runner now freezes 1,024/2,048/4,096 ordinary quoted-string programs plus a 4,096-literal escaped control. Source construction is outside warmup and timing. Each parse validates the top-level declaration, array-literal width, every StringLiteral node, and the sum of decoded bytes.

Frozen checksums:

  • ordinary 1,024: 50,090
  • ordinary 2,048: 101,290
  • ordinary 4,096: 203,690
  • escaped 4,096 control: 203,690

No runtime change or performance claim is included in this commit. The next commit will implement the lexer ownership change and compare it against this exact parent.

9 days ago

Completed by 23df967d: perf(frontend): borrow unescaped string literals (#536).

The lexer now returns unescaped StringLiteral token text as an exact source-backed slice. The first escape switches to arena-owned decoded storage, copies ordinary text as contiguous spans, and preserves the existing validation and legacy-octal paths. The dedicated zero-byte FixedBufferAllocator test proves an ordinary literal performs no allocation; the same allocator returns OutOfMemory for an escaped literal, and the allocation-backed control decodes raw\nspan\u0021tail exactly.

Exact-parent diagnostic growth evidence used 7 order-balanced pairs, 100 parses per sample, identical frozen checksums, and source construction outside warmup/timing:

  • 1,024 ordinary literals: 465,656,309 -> 339,921,574 retired instructions (0.730x); 4,685,824 -> 3,768,320 bytes median peak RSS (0.804x).
  • 2,048 ordinary literals: 903,796,640 -> 648,690,222 instructions (0.718x); 6,668,288 -> 4,505,600 bytes RSS (0.676x).
  • 4,096 ordinary literals: 1,799,642,376 -> 1,295,094,077 instructions (0.720x); 9,289,728 -> 6,176,768 bytes RSS (0.665x).
  • 4,096 escaped control: 1,981,117,216 -> 1,713,182,483 instructions (0.865x); median RSS remains 9,371,648 bytes; decoded checksum remains 20,369,000.

These are non-public diagnostic measurements. The standardized attribution collector intentionally rejected this parser-only runner because it does not emit native counter rows; no metrics were fabricated or substituted.

Validation:

  • lexer suite: 10/10
  • parser suite: 28/28
  • focused frontend suite: 12/12
  • focused TSan allocation/lifetime witness: 1/1
  • full units: 1,594 passed, 1 expected skip, 0 failed or leaked
  • exact-parent test262 string literals, string types, directive prologues, and module code: 758 -> 758; flips 0

The benchmark-first requirement is satisfied by 4b5a705b. Source ownership is unchanged: tokens and AST nodes already borrow the Context-owned input backing, and decoded text remains arena-owned. All acceptance items are satisfied.

Sign in to comment on this issue.