ReviewOS

zig-utils/zig-js

Frontend: decode escaped string literals with one exact allocation

#548
Closed chris opened this 8 days ago ·
8 days ago

Parent: #496 Related: #536

Problem

Lexer.lexString borrows unescaped literals, but the first escape switches to an arena-backed ArrayListUnmanaged. Repeated append growth plus toOwnedSlice makes decoded allocation count and retained capacity depend on escape/source layout. Large escape-heavy sources therefore pay avoidable allocator traffic inside the cold parse boundary.

Scope

  • Add 1K/2K/4K escaped-string benchmark rows with exact decoded-value checks and keep the ordinary-string control.
  • Decode valid string escapes through a bounded first pass that computes the exact byte length and a second pass that fills one allocation.
  • Preserve ECMAScript escape consumption, line continuations, Annex B octal flags, malformed escape diagnostics, lone-surrogate WTF-8, source positions, and unescaped source borrowing.
  • Keep lexer state failure-atomic when the exact allocation fails.

No-workaround rules

  • No worst-case over-allocation, hidden scratch copy, delayed validation, dropped source locations, weakened malformed-escape rejection, or special-case benchmark path.

Acceptance

  • Escaped literals make exactly one decoded-byte allocation and no resize/transfer allocation; unescaped literals still allocate zero bytes.
  • Unit tests cover every escape family, surrogate pairs/lone surrogates, line terminators/continuations, legacy octal, malformed escapes, and deterministic OOM state.
  • Focused frontend, lexer, TSan, full unit, and exact affected test262 comparisons pass.
  • Exact-parent evidence records wall, instructions/cycles/energy availability, allocation requests/bytes, peak RSS, checksums, and growth for 1K/2K/4K rows without hiding source construction or decoding outside the timed parse boundary.
8 days ago

Completed and pushed directly to main:

Escaped strings now validate and size without allocation, encode each escape into fixed four-byte storage, then fill one exact arena slice. The one-escape common case copies prefix/escape/suffix directly; multiple escapes share the same fixed decoder used by template cooking. Unescaped strings remain source-backed. WTF-8 lone surrogates, surrogate-pair folding, all line continuations, Annex B legacy flags, malformed-escape offsets, and retry after deterministic OOM remain exact.

Seven-pair exact-parent diagnostic rows remove exactly one allocation/resize request per literal per parse. Total parser requests fall 33.0-33.2%, allocated bytes 8.0-9.8%, and stable retired instructions 2.6-3.1%. The ordinary 4K control preserves allocation requests/bytes exactly. Wall and energy remain explicitly diagnostic because dispersion exceeds the quiet-reference limit.

Evidence: summary

Verification: lexer 18/18; focused frontend 12/12 Debug and TSan; full units 1,615 passed / 1 expected skip / 0 failed / 0 leaked; exact test262 string 73→73, directive prologue 62→62, and template literal 57→57, with zero failures and zero regressions.

Sign in to comment on this issue.