ReviewOS

zig-utils/zig-js

Dynamic code: canonicalize flat StringData at parser ingress

#668
Closed chris opened this 9 days ago ·
9 days ago

Parents: #473, #493, #496 Related: #480

Problem

Dynamic-code entry points currently feed String Values to the lexer through physical asStr() bytes or toStringV() slices. Once prepared flat-Latin-1 storage is enabled, those bytes are one byte per UTF-16 unit rather than canonical WTF-8. Non-ASCII source can then be rejected, mis-tokenized, or trip byte-level assumptions even though its JavaScript String content is valid source text.

Affected shared boundaries include direct/indirect eval, $262.evalScript, ShadowRealm.prototype.evaluate, and Function/GeneratorFunction/AsyncFunction/AsyncGeneratorFunction constructor arguments.

Scope

  • Present canonical WTF-8 to every dynamic parser/lexer ingress.
  • Borrow existing ASCII and canonical WTF-8 StringCell storage with zero conversion allocation.
  • Perform one bounded Latin-1-to-WTF-8 conversion only when physical flat storage requires it.
  • Preserve ToString ordering, constructor-realm SyntaxErrors, direct-eval lexical semantics, source locations/debug registration, lone-surrogate handling, and OOM propagation.
  • Cover ordinary, realm, ShadowRealm, Function, and generator-family parsing with non-ASCII source.

No-workaround rules

  • No representation forcing, replacement character, source escaping, invalid-byte tolerance, parser special case, Test262 exclusion, or hidden permanent canonical copy.
  • No stacks, queues, jobs, Promise reactions, Worker, or Map/Set iterator changes.

Acceptance

  • Every dynamic parser ingress observes exact JavaScript String content under canonical and flat storage.
  • ASCII/canonical input remains borrowed; flat Latin-1 conversion is bounded and OOM-visible.
  • Focused tree-walker/VM/realm units, relevant nonzero Test262 accounting, and the full unit gate pass with zero regressions.
  • Focused witnesses pass with flat_storage_active disabled and enabled.

Implementation and evidence

  • 355329dc establishes canonical WTF-8 at eval, realm evalScript, ShadowRealm.evaluate, and all Function-family parser ingresses.
  • Canonical and ASCII StringCells borrow their existing bytes; only physical flat-Latin-1 source takes the existing bounded conversion.
  • Focused tree-walker/VM witnesses: 2/2 passed under canonical storage and 2/2 with physical flat storage enabled.
  • A fail-first allocator induced OutOfMemory at flat-source conversion before lexing, with 0 successful allocations.
  • Test262 dynamic-code subtrees: 620 passed, 0 failed.
  • Full unit gate: 1,760 passed, 1 skipped, 0 failed, 0 leaked.
9 days ago

Completed and pushed to main in fix(parser): canonicalize dynamic StringData ingress (#668).

  • Direct/indirect eval, realm evalScript, ShadowRealm.evaluate, and all four Function-family constructors now present canonical WTF-8 to the lexer.
  • Canonical/ASCII source remains borrowed; only physical flat Latin-1 converts.
  • Focused interpreter/VM: 2/2 canonical and 2/2 physical-flat.
  • Flat fail-first allocator: exact OutOfMemory, 0 successful allocations, no parser entry.
  • Test262: 620/620 across all changed dynamic-code subtrees.
  • Full units: 1,760 passed, 1 skipped, 0 failed, 0 leaked.

No stack, queue, job, Promise-reaction, Worker, or Map/Set iterator implementation changed.

Sign in to comment on this issue.