Parents: #473, #493, #496 Related: #480
Problem
Dynamic-code entry points currently feed String Values to the lexer through physical asStr() bytes or toStringV() slices. Once prepared flat-Latin-1 storage is enabled, those bytes are one byte per UTF-16 unit rather than canonical WTF-8. Non-ASCII source can then be rejected, mis-tokenized, or trip byte-level assumptions even though its JavaScript String content is valid source text.
Affected shared boundaries include direct/indirect eval, $262.evalScript, ShadowRealm.prototype.evaluate, and Function/GeneratorFunction/AsyncFunction/AsyncGeneratorFunction constructor arguments.
Scope
- Present canonical WTF-8 to every dynamic parser/lexer ingress.
- Borrow existing ASCII and canonical WTF-8 StringCell storage with zero conversion allocation.
- Perform one bounded Latin-1-to-WTF-8 conversion only when physical flat storage requires it.
- Preserve ToString ordering, constructor-realm SyntaxErrors, direct-eval lexical semantics, source locations/debug registration, lone-surrogate handling, and OOM propagation.
- Cover ordinary, realm, ShadowRealm, Function, and generator-family parsing with non-ASCII source.
No-workaround rules
- No representation forcing, replacement character, source escaping, invalid-byte tolerance, parser special case, Test262 exclusion, or hidden permanent canonical copy.
- No stacks, queues, jobs, Promise reactions, Worker, or Map/Set iterator changes.
Acceptance
- Every dynamic parser ingress observes exact JavaScript String content under canonical and flat storage.
- ASCII/canonical input remains borrowed; flat Latin-1 conversion is bounded and OOM-visible.
- Focused tree-walker/VM/realm units, relevant nonzero Test262 accounting, and the full unit gate pass with zero regressions.
- Focused witnesses pass with
flat_storage_activedisabled and enabled.
Implementation and evidence
355329dcestablishes canonical WTF-8 at eval, realm evalScript, ShadowRealm.evaluate, and all Function-family parser ingresses.- Canonical and ASCII StringCells borrow their existing bytes; only physical flat-Latin-1 source takes the existing bounded conversion.
- Focused tree-walker/VM witnesses: 2/2 passed under canonical storage and 2/2 with physical flat storage enabled.
- A fail-first allocator induced
OutOfMemoryat flat-source conversion before lexing, with 0 successful allocations. - Test262 dynamic-code subtrees: 620 passed, 0 failed.
- Full unit gate: 1,760 passed, 1 skipped, 0 failed, 0 leaked.