ReviewOS

zig-utils/zig-js

TextEncoder: canonicalize encodeInto StringData input

#681
Closed chris opened this 9 days ago ·
9 days ago

Parent: #473 Related: #666, #676

Problem

TextEncoder.encode already coerces source through the representation-aware WTF-8 helper, but TextEncoder.encodeInto still uses toStringV and then decodes those bytes as canonical WTF-8. With physical flat Latin-1 storage active, non-ASCII Latin-1 input is decoded as malformed data and emitted as U+FFFD instead of the required UTF-8 bytes, diverging from encode and misreporting written.

Scope

  • Route encodeInto source coercion through the representation-aware WTF-8 helper.
  • Preserve exactly one ToString coercion, destination validation/order, no-partial-scalar writes, UTF-16 read accounting, byte written accounting, lone-surrogate replacement, shared/resizable buffer locking, ASCII borrowing, exception identity, and OOM propagation.
  • Cover exact non-ASCII bytes and encode/encodeInto parity under canonical and physical flat-Latin-1 storage.

No-workaround rules

  • No flat-storage disablement, eager global canonicalization, replacement of valid Latin-1 text, duplicate coercion, alternate host encoder, hidden retry, or Test262 exclusion.
  • No stack, queue, job, Promise, Worker, agent, or Map/Set iterator changes.

Acceptance

  • encodeInto consumes canonical semantic WTF-8.
  • Valid Latin-1 input emits exact UTF-8 bytes without replacement.
  • read remains UTF-16 code units and written remains output bytes.
  • A custom ToString hook runs exactly once before destination writes.
  • The witness passes with canonical storage and with flat_storage_active enabled, then the switch is restored and not landed.
  • Focused Web API Debug/ReleaseSafe/TSan and full units pass with the exact test inventory reported.
9 days ago

Landed fix(encoding): canonicalize encodeInto StringData (#681) on main.

TextEncoder.encodeInto now consumes canonical WTF-8 like encode. The canonical and physical flat Latin-1 witness verifies exact bytes 99,97,102,195,169,195,191, read = 5, written = 7, byte-for-byte encode parity, and one source coercion.

Evidence: focused Debug, ReleaseSafe, TSan, and physical flat-storage witnesses each 1/1; Test262 contains 0 TextEncoder Web API cases, so no corpus pass is claimed; full units 1774 passed, 1 skipped, 0 failed, 0 leaked across 1775 tests.

Sign in to comment on this issue.