ReviewOS

zig-utils/zig-js

Strings: borrow already-valid UTF-8 at encoding boundaries

#562
Closed chris opened this 9 days ago ·
9 days ago

Parent: #473 Related: #480, #493, #561

Problem

wtf8ToUtf8Bytes constructs a new arena-backed byte list for every input, even when the JavaScript string is already valid UTF-8 and no surrogate pair or lone-surrogate replacement exists. TextEncoder, URL/form encoding, Blob/body, filesystem, and other UTF-8 egress paths therefore retain a complete no-op copy before their actual consumer establishes final ownership.

Required behavior

  • borrow already-valid UTF-8 byte-for-byte with zero conversion allocation;
  • combine a WTF-8 high/low surrogate pair into its exact scalar encoding;
  • replace every lone surrogate with U+FFFD exactly as the Encoding Standard requires;
  • use one bounded, overflow-safe allocation only when transformation is necessary;
  • preserve source lifetime through every consumer and propagate OOM without partial output.

No-workaround rules

  • no skipped lone-surrogate replacement, invalid UTF-8 emission, lossy normalization, ASCII-only public shortcut, or system encoding library;
  • no unbounded cache or representation-dependent pointer assumption;
  • consumers that require owned bytes must still establish their own exact ownership.

Acceptance

  • fixed-zero allocator tests prove valid ASCII/BMP/astral UTF-8 borrows without allocation;
  • transformed pair/lone-surrogate output is byte-exact, one-allocation, and OOM-safe;
  • exact-parent TextEncoder rows cover ASCII, BMP/astral valid UTF-8, lone surrogates, and paired surrogates with frozen checksums;
  • TextEncoder/encoding test262, focused TSan, full units, and relevant Web/API tests pass with zero regressions.
9 days ago

Completed on main:

  • e7df6540 adds frozen TextEncoder controls for ASCII, valid Unicode, lone surrogates, and paired surrogates.
  • 034e431c borrows already-valid UTF-8 with zero conversion allocations; transformed WTF-8 uses one input-sized allocation, combines surrogate pairs exactly, and replaces lone surrogates with U+FFFD.

Seven alternating exact-parent medians improve 41.0% / 44.0% / 14.4% / 47.3% across the four rows with checksums unchanged at 217,792 / 331,328 / 266,688 / 267,200. Fixed-zero and measured-allocation tests cover borrow, one-allocation transform, and OOM behavior.

Validation: full units 1,639 pass / 1 skip / 0 fail / 0 leak; focused TSan clean; conformance 33/33. test262 has no TextEncoder/Web Encoding subtree, so the adjacent standard URI encoding subtrees were compared exactly: encodeURI 31/31, encodeURIComponent 31/31, decodeURI 55/55, decodeURIComponent 56/56 on parent and candidate (173/173, 0 flips/regressions).

Sign in to comment on this issue.