ReviewOS

home-lang/home

Fix TextEncoder.encodeInto scalar boundaries and preserve Deno skips

#608
Closed chris opened this 9 days ago ·
9 days ago

Scope

  • Remove the stale Zig TextEncoder__encodeInto16 fallback that writes U+FFFD when the first valid astral scalar needs four bytes but the destination has only three.
  • Sync the pinned upstream TextEncoder corpus and prove its expanded scalar-boundary coverage.
  • Stop rewriting upstream Deno/Web test.ignore / ignore: true declarations or altering surrogate fixtures in the native corpus runner.
  • Preserve upstream pass/skip accounting in both strict and routed execution.

Acceptance criteria

  • TextEncoder.encodeInto("😀", Uint8Array(3)) returns { read: 0, written: 0 } without mutating the destination.
  • Invalid lone surrogates still encode as U+FFFD when three bytes are available.
  • The selected upstream Deno/Web fixtures run unchanged and match pinned Bun accounting.
  • Debug and ReleaseSafe native runs pass with zero failures and zero unsupported tests.
9 days ago

Completed in 6d9bf217be80896f40320bd888f266b1180ea293 (fix(runtime): preserve text encoder scalar boundaries).

What changed:

  • Ported current Bun UTF-16 → UTF-8 fixed-buffer semantics into both maintained Home string-module copies. Valid astral scalars are now atomic, while lone surrogates still emit U+FFFD when three bytes fit.
  • Removed the stale TextEncoder__encodeInto16 forced-replacement fallback.
  • Added a low-level native regression covering 3-byte astral refusal, exact 4-byte astral output, and exact 3-byte lone-surrogate replacement.
  • Deleted ten Deno/Web corpus rewrite functions and routed those paths unchanged, preserving upstream test.ignore, ignore: true, and the valid surrogate fixture.
  • Routed generated Deno TypeScript through Home’s production Bun parser and matched Bun’s shared own Event.isTrusted getter descriptor.

Verification against pinned Bun 4982b91e3702094330f3be3883354c52b8c01323:

  • All 10 authoritative packages/runtime/test/bun-corpus fixtures are SHA-256 byte-identical to ~/Code/bun/test.
  • Pinned Bun selected corpus: 136 pass / 14 skip / 0 fail.
  • Home Debug strict: 136 pass / 14 skip / 0 fail.
  • Home Debug routed: 136 pass / 14 deferred TODO / 0 fail / 0 unsupported.
  • Home ReleaseSafe strict: 136 pass / 14 skip / 0 fail.
  • Home ReleaseSafe routed: 136 pass / 14 deferred TODO / 0 fail / 0 unsupported.
  • Entire 16-file Deno directory in pinned Bun and Home Debug strict: 226 pass / 17 skip / 1 TODO / 0 fail.
  • Core runtime units: 1,821 pass / 19 skip / 0 fail.
  • Direct native probe now returns {read:0,written:0} with an unchanged 3-byte destination for 😀, and {read:1,written:3} with EF BF BD for a lone lead surrogate.

The broader routed Deno directory improved from 4 failures / 3 TODO to 2 failures / 17 TODO; its remaining URL compatibility failures are outside this focused batch and strict production execution already matches Bun exactly.

Sign in to comment on this issue.