ReviewOS

zig-utils/zig-js

Strings: activate flat Latin-1 StringCell storage

#777
Closed chris opened this 9 days ago ·
9 days ago

Parent: #473 Related: #480, #493

Problem

The owned StringCell model already classifies Latin-1 content, can store non-ASCII Latin-1 as one byte per UTF-16 code unit, and exposes representation-aware UTF-16 and canonical WTF-8 accessors. Production still leaves flat_storage_active disabled, so every Latin-1 string retains its two-byte UTF-8 form and the completed reader-migration witnesses exercise the real layout only through temporary test flips.

Scope

  • Enable physical flat Latin-1 storage for non-ASCII Latin-1 StringCells.
  • Preserve canonical WTF-8 at parser, property-key, embedding, RegExp, JSON, serialization, diagnostic, and public/private ABI boundaries.
  • Keep ASCII and already-canonical storage borrowed; transcode only at byte-canonical egress.
  • Preserve exact UTF-16 length/index/slice/equality/hash/interning semantics, lone surrogates, moving GC, external owners, and OOM atomicity.
  • Measure owned backing-byte reduction without publishing an unmeasured throughput claim.

No-workaround rules

  • No representation forcing at semantic consumers, lossy replacement, invalid-byte tolerance, permanent canonical shadow copy, disabled assertion, source/input special case, or Test262 exclusion.
  • No stack, queue, job, Promise, Worker, or Map/Set iterator implementation changes.

Acceptance

  • Production constructs eligible non-ASCII Latin-1 StringCells in one-byte-per-unit storage.
  • Canonical, ASCII, Latin-1, BMP, astral, and lone-surrogate content remains exact across construction, interning, property publication, JSON/RegExp, structured clone, embedding, and diagnostics.
  • Allocation/OOM, moving-GC, no-GIL, and TSan witnesses pass without leaks or races.
  • Relevant nonzero Test262 accounting and the full parallel unit gate pass with zero regressions.
  • Exact owned-byte evidence quantifies the representation change; no throughput claim is made without an exact-parent benchmark.
9 days ago

Implementation is active locally and the semantic boundary audit is green so far.

  • Production non-ASCII Latin-1 StringCells now use one byte per UTF-16 unit; the checked café fixture shrinks its owned backing from 5 canonical WTF-8 bytes to 4 physical bytes.
  • Exact-parent Test262 is unchanged across 3,291 relevant files: String 1,223/1,223, RegExp 1,879/1,879, JSON 165/165, and language string semantics 24/24.
  • Full units passed 1,967 with 1 expected skip, 0 failures, and 0 leaks after correcting one stale physical-byte test assertion.
  • TSan passes the external ZigString GC/finalizer lifecycle 1/1, every StringData boundary witness 19/19, and all StringCell concurrency/OOM/index tests 23/23, with zero leaks or races.

Final full-unit rerun and deterministic tier fuzzing are still in progress before landing; no throughput claim is being made from the footprint change.

9 days ago

Completed in perf(strings): activate flat Latin-1 storage (#777).

Production now stores eligible non-ASCII Latin-1 StringCells as one byte per UTF-16 unit while canonical WTF-8 remains the exact parser/property/RegExp/JSON/embedding egress contract. The checked café fixture is 5 -> 4 owned backing bytes; this is footprint evidence only, not a throughput claim.

Exact-parent Test262 is unchanged across 3,291 files (String 1,223; RegExp 1,879; JSON 165; language strings 24), all with 0 failures/flips. TSan is green for 23/23 StringCell tests, 19/19 StringData boundaries, and 1/1 external ZigString lifecycle; full units completed 1,967 pass / 1 expected skip / 0 fail / 0 leak; threadfuzz completed 400 seeded programs with 0 failures.

A later exact-source full-suite retry hit the separate nondeterministic parallel async Thread-keepalive VM-stack failure outside this diff; its isolated witness passed 1/1. No stack, queue, job, Promise, Worker, or Map/Set iterator implementation changed.

Sign in to comment on this issue.