ReviewOS

zig-utils/zig-js

Intl.NumberFormat: drive compact patterns from owned CLDR data

#568
Closed chris opened this 9 days ago ·
9 days ago

Parent: #475 Related: #566

Problem

The #566 precision oracle exposed a separate locale-data gap: new Intl.NumberFormat("hi-IN-u-nu-deva", { notation: "compact", maximumSignificantDigits: 3 }).format(1234567) uses the generic million pattern in zig-js, while the locale-specific CLDR result uses the Indian lakh magnitude and Hindi compact affix. nfCompactPattern is currently a hand-selected locale subset rather than a generated table covering all engine-supported locales.

Scope

  • Generate owned compact short/long magnitude patterns and affixes from the repository pinned CLDR inputs.
  • Select locale-specific exponents, including 3/4-digit and Indian lakh/crore systems, with correct fallback and plural-sensitive long forms.
  • Share the generated immutable data across realms and integrate it with NumberFormat text, parts, ranges, and non-Latin numbering.

No-workaround rules

No locale switch expansion by hand, English suffix fallback presented as locale support, system ICU/JSC dependency, copied output fixtures, or unbounded mutable cache. Edit the generator and regenerate tables; never hand-edit generated data.

Acceptance

  • Focused CLDR-derived short/long controls cover every supported compact-pattern family and fallback path.
  • Exact text/part/range oracles cover Hindi lakh/crore, East Asian 10^4 patterns, German long/short, plural forms, threshold carry, and non-Latin digits.
  • Relevant Intl402 cases gain with no regressions.
  • Construction/steady allocation and exact-parent performance remain neutral or improve.
  • Generator determinism, OOM, bounded heap, moving GC, no-GIL, TSan, and full units pass.
9 days ago

Frozen the generated compact-pattern boundary in 340f52be (bench(intl): isolate generated compact patterns (#568)).

The four controls separate text, parts, range text, and range parts across English, German singular/plural, French exact-1, Hindi/Devanagari lakh, Japanese/Chinese/Korean 10^4 families, Hebrew bidi literals, Russian/Arabic plurals, and threshold carry. At 4 jobs / 5 samples, pre-fix zig-js checksums are 1,731,824 / 1,831,056 / 1,733,952 / 3,994,528; the JavaScriptCore oracle is 1,774,992 / 1,835,088 / 1,777,984 / 4,469,520.

The implementation is being generated from the existing pinned cldr-json@a79b499… input and already matches direct text/part/range probes for the distinct pattern families; the remaining work is generator hardening, full corpus/GC/thread validation, and candidate attribution.

9 days ago

Implemented and pushed directly to main:

The engine now generates immutable short/long compact tables from the pinned CLDR revision, including locale-specific exponents, lakh/crore, 10^4 Asian patterns, plural-sensitive affixes, French exact-one mille, bidi/whitespace part boundaries, non-Latin digits, and exact arbitrary-precision BigInt plural operands. Identical plural rows are deduplicated and insensitive magnitudes bypass plural evaluation; the final table is 381,487 bytes versus about 508 KB before that fast path.

Validation on the landed candidate:

  • NumberFormat test262: 249/249 (249 -> 249; 0 flips, no regressions)
  • full Intl402: 3,341/3,341
  • JavaScriptCore differential oracle: 17/17 byte-exact for text, parts, and the covered shared-affix range
  • focused normal: 10/10; focused TSan: 10/10, including bounded heap, moving GC, parallel no-GIL publication, OOM, exact BigInt, and sink equivalence
  • full units: 1,649 passed, 1 skipped, 0 failed, 0 leaked
  • deterministic regeneration: byte-identical from CLDR revision a79b499916d486dca4b0f74fe423ea457705fdd9

At 4 jobs/5 samples, compact text checksum moved 1,731,824 -> 1,774,992 and parts 1,831,056 -> 1,835,088; both now exactly match JavaScriptCore. Final zig-js medians were 805,084 ns text, 1,814,125 ns parts, 1,150,417 ns range, and 3,602,000 ns range parts.

The remaining range checksum difference is independently caused by locale interval separators and compact-affix collapse/repetition, not compact magnitude data. It is explicitly tracked in #570.

Sign in to comment on this issue.