ReviewOS

zig-utils/zig-js

Intl Collator: preserve locale weights when ignoring punctuation

#581
Closed chris opened this 9 days ago ·
9 days ago

Parents: #475, #460 Related: #580, #528

Problem

A public differential against system JavaScriptCore found two independent Collator ordering gaps while freezing the #580 performance corpus:

  1. German phonebook collation reverses the Öl / Oel ordering.
  2. Thai Collators default ignorePunctuation to true, but zig-js currently lets that option suppress later accent/case distinctions wholesale rather than ignoring punctuation characters only.

Exact sign-normalized repros on 2026-08-15:

Math.sign(new Intl.Collator("de-DE-u-co-phonebk").compare("Öl", "Oel"))
// before: -1
// system JSC / after: +1

Math.sign(new Intl.Collator("th").compare("Alpha", "alpha"))
// before: 0
// system JSC / after: +1

Math.sign(new Intl.Collator("th").compare("resume", "résumé"))
// before: 0
// system JSC / after: -1

The #580 scored workloads include only pair/config combinations whose public signs match JSC, so their performance checksums do not conceal these differences.

Scope

  • Make ignorePunctuation remove only locale-recognized punctuation at the relevant collation level; it must not discard accent or case weights.
  • Correct the effective German phonebk ordering for umlaut expansions versus literal digraphs.
  • Cover both explicit and locale-default punctuation behavior, both argument directions, sensitivity levels, usage: "sort" / "search", and locale-extension versus options resolution.
  • Preserve left-to-right coercion/abrupt completion and the bounded invocation-local key ownership established by #580.
  • Differentially validate sign results against system JSC and add focused engine tests for every repaired class.

Security and no-workaround rules

Inputs and locale/options are attacker-controlled. Do not special-case the repro strings, recognize benchmark source, cache unbounded comparison pairs/keys, skip coercion, or weaken OOM/GC accounting. The repair must operate in the general collation-key/weight pipeline.

Acceptance

  • German phonebook umlaut/digraph fixtures match system JSC in both directions.
  • Thai default and explicit ignorePunctuation ignore punctuation without collapsing case/accent differences.
  • Other sensitivity, numeric, caseFirst, search/sort, and locale-extension behavior has no regression.
  • Focused OOM/bounded-heap and TSan coverage passes.
  • test/intl402/Collator, full Intl402, and full sharded units have no regressions.
9 days ago

Completed on main in f54dea87 — fix(intl): preserve Collator weight ordering (#581).

The repair is general:

  • German umlaut expansion retains original case only in the case-level key; primary/accent keys still fold case, so variant phonebook/search comparisons reach the normalized tie-break with the correct tertiary relationship.
  • Accent and case weights are evaluated before punctuation-ignored equality. Punctuation remains absent from every key and final tie-break, but ignorePunctuation no longer erases genuine accent/case differences.

Evidence:

  • expanded public differential: all 121 configuration/pair signs match system JSC exactly.
  • original #580 public differential: all 128 signs match system JSC exactly.
  • covers phonebook locale extension and option, search base/variant, both directions, uppercase/lowercase umlauts, Thai default/explicit punctuation, every sensitivity, and ASCII/Thai punctuation.
  • Collator test262 65 → 65; full Intl402 3,341 → 3,341, both zero failures.
  • full sharded units: 1,676 passed, 1 skipped, 0 failed, 0 leaked.
  • focused normal and suppression-free TSan coverage pass; #580's 32 MiB bounded-heap/OOM witnesses remain clean in the full suite.
  • #580 ReleaseFast workload checksums remain exact (925849600 direct, 818968495 sort), with diagnostic timings still well below the exact parent.

Flips 0 test262 cases; this repairs implementation-defined locale ordering not represented by the current Collator test262 subtree.

Sign in to comment on this issue.