ReviewOS

zig-utils/zig-js

Intl.Collator: per-locale tailorings are missing (sv, da, cs order letters as root)

#977
Closed glennmichael123 opened this 9 days ago ·
9 days ago

Intl.Collator and String.prototype.localeCompare use root collation for every locale, so languages whose alphabet reorders letters sort wrongly. JavaScriptCore and node (both ICU) agree with each other and disagree with zig-js.

Measured, same program under zig-js (Debug test262 --eval), JavaScriptCore (home-tool) and node v24.4.1:

expressionzig-jsJSC / node
"ä".localeCompare("z", "sv")-11
"ö".localeCompare("z", "sv")-11
"å".localeCompare("z", "da")-11
"ch".localeCompare("h", "cs")-11
new Intl.Collator("sv").compare("ä", "z")-11
["z","ä","a","ö"].sort(new Intl.Collator("sv").compare)a,ä,ö,za,z,ä,ö

Swedish and Danish place å ä ö after z; Czech sorts the digraph ch after h. zig-js returns the root order in each case.

What already works

This is a tailoring gap only — the root collation and the option surface are right. All of these match node and JSC today:

  • German "ä" vs "z" (root order is correct for de), German "ß" vs "ss", Spanish "ñ" vs "o", French "cote" vs "côte"
  • case ordering, {numeric: true}, {sensitivity: "base"}, {usage: "search"}, and resolvedOptions().locale

So the comparison pipeline and the CLDR option plumbing are in place; what is absent is the per-locale tailoring table that reorders or contracts elements on top of the root order.

Scope, for whoever picks it up

The engine already generates CLDR-derived tables through tools/gen_cldr_*.ts (locale, numbers, plurals, timedata, tzalias) and Unicode tables through gen_case/gen_norm/gen_grapheme, so the generator pattern exists. A tailoring import is a bigger data question than those: CLDR ships collation rules per locale, and even a reduced set (the ~20 locales whose tailorings reorder primary weights) is meaningfully larger than the current CLDR tables. Worth deciding how much is wanted before building it — a full ICU-equivalent import is likely out of proportion, while the reordering locales alone would close every row above.

test262 does not cover this: Intl.Collator tests assert the option surface and self-consistency, not locale-specific orderings, so the corpus stays green either way.

9 days ago

Implemented on main in 40729c4. Intl.Collator now uses generated CLDR 48.2 primary tailorings for 21 Latin-script locale profiles, including Czech/Hungarian/Welsh/Croatian contractions and Swedish/Danish post-z ordering. The runtime keeps O(1) ASCII weights and a small longest-token table per locale; locales without a profile retain the existing path. Validation: 48/48 locale comparisons exactly matched Node, test262 Intl.Collator stayed 65/65 passing with no regressions, generator output reproduced byte-for-byte, and the full unit suite passed 2,492 with 1 skipped, 0 failed, and 0 leaks.

9 days ago

Independent verification at bcc32b71 using a fresh Debug runner, real CI-pinned zig-gc 0285ef8, and zig-regex 1d3e5c9; Zig 0.17.0-dev.1441, shared 3072 MB memory guard.

The preserved 16-case coll.js probe matches Node exactly in both tree and VM modes, repeated under Asia/Manila, America/New_York, and UTC. This includes Swedish/Danish post-z ordering, Czech ch, German/French/Spanish controls, numeric ordering, base sensitivity, search usage, sorting, and resolved locale.

Important denominator correction: the preserved coll.ts JSC oracle contains only SIX cases, not 16. All six keyed rows match: sv-a-umlaut, sv-o-umlaut, da-aa, cs-ch, sort-sv, and collator-sv. Missing JSC rows were not counted as passes.

No engine change necessary. This confirms the reported fixes on the preserved probe scope, not full ICU-equivalent collation or a new test262 total.

Sign in to comment on this issue.