ReviewOS

zig-utils/zig-regex

ECMAScript case-insensitive backreferences fold only ASCII

#29
Closed glennmichael123 opened this 9 days ago ·
9 days ago

Executed through zig-js (Debug ad1ca106, zig-regex 73cc22d) against node v24.4.1. JavaScriptCore agrees with node. Case-insensitive backreferences fold only ASCII, so every non-ASCII pair fails:

programnodezig-js
/(à)\1/i.exec("àÀ")["àÀ","à"]null
/(σ)\1/i.exec("σς"), and with iumatchnull
/(Σ)\1\1/i.exec("Σσς")matchnull
/(Dž)\1/i.exec("DžDŽ"), /(Dž)\1/iu.exec("Dždž")matchnull
/(ſ)\1/iu.exec("ſs"), /(\w)\1/iu.exec("ſS")matchnull
/(k)\1/iu.exec("KK") (Kelvin sign), /(?<n>k)\k<n>/iu.exec("Kk")matchnull
/(ß)\1/iu.exec("ßẞ")matchnull
/(\u{10400})\1/iu.exec("\u{10400}\u{10428}")matchnull
/(?<=\1(k))x/iu.exec("KKkx") (lookbehind)["x","k"]null
/(é+)x\1/i.exec("ééxÉÉ")matchnull

The non-u negative controls already agree with node, which rejects all of them:

  • /(ß)\1/i on "ßẞ";
  • /(ſ)\1/i on "ſs";
  • /(k)\1/i on "kK";
  • the astral pair without u.

Cause: backreferences always run on the backtracker. Its two comparators have two problems:

  • They fold only ASCII: matchCaptureText at backtrack.zig:1857-1881 (forward) and matchCaptureTextReverse at :1759-1772 (lookbehind).
  • They assume both spans have the same byte length. ſ/s, K/k and ẞ/ß differ in length.

Fix: compare per character through ECMA-262's Canonicalize (22.2.2.7.3), gated on ecmascript + ignoreCase, with separate cursors so the spans may differ in length.

  • With u/v: Canonicalize is simple case folding (unicode_casefold_data).
  • Without u: Canonicalize is the toUppercase rule. It keeps characters whose uppercase has more than one code point, and non-ASCII characters whose uppercase is ASCII (the singletons ſ, K, Å, ẞ, …).
9 days ago

Taking this now, together with the other one filed alongside it; the fix is small and I'm gating it immediately.

9 days ago

Fixed in 452345d; pinned through zig-js in zig-utils/zig-js@15f93fe.

Both backreference comparators, forward and lookbehind, now walk the two spans character by character on separate cursors and compare canonicalizeKey. Under u/v that is simple case folding. Without them it is the same table, except for 67 code units that non-u Canonicalize keeps as themselves; that list was checked against the spec algorithm over every BMP code unit. The change is gated on ecmascript + case_insensitive, so the PCRE path is unchanged.

Evidence:

  • A 34-row zig-regex table matches node v24.4.1 on every row. It covers i/iu, lookbehind, named groups, length-changing pairs, and the non-u negatives ß/ẞ, ſ/s and K/k.
  • Through zig-js, 22 of 22 probes match node (7 before).
  • test-core: 365/365.
  • test262 Debug before/after is identical over String regex methods, RegExp and literals/regexp (1,970 executions), with 0 flips.

Sign in to comment on this issue.