ReviewOS

zig-utils/zig-regex

Backtracker: quantified alternations with captures disagree with ECMAScript (218 fuzzed patterns)

#30
Open glennmichael123 opened this 8 days ago ·
8 days ago

Found by the randomized differential run for #27. The runs are executed through Regex.compileWithFlags(..., .{ .ecmascript = true }), the flags zig-js uses, against node v24.4.1 on the same pattern, input and lastIndex.

After #27 (bc14170), the Thompson engine agrees with node on every run, across 35,980 runs from two seeds over 8,493 generated patterns. What still disagrees is all on the backtracker: 521 runs over 218 distinct patterns, mostly quantified alternations with captures, which requiresBacktracking routes there. These disagreements predate #27; the committed engine shows them too.

Representative rows (byte offsets; captures listed in order, with undefined for a group that did not participate):

patterninput (lastIndex)nodezig-regex backtracker
(((.|b*)){1,2}){2}."BB"match 0..2no match
((b*){2}[ab]|$)\b" dBBAdb "match 6..7no match
((ab|a)(ab){0,3}){0,2}"BaabBa" (1)1..4, groups aab, a, ab1..4, all undefined
(?:((a+){2,3}|(.){3})){0,3}"aABBb"0..3, groups aAB, undefined, B0..3, all undefined
((.)+|($)?)""group 3 undefinedgroup 3 ""
(?:((a?|$)){1,})?""groups undefinedgroups ""
((?:(b?|ab)){3}){1,2}"B"groups ""groups undefined
(?:((\b){3}|\ba)){2,}(?:b?){3} /i"adAaAa"0..1, group 1 a0..0

The families that stand out:

  • match / no-match disagreements;
  • captures lost on a quantified alternation's later iterations;
  • undefined versus "": the backtracker records an empty capture where RepeatMatcher's empty-iteration rule fails the iteration, and the reverse.

The backtracker already knows canMatchEmpty(.backref) returns false incorrectly (noted on #29), and matchStarGreedy keeps only the longest iteration end (backtrack.zig:1019-1048). Both look related.

Reproduce with the generator and harness from #27: gen.js <seed> <patterns> writes node's answers as JSON, and a zig test root runs them through findFrom. I'll attach both if useful.

8 days ago

Landed the bounded greedy-repeat capture rollback in 601edb4, pushed to main. This fixes one family, so this issue stays open.

The failed optional trial in matchRepeat cleared captures belonging to the last accepted iteration. The fix restores that previous matchState on failure/no progress, matching the existing unbounded and reverse-repeat handling. It changes neither refusal policy nor the step budget.

Validation on Zig 0.17.0-dev.1441+d5181a9 / macOS ARM64:

  • New focused regression: failed on the parent, passed after; eight Node-confirmed cases exercise both find and findFrom, including empty/lazy/reverse/capture-clearing controls.
  • zig build test-core -j1 --summary all: 369/369 passed. This is the bounded core gate, not the full fuzz/stress/benchmark suite.
  • Preserved single-corpus differential: 21,575 inputs; 20,775 compiled and 800 refused on each revision; 308 -> 141 mismatches. Exact mismatch-index set comparison: 167 corrected, zero new mismatches. Thompson mismatches remain zero.
  • Independently re-executed all 21,575 saved oracle answers against Node v24.4.1: zero differences.

Corpus SHA-256: c2bc3f553f585bd09edbb380c66f679a72fe989a1f48daea34df663049e2cdbb. This is the preserved single corpus, not the issue's older two-seed 35,980-run inventory. The same harness and inputs were run on parent 1d3e5c9 and the candidate in guarded, resumable ranges; interrupted runs were not counted. No performance claim.

Consumer pin/regression and targeted test262 integration are tracked separately in zig-js#988. The remaining 141 differential mismatches still need investigation.

8 days ago

Published the measured dependency-cost controls in zig-js commit 0d0bb75cb89f3ee1c96963df82288c1fdc4fa741, after freezing the fixture in c47efccf7aa601acc169cf1fdd724f48cc45bbf7.

Four identical-output library controls, seven alternating exact-parent pairs per row, ReleaseFast, Apple M2 Pro. Every sample exceeded 50 ms and matched its expected full result and checksum. Median candidate/parent elapsed-time ratios were 0.998, 1.002, 0.987 and 0.999; sample RSDs were 0.68–2.61%. No samples were discarded. Source, collector and executable hashes, sample uniqueness/order and direct-parent ancestry were independently validated.

This is a desktop diagnostic, not an end-to-end engine/JSC score, a speedup claim, or proof of negligible overhead generally. Thermal, instruction and energy data were unavailable. README has a linked, scoped table; generated scorecards were not changed.

Publication gates: dependency audit 16 classified edges / 0 migration targets / 65 tools; 340 documentation files with resolving links; 51-page docs build with accepted manifest. Guard peak 179 MB. The correctness result remains 167 corrected differential inputs and zero newly mismatching IDs, with 141 mismatches still tracked in zig-utils/zig-regex#30; consumer qualification is in zig-utils/zig-js#988. Hosted CI is still queued, not claimed green.

8 days ago

Additional qualification of zig-regex 601edb49 / zig-js 9da999ae, after the original Debug checks:

  • Direct library root built with zig test -OReleaseSafe -Mroot=src/root.zig -lc: 189/189 passed, guard peak 465 MB.
  • Existing tests/regression.zig fixture, with ReleaseSafe explicitly applied to both the test root and imported regex module: 82/82 before and after, maximum build/run guard footprint 464 MB.
  • The same preserved 21,575-input differential corpus in explicit ReleaseSafe reproduced exactly the same mismatch IDs as Debug, not merely the same counts: 308 before, 141 after; 800 refusals on each side; 167 corrected IDs and zero new mismatches. All 44 bounded range runs completed.
  • Full focused zig-js runtime gate with real pinned zig-gc: 28/28 before, 29/29 after, where the extra case is the new capture regression. Highest guarded process-tree footprint was 2117 MB.

Build-mode caveat: zig build test-core -Doptimize=ReleaseSafe passes 369/369, but its build output shows the library's own root test remaining in Debug. The direct-root and regression commands above explicitly qualify ReleaseSafe rather than treating the build flag as proof.

The earlier 992-case targeted test262 qualification remains valid. A broader whole-RegExp run was terminated by SIGTERM with no final summary (640 MB peak, no guard memory/timeout breach; sender unknown); it is not counted as passing. An all-case retry now uses the runner's existing checkpointable worker mode, with unchanged tests, both binaries, and the shared 3072 MB guard. Its final result is still pending.

No additional engine fix or policy change was made. This does not resolve the remaining 141 mismatches in zig-utils/zig-regex#30 or claim that the full unit/fuzz/CI suites are green. Related consumer issue: zig-utils/zig-js#988.

8 days ago

Extended exact-parent qualification — September 24

All 1,879 pinned test/built-ins/RegExp cases have now been attempted on both preserved Debug binaries. The audited result is:

OutcomeParentCandidate
Positive pass1,6861,686
Expected-negative pass192192
Resource-incomplete11
Observed assertion/parse/negative failures among completed cases00

All 1,878 completed case IDs agree. The resource-incomplete case is character-class-escape-non-whitespace.js: an isolated process exceeded the unchanged 3072 MB guard on both builds (termination observations 3461/3421 MB). It is not counted as a pass or skip. The exact test, raw receipts, and follow-up acceptance criteria are tracked in zig-utils/zig-js#989, linked to the owned-pipeline memory work in zig-utils/zig-js#473. No claim about the leak mechanism or a memory improvement is made.

The separate 339 String-method and 238 regex-literal cases also have identical passing before/after results. Removing overlap with the earlier 992-case qualification gives 2,455 unique passing cases plus 1 resource-incomplete case, out of 2,456 attempted on each build. This is not a full-subtree or full-test262 pass claim. No source inputs, assertions, skip rules, optimization mode, or memory cap were weakened.

Provenance: engine bcc32b717ee2765d2d1143ff0f697390eda642e7, real GC 0285ef801dd221a4ae8e88b31a09ae31627657a3, zig-regex exact parent 1d3e5c9e3f2fb1c421de6433cab503220988617f versus 601edb49ba0579f1a30e4318290a7d7fb919aeed, and clean test262 4249661388e5d3f92a85186213da140a6481490f. Before binary SHA-256: 61f4c677eb2c5d0b3b46529ec952449d77600008bab31fbfa30c5019e35ac87e; after: bd1232b0ae43b6ac27433ad48edc8ffc55ef592d93ec1a655563f3026c0e9817.

Every heavy run used the shared machine lock and explicit 3072 MB ceiling. Interrupted earlier attempts are preserved, excluded from completed-batch counts, and not silently reclassified. Five earlier SIGTERM attempts had no identified sender; one later lost supervisor had no recorded exit/signal. Those are distinct from the reproduced guard exits. The final driver exited 0 after recording all attempts; that driver exit is not a full-pass test result. A fresh process check found no surviving qualification workers.

Auditors: audit-regexp-resource-qualification.js and audit-disjoint-corpus.js; result regexp-resource-qualification.json and raw per-case/batch receipts retained under /private/tmp/zig-js-sep23-verification.xkalXk on the measurement host. The first auditor validates every index, exit, terminal marker, before/after outcome, corpus revision, binary identity, and raw native log; it reports allCasesAttempted: true, complete: false rather than hiding the blocked case.

The original correctness result remains 308 -> 141 differential mismatches, 167 corrected IDs, zero new mismatch IDs over 20,775 compiled inputs (800 unchanged refusals), identical in Debug and explicit ReleaseSafe. Broader zig-utils/zig-regex#30 stays open. Integration and measured cost evidence are already pushed on main via zig-utils/zig-js#988; this follow-up adds evidence, not another engine change.

Sign in to comment on this issue.