ReviewOS

zig-utils/zig-js

Wasm: the full spec corpus is not gated in CI, so 13 failures sit on a green main

#880
Closed glennmichael123 opened this 11 days ago ·
11 days ago

main is legitimately green, and the full wg-1.0 corpus has been failing for a long time. Both are true because no CI leg runs the corpus unfiltered.

Every wasm-spec invocation in .github/workflows/ci.yml is filtered:

zig build wasm-spec ... -Dwasm-spec-filter=linking
zig build wasm-spec ... -Dwasm-spec-filter=f32_bitwise

Run without a filter, the same runner reports:

pass=18778  fail=13  skip=479  crash=0

None of the 13 live in linking or f32_bitwise, so no CI leg has ever observed them:

  • #877 — 2 commands where we accept a malformed module (over-long LEB128). A real conformance gap.
  • #878 — 10 commands rejected correctly but with non-spec diagnostic wording, from #296.
  • #879 — 1 behavioral failure, names.wast:630.

Suggested resolution

Add an unfiltered zig build wasm-spec leg, in Debug. Two reasons for Debug specifically:

  1. It is what the existing filtered legs already use (they pass no -Doptimize).
  2. Debug is the only mode where the executor's appendAssumeCapacity capacity asserts exist. While landing 9ce52ff8 I hit a build that aborted 48 times under Debug while ReleaseFast reported a clean 18,778 pass / 0 crash sweep of the identical binary — the asserts are compiled out, so every abort became a silent out-of-bounds heap write the corpus scored as a pass. A ReleaseFast corpus run cannot detect that class of defect at all.

Gating unfiltered would need #877–#879 fixed (or explicitly inventoried as known-failing with a shrinking budget) so the leg starts green. The 479 skips should also be reviewed against #486's "zero hidden skips" contract — they are mostly assert_malformed_text, i.e. .wat text parsing, which is outside the binary runtime by design, but that has not been re-confirmed recently.

11 days ago

Progress: 20728149 closes #877 and #878. The unfiltered corpus is now 18,790 pass / 1 fail / 479 skip / 0 crash in Debug, from 18,778 / 13.

The single remaining failure is #879 (names.wast:630), under investigation. Once it lands, the unfiltered Debug leg proposed here can be added green with no known-failing inventory.

11 days ago

The unfiltered corpus is now green. c67fa08 closes #879, the last behavioural failure.

passfailskipcrash
Debug18,79104790
ReleaseFast18,79104790

From 18,778 / 13 at the start. All three underlying defects were real, and none was visible to any CI leg: a decoder that accepted malformed modules (#877), diagnostics that did not match the spec (#878), and a wasm↔JS boundary that bypassed the engine's own key-escaping contract (#879).

The precondition in this issue is therefore met: an unfiltered zig build wasm-spec leg can be added green, with no known-failing inventory. Recommend Debug, both because the existing filtered legs already run that way and for the assert coverage described above.

Remaining item before closing: the 479 skips should be re-confirmed against #486's "zero hidden skips" contract. They are predominantly assert_malformed_text (.wat text parsing, outside the binary runtime by design), but that has not been re-audited recently.

11 days ago

Leg added in ea3ff5b.

Rather than putting the multi-minute run on every push — the per-push filtering is a deliberate, documented cost decision, which I'd initially read as an oversight — it is a schedule/dispatch-only job mirroring nightly-threadfuzz-tsan. It needs no submodule and no wast2json (the packed runner reads the checked-in tests/wasm/spec/), and it uploads the inventory JSON so a nightly regression can be diffed against the run that introduced it. Debug, for the assert coverage described above.

It starts green: 18,791 pass / 0 fail / 479 skip / 0 crash.

Skip audit, as requested in this issue:

countreasonverdict
430.wat text parsing outside the binary runtimeby design
40NaN payload implementation-defined at the JS API boundarynot observable through JS Number
5Core 1.0 vs Core 2.0 rule choices (engine implements 2.0, matching V8)deliberate, each locked by a named test
4packing generator emitted invalid watdefect — #881

So 475 satisfy #486's "zero hidden skips" contract: each is declared with an explicit reason in the inventory. The remaining 4 are untested surface with no semantic justification, now tracked separately in #881.

That completes both items in this issue. Closing; #881 carries the leftover.

Sign in to comment on this issue.