ReviewOS

zig-utils/zig-js

CI: serialized OOM witness can throw an unprintable exception

#100
Closed chris opened this 9 days ago ·
9 days ago

Fresh main run 29364341824 failed threads-gil (shard 0/4) after 3cc2a58c.

Exact failing case:

  • semantics/oom-one-thread.js
  • elapsed: 13,305 ms
  • runner diagnostic: FAIL ...: ?
  • every other shard-0 corpus case passed (57/58)

This case runs with a 192 MiB heap limit and expects a catchable OOM on one allocating thread while siblings complete and all threads recover. The ? means either the thrown value was absent/unstringifiable or diagnostic stringification itself failed under pressure, so the exact semantic failure is currently hidden.

Acceptance:

  • reproduce the case independently before changing behavior;
  • preserve the real 192 MiB pressure and catchable-OOM/recovery assertions;
  • determine the original exception without weakening the witness;
  • add focused diagnostics or regression coverage for the root cause;
  • pass the focused case in serialized and no-GIL modes as applicable, then one full CI corpus gate.

Evidence: https://github.com/zig-utils/zig-js/actions/runs/29364341824/job/87192101292

9 days ago

Independent focused reproduction command: zig build threads-test -Dthreads-case=semantics/oom-one-thread.js

Result on the same main source/dependencies: PASS in 7,205 ms at the real 192 MiB cap. No behavior changed.

This narrows the failure to cumulative whole-shard state/interleaving: CI reached this case after 53 earlier corpus files in the same process, then ctx.evaluate failed and diagnostic stringification returned ?. Once the remaining jobs settle, I will rerun only failed jobs once. A green isolated rerun will not close this issue; we still need either a cumulative reproduction or allocation-free diagnostics that preserve the original exception.

9 days ago

The failed-only GitHub rerun executed the complete 58-case serialized shard again and passed: https://github.com/zig-utils/zig-js/actions/runs/29364341824/job/87201195763

Current evidence:

  • first cumulative shard attempt: oom-one-thread.js failed after 53 prior cases with unprintable ? exception
  • focused standalone case at the same cap: pass in 7,205 ms
  • full failed-only shard rerun: pass

Keeping #100 open. This is intermittent cumulative-pressure/diagnostic debt, not a justified runtime change or a reason to weaken the OOM witness.

9 days ago

Diagnostic hardening is now on main as 604afc81 (test(threads): preserve OOM diagnostics (#100)). The runner no longer discards the original evaluation error or collapses a failed JS ToString to ?: successful rendering includes the evaluation error, while the allocation-free fallback reports the evaluation error, exception type/error class, and stringification error without invoking JavaScript. The exact 192 MiB focused witness remains unchanged and passes in 7,498 ms. Keeping #100 open until a cumulative-pressure recurrence identifies the underlying intermittent runtime outcome.

9 days ago

Post-604afc81 non-recurrence update (July 16): five subsequent complete main CI runs passed, including every serialized corpus shard:

The diagnostic hardening has therefore survived repeated cumulative full-corpus execution, but the original intermittent runtime outcome has not recurred and cannot yet be identified. Keeping #100 open with the 192 MiB witness unchanged. The next recurrence should now report the original evaluation error, exception type/error class, and any stringification error instead of ?; that evidence—not speculative behavior changes—will determine the root fix.

9 days ago

Cumulative-pressure reproduction attempt (July 18, HEAD ebfdc3be), following the same protocol as the earlier updates:

  • Focused serialized witness: zig build threads-test -Dthreads-case=semantics/oom-one-thread.js → PASS (10,863 ms at the real 192 MiB cap).
  • Full cumulative serialized shard (zig build threads-test -Dthreads-shard-index=0 -Dthreads-shard-count=4, one process, CI's exact command): 4 consecutive attempts, all green — 58/58 corpus files passed each; the witness ran at its original cumulative position 53/58 and passed in 10.3–11.7 s every time (attempt wall times 391–510 s).
  • Focused no-GIL witness (parallel-js): PASS ×2 (9,338 ms / 9,454 ms).

No FAIL lines and no hardened exception=…, stringify=… diagnostics appeared in any of the 232 cumulative case-runs, so the 604afc81 diagnostic path was never exercised — the intermittent ?-exception outcome did not recur under whole-shard cumulative state. Keeping this issue open per the standing criteria: the 192 MiB witness stays unchanged, and the next genuine recurrence will now report the original evaluation error, exception type/error class, and stringification error instead of ?.

9 days ago

Resolved by the allocation-free diagnostic and reserved-OOM work already landed in 604afc81, with current-main cumulative verification completed today.

Current evidence (witness unchanged at the real 192 MiB cap):

  • focused serialized: PASS, 10,421 ms;
  • focused no-GIL: PASS, 9,766 ms;
  • current weighted serialized shard containing the witness: 61/61 PASS in 399,701 ms; semantics/oom-one-thread.js passed at position 57 in 10,818 ms after allocation-steal GC, OSR/catch, relocating-Wasm, watchpoint, Atomics, transition, and object-model stress;
  • the separate shard-0 CI gate also passed 60/60 in 539,884 ms.

The runner now preserves the host evaluation error and has an allocation-free fallback that prints exception type/error class, own message, and stringification failure. Context heap_limit_bytes names an escaping top-level OOM proves the reserved OutOfMemoryError is always present without allocating, and the recovery-pressure regression proves successful allocations are never torn. The original one-off semantic outcome has not recurred across the historical repeated shards or these current gates; importantly, any future unrelated recurrence can no longer collapse to ?. No cap, recovery assertion, or OOM-shape check was weakened.

Sign in to comment on this issue.