ReviewOS

zig-utils/zig-js

Threads: fix failing corpus cases mc-lock-stop-vs-park and oom-one-thread

#722
Closed chris opened this 9 days ago ·
9 days ago

Parent: #478, #502 Related CI: `threads-gil`, `nogil-corpus-functional`

Problem

Two PR-249 thread corpus cases fail on Linux CI and keep otherwise-green gates red:

  • `cve/mc-lock-stop-vs-park.js` — `Error: expected 400 but got 401` (observed in nogil-corpus-functional shard 1/4 on run 32589497696 and repeatedly in earlier runs referenced from #662/#663)
  • `semantics/oom-one-thread.js` — `OutOfMemoryError: Context heap limit exceeded` (observed in threads-gil shard 3/4 of the same run)

These are engine behavior mismatches or overly strict case expectations, not sanitizer aborts. Until fixed (or the cases are corrected with evidence), the comprehensive gate boxes in #662 and #663 cannot close because every descendant matrix records these failures.

Required work

  • Diagnose `mc-lock-stop-vs-park`: identify which lock/park/stop state machine transition produces 401 instead of 400; fix the engine or justify a case correcti
9 days ago

Run 32592838252 narrows this issue: `cve/mc-lock-stop-vs-park.js` passed in the no-GIL functional gate this time (it failed in run 32589497696's shard), while `semantics/oom-one-thread.js` failed again in threads-gil shard 3/4. The stop-vs-park case looks flaky/order-dependent rather than deterministically broken; oom-one-thread remains deterministic.

9 days ago

Diagnosed and fixed the deterministic half in bf069289 (`fix(gc): wait bounded for peer parks during heap-cap recovery (#722)`).

Root cause (`semantics/oom-one-thread.js`): allocation-failure recovery in GIL mode refused to collect unless every peer was already parked-and-published. Under sustained multi-thread allocation that predicate almost never held at the failing instant, so after the hog released its ~256MB hoard it hit a second OOM inside its one-retry tolerance window before any peer reached its next natural park — an uncaught `OutOfMemoryError`, exactly the CI failure. Recovery now waits bounded (50 ms) on the same all-parked safety predicate before refusing; peers park on their own via their existing Atomics.wait backoffs and GIL release, so no new handshake, lock, or shared state is introduced, and timeout preserves the old refusal exactly.

Evidence: 10/10 case passes in GIL mode and 5/5 in no-GIL mode locally; 46/46 heap-filtered un

9 days ago

Second-stage diagnosis and fix in a2f30927 (`fix(gil): publish blocked acquirers so heap-cap recovery can collect (#722)`).

bf06928's bounded wait was necessary but not sufficient: `allOthersParked` requires every peer to be parked-and-published, yet a peer blocked on contended GIL `acquire()` blocks opaque in `lockUncancelable` with no published scan range. Once any peer queued on the lock — which is the steady state once the hog holds it — the predicate could never become true while recovery held the GIL, so recovery still refused and the case kept dying with the uncaught second OOM (run 32599447828, threads-gil shard 3/4, same 12 s signature). Contended `acquire()` now publishes the blocked stack exactly like `wait`/`yieldIfContended` already do: the blocked stack is stable and the conservative scan roots its temporaries, so a collector holding the GIL can safely includ

9 days ago

Precise diagnosis from run 32606818819's runner diagnostics (threads-gil shard 3/4):

  • `heap budget: used 201326495 / limit 201326592` — the heap is saturated to within 97 bytes of the cap at the moment of failure
  • thread state: id=1 `threw=true`, all joins settled — consistent with the hog's post-release retry allocation, i.e. recovery ran but freed nothing

Conclusion: this is no longer a liveness/scheduling problem (both prior fixes stand — publication of blocked acquirers is independently correct). The hoard becomes garbage precisely, but the hog's own native stack retains stale word-copies of hoard-element pointers from the allocation storm, and the conservative scan therefore pins every element array as live. A full collection legitimately frees zero bytes, so the retry OOMs again and the second throw escapes. That also explains platform variance: whether stale slots survive depends on stack reuse, which differs between macOS an

9 days ago

Fresh descendant-main reproduction in CI 32804870572, no-GIL functional shard 1/4: cve/mc-lock-stop-vs-park.js failed after 438 ms with expected 400, got 401; all six thread completion records were done=true exited=true threw=false joins-settled=true. The shard passed 62/63 cases. This run contains only unrelated Math/iterator-protocol changes after the previously known reproduction.

9 days ago

Fresh reproduction on 81ce859 in scheduled CI, no-GIL functional shard 1/4: 62/63 pass; cve/mc-lock-stop-vs-park.js throws expected 400 but got 401 after 475 ms. All six thread records report done=true, exited=true, threw=false, joins-settled=true. This is the existing functional boundary, not a sanitizer abort. No corpus expectation or lock/park implementation was changed by the in-progress #790 declaration work.

9 days ago

CI recurrence on 0ad2b5d9 in run 33140348656 / nogil-corpus-functional (shard 1/4):

  • cve/mc-lock-stop-vs-park.js failed in 375 ms with Error: expected 400 but got 401.
  • The workflow otherwise completed with 52 successful jobs and one intentional skip; this was its only failure.
  • This is the existing #722 witness, so I am recording the recurrence here rather than attributing it to the synchronous Proxy metadata change or opening a duplicate.

Raw failed-job log is retained locally at /Users/chris/.cache/zig-js-proxy.Hdh6eC/ci-33140348656-nogil-functional-shard1.log.

9 days ago

Fresh recurrence at 6bb4b77 in CI 33204727283, nogil-corpus-functional shard 1/4: 62/63 passed; cve/mc-lock-stop-vs-park.js failed after 486 ms with expected 400 but got 401. All six completion records are done=true, exited=true, threw=false, joins-settled=true. The saved log is /Users/chris/.cache/zig-js-class-name.mw5loL/ci-nogil-functional.log. This is the existing functional witness, not a sanitizer failure; no native lock/park/stack/job or corpus expectation changes are being made in the separate class naming work.

9 days ago

The same functional mismatch recurred on CI 33209229291, at 4f7718d: nogil-corpus-functional shard 1/4, job 98978114065, cve/mc-lock-stop-vs-park.js fails after 513ms with Error: expected 400 but got 401. The job reports one regression against pr249-execution-nogil.json. This is the case assertion, not a sanitizer abort. The class-key and program-exit fixes did not alter this protected corpus or the lock/park/stop lifecycle. The exact job log is retained; the matrix is not green.

9 days ago

Completed by the combined #722 landings, with the final remaining case fixed in 35b67e61 — test(threads): close stop sentinel race (#722).

The recurring 401 was not an engine stop/park transition: the stopper could load stop === 0, then the conductor could publish stop = 1 followed by wake-up sentinel req = 401, and the stopper could legally acquire/count that sentinel before reaching its next loop condition. Sequential consistency permits that total order. The case now rechecks stop after acquiring req; observing 401 therefore observes the earlier stop publication and exits, while a real round acquired before stop still completes. The expected count remains exactly 400, and the waiters, cell-lock contention, native parks, and 400-round stop storm are unchanged.

Evidence for the final case:

  • current-main uncorrected case: failed first run, expected 400 / got 401
  • corrected no-GIL case: 50/50 passed
  • corrected serialized case: 10/10 passed
  • corrected unsuppressed TSan no-GIL case: 1/1 passed
  • reference inventory audit: 253/259 promoted, 0 blocked/uncategorized
  • terminal disposition probes: 5/5 matched

The earlier oom-one-thread half remains implemented by the #722 GC/GIL landings; after #828 replaced the above-cap grace with a precharged hard-cap reserve, that exact case still passes 1/1 in GIL and 1/1 no-GIL. Both original issue failures are now resolved without weakening either oracle.

Sign in to comment on this issue.