ReviewOS

zig-utils/zig-js

CI: midgc termination-reaction seed can miss a parallel collection

#96
Closed chris opened this 9 days ago ·
9 days ago

Fresh main run 29355087082 failed threadfuzz-midgc after f4602d8.

Exact CI command: THREADFUZZ_SEED_TIMEOUT_MS=300000 zig build threadfuzz -Dfuzz-midgc=true -Dfuzz-iters=20

Exact failure: seed 20: midgc termination-reaction did not finish a parallel collection

The job completed 900 programs with one failure in about 798 seconds. This is separate from the fixed metadata-ownership TSan shard and predates the prototype-hoist commit.

Acceptance:

  • Reproduce the exact seed/case independently before changing behavior.
  • Determine whether collection progress, termination/reaction scheduling, or the witness timeout is wrong.
  • Preserve exact lifecycle semantics and avoid weakening the assertion merely to hide a race.
  • Add focused regression coverage.
  • Validate the focused case under normal and TSan builds, then let the full midgc profile run once in CI.

Evidence: https://github.com/zig-utils/zig-js/actions/runs/29355087082/job/87160648718

9 days ago

Confirmed the exact failure in completed run https://github.com/zig-utils/zig-js/actions/runs/29357367045/job/87168344170:\n\n- command: THREADFUZZ_SEED_TIMEOUT_MS=300000 zig build threadfuzz -Dfuzz-midgc=true -Dfuzz-iters=20\n- seed 20: midgc termination-reaction did not finish a parallel collection\n- 900 programs from seed 1, 1 failure\n- process exit 1 after about 785 seconds\n\nThe workflow was later cancelled by a newer main push, but this job had already completed and the failure is real. It matches this issue exactly and is unrelated to the #98 VM quick-path change; no speculative code change was made.

9 days ago

Current main validation note: both threadfuzz-midgc and tsan-threadfuzz-midgc passed in https://github.com/zig-utils/zig-js/actions/runs/29364341824. Keeping #96 open because one green main run does not reproduce or explain the prior seed-20 missed-collection failure, and its acceptance requires a focused root cause/regression.

9 days ago

Fresh main-branch reproduction on e4b72407:

  • CI run: https://github.com/zig-utils/zig-js/actions/runs/29369732113
  • failing job: threadfuzz-midgc
  • command: THREADFUZZ_SEED_TIMEOUT_MS=300000 zig build threadfuzz -Dfuzz-midgc=true -Dfuzz-iters=20
  • result: seed 20: midgc termination-reaction did not finish a parallel collection
  • summary: 900 programs from seed 1, 1 failure, wall 786581 ms

All other jobs in that run completed successfully. This predates and is unrelated to #102.

9 days ago

Root cause reproduced locally under deliberate CPU/process contention on current main.

The focused command passes in isolation, but concurrent repetitions reproduced both scheduler-sensitive safe-abort paths:

  • publication starvation: 5 attempts, 5 aborts, 4 publication-timeout aborts, 1 round-limit abort;
  • continuous born-cell churn: 2 attempts, 2 round-limit aborts, 128 born-growth rounds, 256 successful peer publications, and 0 finish attempts.

The collector is behaving according to its abort-safe contract: it must not sweep until peer roots publish and the born-cell set has a stable round, because a just-published cell can still have an initializing payload. The witness was wrong to require a finishing sweep after a finite allocation burst without providing the non-allocating convergence window already required by the collector design and the core M3 test.

The fix keeps the strict finishing-sweep assertion and exact termination/asyncJoin/waitAsync cleanup oracles. It adds a non-allocating compute window while all termination roots remain live, and adds failure-only counter deltas so any future miss identifies publication timeout, round limit, deferred work, born growth, and peer publication behavior.

Focused validation:

  • pre-fix: reproduced 1 failure in 32 contended runs and 1 failure in 40 contended runs across seeds 6, 10, 18, and 20;
  • post-fix: 40/40 contended focused runs passed;
  • TSan: seeds 6, 10, 18, and 20 all passed individually with no sanitizer report.

I will keep this issue open until the full main-branch midgc CI profile passes once.

9 days ago

Completed by the bounded convergence/witness fixes tracked in this workstream. Replacement main run 29466730465 passed the full 20-seed / 900-program threadfuzz-midgc profile in 15m46s and the suppression-free TSan mid-GC profile in 14m16s; the prior termination-reaction miss did not recur, and focused lifecycle semantics remain intact.

Sign in to comment on this issue.