ReviewOS

zig-utils/zig-js

CI: eliminate mid-GC seed-1 teardown stall on current Zig

#418
Closed chris opened this 9 days ago ·
9 days ago

Roadmap: #134 GC parent: #144 Observed after: #413 / zig-gc#40 Exact runs: 29791895246, 29792663816

Two consecutive current-toolchain main runs hang the non-TSan threadfuzz-midgc profile in seed 1 for more than the unchanged 300-second per-case watchdog. Both timeouts occur after the active Context is disarmed, and teardown then fails its 30-second grace period. The exact seed passes all 45 cases locally in 31.7 seconds under Zig 0.17.0-dev.956, while CI uses 0.17.0-dev.1426.

Acceptance:

  • Record the active mid-GC subcase name in watchdog timeout diagnostics without weakening or extending either timeout.
  • Reproduce with the exact CI Zig revision, seed, dependency snapshot, and non-TSan optimization mode.
  • Identify whether the stall is engine teardown, allocator/compiler behavior, or a harness lifecycle race; retain an independently runnable focused command.
  • Fix the underlying wait/teardown ownership so timeout cancellation is not required for ordinary completion.
  • Preserve all 45 semantic, collection, finalization, and cleanup oracles.
  • Pass the focused subcase repeatedly, seed 1 as a whole, and the 20-seed / 900-program exact main job.
  • Keep the watchdog and clean-teardown failure modes intact and reconcile exact main CI before closure.
9 days ago

Diagnostic checkpoint pushed in f45216f. The watchdog now reports the independently timed mid-GC subcase while preserving the 300 s timeout and 30 s teardown grace. Exact Zig 0.17.0-dev.1426+58a94ea was downloaded and verified locally; seed 1 completed all 45 cases with zero failures in 33,597 ms. Because both GitHub runs still stalled at seed 1, the remaining bug is scheduling/load-sensitive rather than a compiler-version-only deterministic regression. The new main run will identify the precise case if it recurs.

9 days ago

Additional bounded load check: two simultaneous seed-1 replays under exact Zig 0.17.0-dev.1426+58a94ea both completed all 45 cases with zero failures in 36,450 ms and 36,312 ms. This rules out simple local CPU contention and makes the two dev.1426 Linux-runner stalls more specific to runner/platform scheduling or teardown. Replacement run 29795137428 is currently executing the diagnostic 20-seed job under the newly selected dev.1441 toolchain.

9 days ago

Exact main CI has now isolated the stall:

The named-case diagnostic in f45216fd therefore succeeded: this is specifically the late property Atomics.waitAsync / child-thread settlement path, not generic seed teardown. The existing focused command is threadfuzz midgcpropwaitasynclate <iters> <seed>; next work is bounded focused stress and lock/lifecycle diagnosis.

9 days ago

Root fix pushed to main in 707f2fa5 after the named CI failure:

  • marks the tracer-visible property-waiter table lock as trace-sensitive
  • suppresses mid-script collection while any tracer-visible native critical section is held
  • fully constructs the Atomics.waitAsync result before publishing its ticket, eliminating allocator/collector re-entry under the waiter lock and making publication failure-atomic
  • marks the late-settlement ThreadRecord.join_mutex handoff trace-sensitive until its reaction enqueue is published

Focused evidence: waiter-lock invariant 3/3, Atomics.waitAsync 4/4, #420 lifecycle 3/3, all zero leaks; post-fix midgcpropwaitasynclate passes 200/200 seeds on Zig 0.17.0-dev.1426+58a94eaae. The fresh exact Linux main run is the remaining closure gate.

9 days ago

Exact main run 29797102425 (c3cfb00) showed the ordinary threadfuzz job green after 707f2fa. The dedicated threadfuzz-midgc job was still running when the #421 follow-up push superseded the run, so this is useful positive evidence but not yet the required clean exact mid-GC gate. The two completed red TSan jobs were diagnosed as #421's new fail-closed realm assertion exposing missing Worker activation boundaries; 138d203 fixes those and passed their exact local reproductions. #418 remains open pending the fresh unsuperseded threadfuzz-midgc result.

9 days ago

The required unsuperseded exact gate has now completed its full breadth run successfully: run 29801586207, threadfuzz-midgc reports 900 programs from seed 1, 0 failures in 1,527,560 ms. The paired TSan mid-GC job also passed 90/90 programs with no sanitizer report. The workflow still has three unrelated long stress jobs in progress, so I am retaining the final exact-main checkbox/closure until the run itself reaches a green terminal state.

9 days ago

Complete. 707f2fa5 removes the waitAsync/collector lock inversion and preserves all watchdog/teardown oracles. Exact main run 29801586207 is fully green: mid-GC 900/900, TSan mid-GC 90/90, ordinary threadfuzz 400/400, and lifecycle 3,360/3,360 programs, all with zero failures.

Sign in to comment on this issue.