ReviewOS

zig-utils/zig-js

Threads: expand concurrent lifecycle and semantic fuzzing

#13
Closed chris opened this 10 days ago ·
10 days ago

Parent: #1

Goal

Keep expanding generated and deterministic stress coverage for exceptions, termination, cleanup, waiters, and cross-thread lifecycle composition.

Current baseline

The fuzzer has broad, mid-script-GC, lifecycle, amplified, TSan, ReleaseSafe, and deterministic-result profiles. Deterministic subprograms compose shared-realm Threads, script/module Workers, asyncJoin, wait/waitAsync, Lock/Condition APIs, ThreadLocal, buffers, weak collections, FinalizationRegistry, exceptions, teardown, multi-context create/destroy, mixed property/typed-array waiter races, and weak cleanup ordering. The lifecycle and mid-GC oracles now explicitly join/await load-sensitive asyncJoin and async-hold cleanup paths where CI scheduling can delay reaction delivery, preserving exact invariants without relying on fixed spin windows.

Long-running profiles now use a deterministic per-seed watchdog. THREADFUZZ_SEED_TIMEOUT_MS defaults to 120000 ms, can be raised for slow TSan/local machines, and can be disabled with 0; watchdog failures exit 124 and print the exact profile + seed for replay. Aggregate profiles print wall-ms, which is the required runtime evidence before raising CI/nightly depth.

Nightly TSan depth can now be raised manually through workflow-dispatch inputs without editing YAML: nightly_tsan_default_iters, nightly_tsan_midgc_iters, and nightly_tsan_lifecycle_iters. Scheduled defaults remain conservative; deliberate depth experiments should use the manual inputs, capture runtime/flake evidence, then only promote stable values into the scheduled defaults.

Remaining work

  • Add higher-depth nested Thread/Worker/asyncJoin graphs.
  • Expand exception identity and rejection/thenable publication under termination.
  • Expand timeout/notify/abandon races for property and typed-array waiters.
  • Add cleanup ordering combinations across WeakRef/WeakMap/WeakSet/FinalizationRegistry.
  • Add multi-context and repeated create/destroy stress.
  • Preserve deterministic seed reproduction, bounded per-program timeouts, and aggregate wall-ms runtime evidence.
  • Stabilize load-sensitive mixed waiter and worker async-hold cleanup oracles without weakening their exact final-count invariants.
  • Add manual nightly-depth controls for TSan default, mid-GC, and lifecycle profiles.
  • Periodically raise CI/nightly depth only after measuring runtime and flake rate.

Acceptance criteria

  • Every new bug gets a minimized deterministic witness plus a generated profile.
  • TSan and non-TSan profiles agree on semantic oracles.
  • Long sweeps remain reproducible from printed seed/flags.
  • CI/nightly depth increases include measured runtime and flake evidence, not just a larger iteration count.
10 days ago

Landed concurrent multi-context and repeated create/destroy stress in e937698c.

The new threadfuzz multictx <iters> <seed> mode starts 3–4 host OS threads together. Each host repeatedly creates an independent no-GIL Context, alternates exact atomic/join/exception-identity oracles with abrupt top-level failure after a child parks, destroys the Context, and recreates it for 3–5 rounds. It is also the 54th deterministic subprogram in the lifecycle profile.

Validation:

  • zig build threadfuzz-bin
  • ./zig-out/bin/threadfuzz multictx 2 1 — 2/2 passed
  • ./zig-out/bin/threadfuzz multictx 8 100 — 8/8 passed
  • git diff --check

The full lifecycle/TSan sweeps remain batched with subsequent issue #13 work as planned.

10 days ago

Completed the combined higher-depth graph item in 42db3666.

The existing module-Worker fanout profile now runs its seven-module import graph concurrently with shared-realm parent Threads that each spawn 2–3 child Threads. Both child and parent asyncJoin reactions publish exact values into a dedicated retained-SAB slot, while blocking joins, total work, module-only work, and module leaf-call counters remain independent exact oracles. This covers reaction rerouting out of exiting parent-thread queues rather than testing Worker, nested Thread, and asyncJoin graphs only in separate profiles.

Added focused reproduction: threadfuzz modulefanout <iters> <seed>.

Validation:

  • zig build threadfuzz-bin
  • ./zig-out/bin/threadfuzz modulefanout 4 1 — 4/4 passed
  • ./zig-out/bin/threadfuzz modulefanout 6 20 — 6/6 passed
  • git diff --check

Full lifecycle and TSan validation remains batched with the remaining #13 slices.

10 days ago

Completed the termination/publication composition in dcca4b7.

The existing termreact profile now keeps completed thrown-object and user-thenable completions live while sibling Threads with child-owned typed-array waitAsync tickets are forcibly terminated by abrupt top-level failure. It verifies same-object exception identity through blocking join and asyncJoin rejection, raw thenable return through join, two exact thenable assimilations through Promise/asyncJoin, terminated-sibling rejection roots, and zero leaked waiter tickets in the same lifecycle window.

Validation:

  • zig build threadfuzz-bin
  • ./zig-out/bin/threadfuzz termreact 4 1 - 4/4
  • ./zig-out/bin/threadfuzz termreact 8 100 - 8/8
  • git diff --check

Full lifecycle/TSan remains batched with the rest of #13.

10 days ago

Completed the mixed waiter race slice in 2582d5d.

The new waitrace profile composes property and typed-array waitAsync notify, timeout, and top-level-failure abandon paths in one deterministic realm. It caught and fixed a real property-waiter teardown gap: child-owned property waitAsync tickets were transferred to the realm queue on thrown/terminated Thread exit instead of being abandoned like typed-array waitAsync tickets, so later Atomics.notify could still observe dead-owner property tickets.

Fix:

  • added queue-scoped property waitAsync abandon on the thrown/terminated Thread path
  • preserved the normal-completion transfer path used by returned property waitAsync promises
  • added a table-level unit witness and included waitrace in lifecycle coverage/docs

Validation:

  • zig build threadfuzz-bin
  • ./zig-out/bin/threadfuzz waitrace 1 101 — 0 failures
  • ./zig-out/bin/threadfuzz waitrace 8 100 — 0 failures
  • zig build test -Dtest-filter="property waitAsync abandon"
  • git diff --check

Full lifecycle/TSan remains batched with the rest of #13.

10 days ago

Completed the weak cleanup-ordering slice in 312ab69.

The new weakorder profile adds an exact ordering oracle for FinalizationRegistry cleanup delivery across WeakMap values, WeakSet values, direct finalization targets, WeakRefs, and unregister-suppressed records while property and condition waiters are parked. It verifies live WeakMap ephemeron values stay reachable only through live weak keys, dead weak refs clear, unregister records do not deliver, and cleanup delivery preserves the seeded registration order via count, sum, and rolling-hash checks.

Validation:

  • zig build threadfuzz-bin
  • ./zig-out/bin/threadfuzz weakorder 2 1 — 0 failures
  • ./zig-out/bin/threadfuzz weakorder 8 100 — 0 failures
  • git diff --check

Full lifecycle/TSan remains batched with the remaining #13 reproducibility/depth work.

10 days ago

Implemented deterministic per-seed watchdog coverage in 3800613 (test(threadfuzz): add per-seed watchdog (#13)).

What changed:

  • long-running threadfuzz profiles now track the active profile + seed while each generated/replayed program executes;
  • THREADFUZZ_SEED_TIMEOUT_MS defaults to 120000 ms, can be raised for slow TSan/local machines, and can be disabled with 0;
  • watchdog failures exit 124 and print the profile + seed so the hung case remains reproducible with the same profile, one iteration, and that seed;
  • docs now describe the timeout knob and reproduction behavior.

Validation:

  • zig build threadfuzz-bin
  • THREADFUZZ_SEED_TIMEOUT_MS=1000 ./zig-out/bin/threadfuzz verify 1 1 → 0 failures
  • THREADFUZZ_SEED_TIMEOUT_MS=0 ./zig-out/bin/threadfuzz verify 1 1 → 0 failures
  • THREADFUZZ_SEED_TIMEOUT_MS=1000 ./zig-out/bin/threadfuzz vwait 2 1 → 0 failures
  • THREADFUZZ_SEED_TIMEOUT_MS=1 ./zig-out/bin/threadfuzz lifecycle 1 1 → expected watchdog exit 124, printed threadfuzz lifecycle: seed 1 exceeded per-seed timeout 1 ms...
  • git diff --check

Leaving only the runtime/flake measurement item open before raising CI/nightly depths.

10 days ago

Updated the fuzzing/CI evidence in 4bf4e35.

Findings:

  • The TSan midgc profile is 45 subprograms per seed and can exceed the default 120s seed watchdog even for one seed.
  • With a 300s profile-local watchdog, the seed completed locally after fixing the nested asyncJoin cleanup oracle and active HoldJob roots.
  • The normal watchdog remains unchanged; only TSan midgc CI/nightly commands set THREADFUZZ_SEED_TIMEOUT_MS=300000.

Validation:

  • zig build threadfuzz -Dfuzz-midgc=true -Dfuzz-iters=1
  • zig build threadfuzz -Dtsan=true -Dfuzz-lifecycle=true -Dfuzz-iters=1
  • THREADFUZZ_SEED_TIMEOUT_MS=300000 zig build threadfuzz -Dtsan=true -Dfuzz-midgc=true -Dfuzz-iters=1

This is not a depth increase; it is a sanitizer runtime budget correction for an already-gated profile.

10 days ago

Landed another lifecycle-fuzz cleanup uncovered by the tsan-threadfuzz-lifecycle run on aa697df.

Commits:

  • zig-gc f8205c7 (fix(gc): serialize weak scratch globally): weak-slot scratch registration now uses one process-wide lock, so parallel create/destroy cycles do not present allocator address reuse as two separate heap locks protecting the same scratch page under TSan.
  • zig-js a0375cb (fix(threads): pair backing unlocks with acquisitions): object backing-store lock helpers now remember whether the lock was actually acquired before unlocking; this fixes the lifecycle panic where element_locks_enabled changed between the lock guard and deferred unlock during Thread API installation.

Focused validation:

  • zig build test in zig-gc
  • zig build test -Dtest-filter="Object backing lock pairs unlock with actual acquisition"
  • zig build threadfuzz -Dtsan=true -Dfuzz-lifecycle=true -Dfuzz-iters=2 → threadfuzz lifecycle: 112 programs from seed 1, 0 failures

CI rerun for a0375cb: https://github.com/zig-utils/zig-js/actions/runs/29116066646

10 days ago

CI follow-up after a0375cb: pushed 87c4c8b to address the lifecycle crash in tsan-threadfuzz-lifecycle. The failure was an Environment.binding_lock unlock-without-acquire during parallel context creation (Thread/Atomics builtin install raced a process-wide synchronization flag flip).\n\nThe fix enables the shared parallel synchronization protocols before threaded builtins are installed and makes the helper enable-only so concurrent context startup cannot disable the protocols underneath another context.\n\nLocal checks kept intentionally focused:\n- zig build test -Dtest-filter="Object backing lock pairs unlock with actual acquisition"\n- zig build test -Dtest-filter="enable_threads: Object backing allocator stays synchronized under parallel_js"\n\nSeparate from that crash, the same CI run produced reproducible midgc termination-reaction misses (e.g. midgcworkertermcondasync seed 11, midgcworkertermwaitasync seed 19, worker/module term teardown seeds 18/6). I reproduced those locally with the focused non-TSan subcommands and am keeping that investigation separate from 87c4c8b.

10 days ago

Follow-up pushed: 641c5e9 (fix(threads): keep binding locks enabled across contexts).\n\nThe 87c4c8b CI run proved the first fix was still too late: tsan-threadfuzz-lifecycle hit the same unlock-without-acquire, but this time during create-time global setup at Context.createWithTestingOptions -> self.env.get(""), before Thread/Atomics builtin installation. That means the process-wide binding-lock flag itself cannot start false when contexts can be created concurrently; another context can flip it between a lock helper's load and its paired unlock helper.\n\n641c5e9 makes Environment.binding_locks_enabled start true. This trades a small default binding-lock cost for eliminating the false→true pair-mismatch class. A lower-overhead owner-token lock helper can recover perf later, but correctness wins for now.\n\nLocal validation after the change:\n- zig build test -Dtest-filter="Object backing lock pairs unlock with actual acquisition"\n- zig build test -Dtest-filter="enable_threads: Object backing allocator stays synchronized under parallel_js"\n- THREADFUZZ_SEED_TIMEOUT_MS=60000 ./zig-out/bin/threadfuzz lifecycle 2 1 -> 112 programs from seed 1, 0 failures\n\nLocal mac TSan no longer reproduced the mutex assertion, but did exceed the 120s per-seed watchdog, so I’m leaving CI to judge the sanitizer runtime on Linux.

10 days ago

Pushed 4d92f18 for the midgc lifecycle/termination reaction losses found after the lifecycle lock fixes. Focused repros that previously missed asyncJoin rejection counts now pass locally:

  • midgcworkertermteardown 5 18
  • midgcmoduletermteardown 5 6
  • midgcworkertermwaitasync 5 19
  • midgcworkertermcondasync 5 10

Also ran midgc 1 1 successfully (45 programs) plus the targeted Promise reaction-list and Thread asyncJoin pending-list unit tests. CI is running on the new head; the remaining watch item is whether tsan-threadfuzz-midgc stays within its per-seed budget.

10 days ago

CI note for commit 4d92f18 (run https://github.com/zig-utils/zig-js/actions/runs/29120379354): tsan-threadfuzz-lifecycle failed in worker terminate/ThreadLocal asyncHold with release join sum off by +1 on seeds 1 and 2. This did not reproduce locally with the focused selector workertlsasyncholdcleanup 2 1, the full TSan lifecycle command zig build threadfuzz -Dtsan=true -Dfuzz-lifecycle=true -Dfuzz-iters=2, or an amplified focused pass workertlsasyncholdcleanup 20 1. Treating as a real CI flake under #13 until rerun/extra evidence says otherwise; no broad suite rerun locally.

10 days ago

Additional CI signal from run https://github.com/zig-utils/zig-js/actions/runs/29120379354: tsan-threadfuzz-midgc also exposed a real data race in Context.hasRunningJsThreads versus concurrent Thread construction. Fixed by d7c69d6; focused TSan validation passed with midgcrestrictlife 2 1. The separate tsan-threadfuzz-lifecycle release-join +1 failure remains tracked here as non-reproduced/flaky pending the next CI run.

10 days ago

CI/harness note: c03bdf6 adjusts the midgc watchdog after run 29121153787 showed tsan-threadfuzz-midgc timing out at the aggregate seed level with no TSan race. The midgc profile has 45 independent subcases per seed; under TSan, that whole seed can exceed 300s even if no individual subcase hangs. The harness now refreshes the watchdog for each midgc subcase, preserving hang detection while avoiding false aggregate timeouts.

10 days ago

Landed 98dc9e8 (test(threadfuzz): report profile wall time).

What changed:

  • Aggregate threadfuzz profiles now print wall-ms in their summary lines: default, amplify, broad, verify/v* deterministic modes, lifecycle, and midgc.
  • Focused one-off repro selectors keep their concise existing output.
  • docs/threads/testing.md now calls out wall-ms as the runtime evidence source for future CI/nightly depth increases.

Validation:

  • zig build threadfuzz-bin --summary none
  • ./zig-out/bin/threadfuzz verify 1 1 → threadfuzz verify: 1 programs from seed 1, 0 failures, wall-ms 109
  • ./zig-out/bin/threadfuzz 1 1 → threadfuzz default: 1 programs from seed 1, 0 failures, wall-ms 412
  • bun run docs:build
  • git diff --check

This does not raise depth yet; it makes the CI/nightly logs carry the runtime measurement #13 requires before any future depth increase. CI is pending at https://github.com/zig-utils/zig-js/actions/runs/29135914952.

10 days ago

Pushed a lifecycle-fuzzer stability chunk in 194ab8b (test(threadfuzz): park abandoned waiter fuzzers).

Root cause from CI run https://github.com/zig-utils/zig-js/actions/runs/29141581898: the threadfuzz-lifecycle leg failed in the mixed property/typed-array waitAsync race subprogram at seeds 5, 11, 21, 27, and 51. The failing oracle showed normal waiters not settling before the teardown point (bad mixed waiter normal score ...).

What changed:

  • The abandoned waiter threads still register outstanding property/typed-array waitAsync tickets and remain alive for the teardown/leak oracle.
  • After publishing readiness, they now park with Atomics.wait(...) instead of spinning forever.
  • This preserves the abandonment semantics while avoiding CI-runner CPU starvation that could delay normal notified/timeout waiter promise settlement.

Focused validation:

  • zig build threadfuzz-bin --summary none
  • ./zig-out/bin/threadfuzz lifecycle 1 5
  • ./zig-out/bin/threadfuzz lifecycle 1 11
  • ./zig-out/bin/threadfuzz lifecycle 1 21
  • ./zig-out/bin/threadfuzz lifecycle 1 27
  • ./zig-out/bin/threadfuzz lifecycle 1 51
  • zig fmt conformance/threadfuzz.zig
  • git diff --check

Keeping #13 open for broader fuzzing/depth work; this is a targeted stability fix for a newly surfaced CI lifecycle profile failure.

10 days ago

Follow-up to the previous lifecycle-fuzzer stability fix: pushed dd74dbc (test(threadfuzz): avoid waiter-table pollution).

The first parked-waiter fix avoided hot CPU spinning, but the fresh Linux CI log showed the mixed waiter subprogram now failing for essentially every lifecycle seed. Root cause: the abandoned threads were parking with Atomics.wait(gate, 'ready', ...), which added extra property waiters into the same waiter machinery that the oracle is trying to validate.

Correction:

  • Abandoned waiter threads still register outstanding property/typed-array waitAsync tickets and remain alive for teardown/leak checks.
  • After publishing readiness, they now sleep with sleepMs(1000) instead of using Atomics wait on the shared gate.
  • This keeps the abandoned threads alive without polluting the property or typed-array waiter tables under test.

Focused validation:

  • zig fmt conformance/threadfuzz.zig
  • zig build threadfuzz-bin --summary none
  • ./zig-out/bin/threadfuzz lifecycle 1 1
  • ./zig-out/bin/threadfuzz lifecycle 1 5
  • ./zig-out/bin/threadfuzz lifecycle 1 50
  • git diff --check

Keeping #13 open; this is a targeted CI/lifecycle fuzzer oracle hardening chunk.

10 days ago

Pushed another lifecycle fuzzer hardening chunk: f9e5e60 (test(threadfuzz): settle worker async-hold joins). The latest TSan lifecycle failure on run https://github.com/zig-utils/zig-js/actions/runs/29142720376 exposed a separate oracle race in the worker-terminate/ThreadLocal asyncHold cleanup case: release asyncJoin callbacks could lag behind the teardown throw under TSan, and the fuzzer also accepted any JS throw as the intended teardown point. The fix moves release-join count/sum assertions into the post-teardown drain/check path and adds a sentinel so pre-teardown assertion failures are reported directly instead of surfacing later as an aggregate checked-count mismatch.\n\nFocused validation before push:\n- zig fmt conformance/threadfuzz.zig\n- zig build threadfuzz-bin --summary none\n- ./zig-out/bin/threadfuzz lifecycle 1 2\n- zig build threadfuzz-bin -Dtsan=true --summary none\n- ./zig-out/bin/threadfuzz lifecycle 1 2\n- zig build threadfuzz -Dtsan=true -Dfuzz-lifecycle=true -Dfuzz-iters=2\n- git diff --check

10 days ago

Fuzzer harness update landed in db57aa3. The failing mid-GC ThreadLocal asyncHold cleanup case exposed a bad oracle rather than a runtime regression: after the mid-GC variant deliberately releases TLS-owner threads, observing some finalizers before the top-level teardown throw is valid. The harness now records/bounds that early subset and still checks exact final cleanup totals.\n\nFocused local coverage passed for both script and module worker variants across seeds 1-20, plus 5-seed non-mid-GC lifecycle smoke coverage to preserve the zero-early-cleanup side of the invariant.

10 days ago

CI note from scheduled run https://github.com/zig-utils/zig-js/actions/runs/29146536239 on zig-js@a510e76: threadfuzz-lifecycle reported mixed waiter normal-score failures across the Linux scheduled profile, while the same current-head focused checks pass locally.\n\nLocal checks after the GC metadata fix:\n- ./zig-out/bin/threadfuzz waitrace 1 1 -> 1 program, 0 failures\n- ./zig-out/bin/threadfuzz lifecycle 1 1 -> 56 programs, 0 failures, wall-ms about 10521\n\nI did not patch the mixed-waiter oracle blindly from this signal. Treating it as an open CI-only lifecycle signal pending the fresh push run (https://github.com/zig-utils/zig-js/actions/runs/29157321812) or a reproducible focused seed/log that points to either an oracle timing budget issue or a real asyncJoin/waitAsync settlement loss.

10 days ago

CI follow-up for run 29161197554: threadfuzz-lifecycle failed in the mixed waiter race lifecycle subprogram for every seed with bad mixed waiter normal score ... before the intended teardown throw. The runtime path looked healthy; the oracle was relying on a fixed spin budget for asyncJoin reactions from finite normal waiters, which is brittle under CI load.

Pushed 4b05124 (test(threadfuzz): stabilize mixed waiter lifecycle). The fuzzer now keeps handles for the finite normal waiter Threads and, if asyncJoin reactions have not all drained by the first spin budget, joins those normal threads and drains the run loop/microtasks before asserting the pre-teardown score. The abandoned infinite waiters and teardown checks are unchanged.

Focused validation:

  • zig build threadfuzz -Dfuzz-lifecycle=true -Dfuzz-iters=2 --summary all
  • git diff --check

This keeps the lifecycle profile testing asyncJoin delivery without turning CI scheduler delay into a false teardown failure.

10 days ago

CI follow-up for run 29167915772: the required gate failed in tsan-threadfuzz-midgc on the worker terminate/ThreadLocal asyncHold subprogram, seed 1, with bad worker terminate/ThreadLocal asyncHold release join sum 2910008.\n\nPushed 14a1634 (test(threadfuzz): await worker async-hold joins). Root cause was another load-sensitive fuzzer oracle window: the subprogram stored release-thread join promises only as spin-drained .then callbacks, so under TSan/CI the post-teardown check could observe a skewed release-join sum. The harness now stores the original release join promises and, after the intended teardown throw, awaits Promise.all(...) from Zig with machine.awaitValue before asserting the exact release-marker sum/count. This preserves the invariant instead of weakening it.\n\nFocused validation:\n- zig fmt conformance/threadfuzz.zig\n- zig build threadfuzz-bin --summary none\n- ./zig-out/bin/threadfuzz midgcworkertlsasyncholdcleanup 3 1\n- zig build threadfuzz-bin -Dtsan=true --summary none\n- ./zig-out/bin/threadfuzz midgcworkertlsasyncholdcleanup 3 1\n- ./zig-out/bin/threadfuzz midgcmoduleworkertlsasyncholdcleanup 1 1 (with the TSan-built binary)\n- git diff --check\n\nA fresh full CI gate is now queued/running for 14a1634.

10 days ago

Follow-up from the async-hold worker cleanup fix and CI-sharding work: fresh full CI is green for b2e7dbe5e9bc2a27a87ff1bad8be8b0be777e46e:

https://github.com/zig-utils/zig-js/actions/runs/29170980761

Relevant fuzzer legs that passed in that run include threadfuzz-lifecycle, threadfuzz-midgc, tsan-threadfuzz-lifecycle, and tsan-threadfuzz-midgc. Keeping this open for the remaining planned depth/nightly expansion work.

10 days ago

Scheduled/nightly fuzzing paid off here: https://github.com/zig-utils/zig-js/actions/runs/29186685965 caught one nightly-threadfuzz-tsan (midgc-5) failure while the rest of the scheduled matrix passed (33 jobs green).

The failure was not a runtime data-race finding; it exposed an over-strict mid-GC witness around Atomics.Mutex.lockIfAvailable timeout cleanup. Landed e00d5858 to split acquire-path cleanup from timeout-path cleanup and updated the docs with the corrected invariant.

Local reproduction/validation used the non-TSan 5-seed mid-GC window once to avoid wasting cycles while still covering the failed seeds: /Users/chris/.local/bin/zig build threadfuzz -Dfuzz-midgc=true -Dfuzz-iters=5 -> 225 programs from seed 1, 0 failures.

New CI pending: https://github.com/zig-utils/zig-js/actions/runs/29193634788

10 days ago

Follow-up: the push run after the scheduled-fuzz finding completed green.

Run: https://github.com/zig-utils/zig-js/actions/runs/29193634788 Result: 31 success, 1 expected skip, 0 failures.

This confirms the corrected Atomics.Mutex.lockIfAvailable mid-GC witness in normal push CI; the nightly-only nightly-threadfuzz-tsan job is expected to remain schedule-gated.

10 days ago

Pushed 36c0159a (ci(threadfuzz): parameterize nightly depth) as a #13 support chunk.

What changed:

  • Added workflow_dispatch inputs for the nightly TSan fuzz iteration counts: nightly_tsan_default_iters, nightly_tsan_midgc_iters, and nightly_tsan_lifecycle_iters.
  • Converted the nightly-threadfuzz-tsan matrix to keep the same scheduled defaults (120, 5, 5) while allowing manual dispatch runs to override each profile independently.
  • Documented the manual inputs in docs/threads/testing.md as the intended way to collect wall-ms and flake evidence before increasing scheduled/nightly depth.

I deliberately did not raise the scheduled defaults in this chunk: the latest scheduled run still had one nightly-threadfuzz-tsan (midgc-5) failure before the follow-up oracle fix, so #13's acceptance rule says to gather a clean manual/scheduled evidence run first.

Focused validation:

  • workflow-token check for the new dispatch inputs/env wiring
  • bun run docs:build
  • git diff --check

CI for current head: https://github.com/zig-utils/zig-js/actions/runs/29201371564 is pending.

10 days ago

Routing note: the CI run linked for 36c0159a was cancelled by the newer #16 nursery-profile push. The workflow-dispatch depth parameterization remains on main; use current combined-head CI for b79f9774: https://github.com/zig-utils/zig-js/actions/runs/29201561685 (queued at the time of this note).

10 days ago

Routing note: the current combined-head CI moved again with the #12 memory-model docs push. The #13 workflow-dispatch depth parameterization remains on main; use current combined-head CI for d0092f5e: https://github.com/zig-utils/zig-js/actions/runs/29201642565 (queued at the time of this note).

10 days ago

Green gate note for d0092f5e: https://github.com/zig-utils/zig-js/actions/runs/29201642565

The manual nightly-depth workflow-dispatch controls and lifecycle/mid-GC fuzzing maintenance are now covered by the successful combined gate. No local rerun was needed beyond the CI batch.

10 days ago

Green gate note for c6856a28: https://github.com/zig-utils/zig-js/actions/runs/29203285907

Recorded the CI watchdog calibration in docs/threads/testing.md: mid-GC local/CI commands now show THREADFUZZ_SEED_TIMEOUT_MS=300000, matching the workflow. This keeps seed timeouts useful as hang detectors while avoiding false failures from loaded hosted runners.

10 days ago

Green gate note for c92bbc01: https://github.com/zig-utils/zig-js/actions/runs/29204523233

The agent broadcast/waiter lifecycle surface now has an explicit wraparound invariant test (agent broadcast generation never publishes idle zero and ack checks survive wrap) alongside the existing real broadcast rendezvous and waitAsync cross-agent tests. This keeps the lifecycle/fuzzing tracker’s agent coordination assumptions executable without adding another heavy gate.

10 days ago

Progress update: pushed 5a9217d44e85b5e247e947d1f50f74d8b52e41b2 (fix(agent): keep waitAsync ticket ids live-unique).

This adds a focused regression case for Atomics.waitAsync ticket generation at counter wrap: zero is skipped, and IDs that are still live are not reused. It gives the lifecycle/fuzzing track a concrete edge case to preserve as broader concurrent coverage grows.

Validation:

10 days ago

New minimized runtime-race witness from #41: four OS threads repeatedly claim/release one async activation ownership word and perform exact unsynchronized-counter increments inside the claim. It is green in normal and focused TSan modes; the original api/condition-async-wait.js corpus case also passed in the full replacement gate. This preserves the rule that every newly found race gets a deterministic focused witness.

10 days ago

CI exposed one remaining machine-speed-dependent lifecycle oracle in run https://github.com/zig-utils/zig-js/actions/runs/29231655050: tsan-threadfuzz-midgc, seed 1, exhausted a fixed 100,000-iteration pump window after releasing property/Condition waiters, before their asyncJoin rejection tasks arrived. There was no TSan race report and the failure was outside Worker channel delivery.\n\nPushed bdb369e7 (test: join released midgc waiters). The Worker/ThreadLocal/asyncHold mid-GC oracle now joins the released property, Condition, and TLS owner threads before pumping and checking the exact rejection counts, so it verifies settlement rather than runner speed while preserving the same semantic oracle.\n\nFocused TSan validation passed for the exact failing seed in both script-Worker and module-Worker variants:\n- threadfuzz midgcworkertlsasyncholdcleanup 1 1\n- threadfuzz midgcmoduleworkertlsasyncholdcleanup 1 1\n\nReplacement full gate: https://github.com/zig-utils/zig-js/actions/runs/29232877388

10 days ago

The replacement gate for bdb369e7 is fully green: https://github.com/zig-utils/zig-js/actions/runs/29232877388 — 31 success, 1 expected skip, 0 failures. The previously failing 90-program tsan-threadfuzz-midgc job passed with the released-thread joins, alongside lifecycle, mid-GC, no-GIL corpus, ReleaseSafe, and deterministic fuzz legs.

10 days ago

Closing this tracker as completed. The generated and deterministic profiles now cover the listed lifecycle, termination, waiter, cleanup, Worker, multi-context, weak/finalization, and mid-script-GC compositions; per-seed watchdogs and printed replay flags keep failures reproducible; TSan and non-TSan semantic oracles agree; and the final fixed-window waiter seam now joins released threads before asserting exact asyncJoin settlement.\n\nThe remaining unchecked body item—periodically adjusting CI/nightly depth after runtime and flake measurement—is ongoing test-infrastructure maintenance, not missing engine behavior. Future depth changes should carry their own runtime evidence, and every new bug should still get a narrow issue plus minimized deterministic witness.\n\nAcceptance gate: https://github.com/zig-utils/zig-js/actions/runs/29232877388

Sign in to comment on this issue.