ReviewOS

zig-utils/zig-js

CI: identify and rebalance slow threads-gil shard 1 cases

#33
Closed chris opened this 9 days ago ·
9 days ago

Follow-up from #32.

Evidence

The full push gate for b2e7dbe5e9bc2a27a87ff1bad8be8b0be777e46e completed green:

https://github.com/zig-utils/zig-js/actions/runs/29170980761

The unit/TSan-unit timeout was fixed by sharding and bounding the heap-limit fixture, but threads-gil (shard 1/4) remained the final long pole. It started at 2026-07-11T22:43:56Z and completed near the end of the run, after all other jobs were green.

Goal

Make the serialized/GIL threads corpus shard distribution less lopsided and easier to diagnose when one shard is slow.

Sketch

  • Add per-case progress output or a case timing summary to conformance/threads_test.zig when running shard mode.
  • Use timing evidence to rebalance the shard partition, or move known heavy cases to a separate named leg.
  • Keep the local zig build threads-test workflow convenient.

Acceptance criteria

  • A slow serialized corpus shard reports the active/slow case(s), not only the shard number.
  • The main CI gate does not routinely wait ~50+ minutes on threads-gil (shard 1/4) after all other legs have cleared.
  • Any rebalancing preserves the existing PR-249 allowlist semantics.
9 days ago

Progress in 27bd8150ec0589525e83c112d399226ea9075f1f:

  • threads-test shard mode now logs RUN <n>/<selected> <path> before each selected case.
  • PASS/SKIP/FAIL/MISS/context-failure lines now include elapsed milliseconds.
  • Shard mode prints total shard elapsed time and a top-8 slowest-case summary at the end.
  • The same timing is available for one mode case runs, while the extra summary is limited to explicit shard mode.

Focused validation:

  • zig fmt conformance/threads_test.zig
  • zig build threads-test-bin --summary none
  • ./zig-out/bin/threads-test shard 0 300
  • zig build threads-test-bin -Dtsan=true --summary none
  • git diff --check

This does not close the issue yet: it gives the next CI run the case-level evidence needed to rebalance threads-gil (shard 1/4) without guessing.

9 days ago

More progress:

Timing evidence from 27bd8150d271d5b21ac168babcf0db618e91e281

Run: https://github.com/zig-utils/zig-js/actions/runs/29172632725

Serialized/GIL shard summaries:

  • shard 0: 416,311 ms; slowest cve/mc-val-llint-cache-storm.js at 190,214 ms
  • shard 1: 3,166,165 ms; slowest dw1-sort-comparator-callsite-shapes.js at 2,543,010 ms
  • shard 2: 286,730 ms; slowest cve/mc-tear-typedarray-detach-grow-shrink.js at 169,210 ms
  • shard 3: 595,693 ms; slowest jit/spawned-thread-butterfly-stress.js at 137,946 ms

So shard 1 was both an unlucky pile-up and dominated by one pathological case.

Fixes pushed

  • 3eb7d389 fixes the unrelated tsan-threadfuzz-midgc failure from that timing run by giving the worker terminate/ThreadLocal/asyncHold mid-GC oracle TSan-aware settle budgets. Focused validation:
    • zig build threads-test-bin threadfuzz-bin --summary none
    • zig build threadfuzz-bin -Dtsan=true --summary none
    • ./zig-out/bin/threadfuzz midgcworkertlsasyncholdcleanup 1 2
    • ./zig-out/bin/threadfuzz midgcmoduleworkertlsasyncholdcleanup 1 2
  • 515fcb91 replaces plain modulo sharding for default serialized threads-test shard runs with a deterministic weighted plan based on the observed per-case timings. The new 4-way estimate is:
    • shard 0: 2,543,010 ms, 1 case (dw1-sort-comparator-callsite-shapes.js isolated)
    • shard 1: 642,848 ms, 76 cases
    • shard 2: 642,928 ms, 76 cases
    • shard 3: 642,230 ms, 76 cases

Additional validation:

  • zig fmt conformance/threadfuzz.zig conformance/threads_test.zig
  • zig build threads-test-bin threadfuzz-bin --summary none
  • ./zig-out/bin/threads-test one smoke.js
  • ./zig-out/bin/threads-test shard 299 300
  • git diff --check

New CI run for the pushed fix: https://github.com/zig-utils/zig-js/actions/runs/29174149607

I also opened #34 for the underlying single-case performance problem (dw1-sort-comparator-callsite-shapes.js taking ~42.4 minutes). This issue should stay open until the new CI run proves the rebalance removes the ~55-minute shard pile-up.

9 days ago

Verified by the follow-up push run:

https://github.com/zig-utils/zig-js/actions/runs/29174149607

Head: 515fcb918f7cbd0ba01af14f54f0f1b0dd55d750 Result: green.

Weighted serialized/GIL shard results:

  • shard 0: selected 1/229, estimate 2,543,010 ms, actual 2,557,570 ms; isolated dw1-sort-comparator-callsite-shapes.js
  • shard 1: selected 76/229, estimate 642,848 ms, actual 654,366 ms
  • shard 2: selected 76/229, estimate 642,928 ms, actual 456,054 ms
  • shard 3: selected 76/229, estimate 642,230 ms, actual 728,433 ms

This closes the original problem: the serialized corpus no longer hides progress, and shard 1 is no longer a ~55-minute accidental pile-up after all other legs finish. The remaining long pole is the single isolated dw1-sort-comparator-callsite-shapes.js case, tracked separately in #34.

Sign in to comment on this issue.