ReviewOS

zig-utils/zig-js

Bench: make zig-js and JSC comparison symmetric and statistically robust

#45
Closed chris opened this 9 days ago ·
9 days ago

Parent: #44

Completed fairness protocol

The comparison now has three explicit directly comparable boundaries plus a separate shared-realm capability panel:

  • direct single warmed context;
  • independent warmed contexts on persistent creator threads, using identical semaphore dispatch/evaluation/completion timing;
  • independent cold lifecycles, timing thread/context/source setup through context destruction and join;
  • zig-js shared-realm JavaScript Threads, reported separately because the public JSC embedding API has no shared-object-graph equivalent.

Every parallel mode includes its own one-lane baseline, so scaling never mixes incompatible lifecycle costs.

Acceptance criteria

  • Directly compared single-context rows execute identical workload and invocation bytes with equal work and checksum validation.
  • Add zig-js independent warmed contexts on OS threads, matching JSC independent warmed contexts.
  • Report shared-realm zig-js as a separate capability panel, not a direct JSC ratio.
  • Separate cold thread/context lifecycle from steady-state worker throughput.
  • Calibrate equal job counts so the fastest engine has a sufficiently long timed interval; full rows enforce a 50 ms median floor.
  • Alternate runner order deterministically to reduce thermal/order bias.
  • Report median, min/max, and relative standard deviation from retained raw samples.
  • Define steady-state and GC-state policy for both engines.
  • Record power state when available in addition to compiler/runtime/build/commit metadata.
  • Add standalone harness self-tests for missing groups, duplicate rows, checksum mismatch, too-short samples, and dirty-result publication.
  • Re-run one full matrix only after the revised harness passes quick validation.

Evidence

  • Initial fairness protocol: 82207c2
  • Clean-publication guard: e70eeff
  • Symmetric steady/cold runners, mode-local baselines, power metadata, and harness tests: c9f68e3
  • Quick 88-group integration matrix passed locally on 2026-07-13.
  • Full 616-sample report, raw TSV, stable docs, and README: 2ebda1a
  • Source commit measured by the full report: c9f68e3
9 days ago

First fairness slice landed across 82207c2, e70eeff, and 6fb3b28. The revised full matrix validates 224 samples, enforces a 50 ms floor, alternates runner order, publishes min/max and RSD, and removes the invalid shared-vs-isolated parallel ratio. #45 remains open for symmetric zig-js independent contexts, steady-state parallel timing, stronger GC/power policy, and standalone harness tests.

9 days ago

The first fairness slice passed the final combined gate (31 success, 0 failure): https://github.com/zig-utils/zig-js/actions/runs/29280193748. Remaining unchecked criteria stay open.

9 days ago

Completed by c9f68e3 and documented with the accepted 616-sample matrix in 2ebda1a. The remaining performance gaps continue under #44 and workload issues #46-#50.

Sign in to comment on this issue.