ReviewOS

zig-utils/zig-js

Bench: restore full-matrix timing-floor margin

#99
Closed chris opened this 9 days ago ·
9 days ago

Parent: #50 Related: #44, #49

The full July 14 publication run at consumer commit 84d2f9f4 completed the entire ten-workload matrix, then correctly rejected the result because zig-js shared/direct_calls/1 lane/500 jobs dipped below the 50 ms median floor. The same row is close enough to the threshold to pass or fail with normal host variation, so the current calibration no longer provides a durable noise margin.

Fresh focused evidence on the same built runner:

  • 500 jobs median: 63.74 ms (but the full run observed <50 ms)
  • 600 jobs median: 76.06 ms
  • checksums remain deterministic for both calibrations

Acceptance:

  • Raise only the direct_calls equal-work job count enough to preserve a useful margin above the 50 ms floor on the reference M3 Pro.
  • Keep identical work counts for zig-js and JSC in every comparable mode.
  • Pass the quick harness validation.
  • Publish one complete validated report and refresh README/benchmark documentation from its exact scores.
  • Preserve the 1,540-sample matrix and do not weaken the timing-floor validator.
9 days ago

The first calibration exposed another marginal row on the next full run: zig-js/single/arguments_calls/400 jobs also dipped below 50 ms. Auditing the last valid publication shows several workload minima with too little margin: arithmetic 55.38 ms, properties 58.00 ms, direct_calls 64.21 ms, arguments_calls 60.03 ms, arrays 66.52 ms, fibonacci 67.46 ms, and closure_calls 71.08 ms.

I am broadening this issue from the first observed row to one conservative full-matrix recalibration. The 50 ms validator and equal-work engine comparison remain unchanged.

9 days ago

Completed in ce652645 and 3d790a60; published by 3cc2a58c.

  • Audited every workload minimum after two legitimate full-run floor rejections rather than weakening validation.
  • Recalibrated equal work counts for both engines and every comparable mode.
  • Focused former-risk medians were 77–91 ms.
  • 7/7 harness tests and the complete quick matrix pass.
  • Final full matrix passes with 1,540 samples, 220 rows/groups, exact checksums, and a 58.556 ms minimum median.
  • README and benchmark guide document the new counts and floor rationale.

Validated report: https://github.com/zig-utils/zig-js/blob/main/docs/.data/benchmark-comparison-2026-07-14.md

Sign in to comment on this issue.