Perf: add benchmark regression gates and historical evidence

Parent: #44 Related: #45, #53, #62, #97

Goal

Keep performance work measurable and prevent accepted wins from silently regressing without turning ordinary CI into a noisy hardware contest.

Landed controls

• Versioned raw TSV samples and rendered reports retain exact date, host/CPU, OS, Zig, zig-js/zig-gc/zig-regex revisions, system JavaScriptCore version, and power metadata.
• The comparison driver validates the complete symmetric matrix, exact sample indexes/counts, stable checksums, cross-engine equal work, runner alternation, a 50 ms full-run timing floor, and per-row min/max/RSD.
• Publishing raw/report evidence refuses a dirty tracked worktree and captures metadata before generated outputs modify the tree.
• zig build benchmark-comparison-test exercises matrix and publication guards in ordinary CI without compiling or running benchmarks.
• The manual-only Performance workflow, added in c7e52fb1, runs the full macOS/JSC matrix with configurable sample/lane counts and retains its artifacts for 90 days. It never runs on pushes or pull requests and never gates ordinary CI.
• 0ff34466 adds the controlled history/publication tool. It reruns the full raw-matrix validation, reproduces the paired Markdown report byte-for-byte, and then generates the marker-delimited README scorecard.
• History comparison requires exact host, OS, Zig, zig-gc, zig-regex, JavaScriptCore, matrix, jobs, and sample-count matches. Volatile battery details are normalized while power source and charging state remain part of the identity.
• Every engine row is retained with baseline/current medians, delta, and both RSDs. A zig-js row gates only when median wall time worsens by more than 10% and both RSDs are at most 5%; JSC rows remain visible environmental controls. The exact baseline/current report paths and zig-js revisions are recorded in the artifact.
• The manual workflow always generates a README scorecard artifact and optionally accepts a committed controlled baseline pair to produce the per-row history artifact or reject an incompatible run. Failed validation still uploads available evidence.

Current accepted evidence

• measured source: 12aa217c2f66c0fe7fa95e38ec414ea5ec15a993
• report: https://github.com/zig-utils/zig-js/blob/main/docs/.data/benchmark-comparison-2026-07-15-128-byte-slab.md
• raw 1,540 samples: https://github.com/zig-utils/zig-js/blob/main/docs/.data/benchmark-comparison-2026-07-15-128-byte-slab.tsv
• 10/10 direct wins (2.67x geometric-mean zig-js throughput)
• 10/10 eight-lane warmed-independent wins (2.95x; 5.14x zig-js scaling vs 4.77x JSC)
• 10/10 eight-lane cold-lifecycle wins (2.91x; 5.18x vs 4.76x)
• 3.97x shared-realm geometric-mean eight-lane scaling; #97 owns the remaining object-churn deficit

The generated calculation corrected the previously hand-copied 2.63x / 2.94x / 2.94x aggregate summary to the raw-derived 2.67x / 2.95x / 2.91x, demonstrating the drift this issue was intended to eliminate.

Acceptance criteria

• Store versioned raw samples and rendered reports with exact commit/environment/power metadata.
• Add deterministic standalone harness/unit checks to ordinary CI.
• Add a manually triggered full performance workflow that does not gate ordinary CI.
• Compare only like-for-like host/runtime configurations and retain explicit historical baselines.
• Define a minimum timing floor, expose repeated-sample dispersion, and apply a conservative noise-aware policy only to controlled runs.
• Fail publication when the tracked worktree is dirty and capture metadata before generated outputs change the tree.
• Link optimization commits and resulting benchmark deltas from their workload issues.
• Keep README headline numbers generated from the latest accepted report rather than manually copied.

Progress in e70eeff and 6fb3b28: publication now refuses a dirty tracked worktree, captures metadata before writing tracked outputs, and retains the exact source commit plus 224 raw samples. #50 remains open for CI harness tests, controlled performance runners/history, derived regression thresholds, and generated README tables.

The publication guard and dated evidence passed the final combined gate (31 success, 0 failure): https://github.com/zig-utils/zig-js/actions/runs/29280193748. Remaining CI/history/generation work stays open.

Latest accepted evidence is now published:

• measured source: 42d0ec26
• 616-sample TSV + generated report: ccee8842
• README + benchmark guide: 09d543c3

The clean full matrix passed every checksum and 50 ms floor guard. zig-js wins all 36 symmetric single/steady/cold rows. README headline values are still manually summarized, so the generated-headline/workflow acceptance items remain open.

Final validation for the published comparison is green on main at 889611de:

• CI run: https://github.com/zig-utils/zig-js/actions/runs/29305518043
• 32/32 runnable jobs passed; the single nightly-only entry was correctly skipped
• Includes the docs build, benchmark comparison harness, all unit shards, thread/reference/no-GIL checks, thread-fuzz profiles, TSan unit coverage, and all four TSan no-GIL corpus shards
• The two follow-up CI fixes are included: isolate property quickening mode in its unit test (e8f98e49) and honor the committed Bun lockfile for docs (889611de)

The measured report (ccee8842) and README summary (09d543c3) are therefore validated end-to-end. I am leaving #50 open only for its remaining automation work: generate/check the README summary from the committed report so future benchmark updates cannot drift.

Pushed 0e22e4bb (bench: align comparison allocators (#50)). Every measured zig-js context now uses the same process-wide libc allocator, matching JSC's process-scoped cached allocator model instead of mixing the runner allocator into direct/shared rows. The generator and benchmark guide document the current method while preserving the July 13 report as immutable historical evidence.

Validation:

• ReleaseFast comparison runners built successfully
• benchmark harness contract: 7/7 passed
• exact array checksum stayed 23508000000 at one lane and 188190000000 at eight shared lanes
• 7-sample medians were neutral within noise: direct arrays ~61.10 -> ~61.30 ms; shared one lane ~63.15 -> ~63.49 ms; shared eight lanes ~157.59 -> ~156.23 ms

This is a fairness/alignment change, not a claimed speedup. The next accepted full matrix will measure the unified allocator rather than rewriting the saved report.

Historical evidence update: fdb32d8 publishes the clean db131e0 ten-workload run with a rendered report, exact environment/power metadata, README summaries, and all 1,540 raw samples. The expanded matrix includes polymorphic properties and bounded object churn; README now shows direct, eight-context, and shared-realm tables and explicitly reports the remaining allocation gap.

Fresh full publication from clean measured commit c98a36c4, published by e4b72407:

• report: https://github.com/zig-utils/zig-js/blob/main/docs/.data/benchmark-comparison-2026-07-14.md
• all 1,540 raw samples: https://github.com/zig-utils/zig-js/blob/main/docs/.data/benchmark-comparison-2026-07-14.tsv
• direct: zig-js wins 9/10 rows; 2.56x geometric-mean throughput lead; object churn 197.469 ms vs JSC 119.400 ms (0.60x);
• 8 warmed independent contexts: 2.63x geometric-mean lead, 5.16x zig-js scaling vs 5.10x JSC; object churn 714.208 ms vs 169.960 ms, improved from 844.149 ms;
• 8 cold lifecycles: 2.63x geometric-mean lead, 5.13x zig-js scaling vs 5.01x JSC; object churn improved from 641.918 ms to 570.335 ms;
• shared-realm geometric-mean scaling: 3.89x; object churn improved from 12,368.464 ms to 11,089.300 ms (10.3%), but its faster one-lane baseline leaves scaling at 0.29x.

The exact-parent A/B for #101 measured 12.2% faster shared8 and 3.4% faster direct with exact checksums. #101 is complete; object churn remains the sole direct JSC loss and shared lanes above 1x remains unmet, so the parent performance issues stay open.

Docs CI follow-up landed in 5100a966 (ci(docs): lock parser-valid highlighter (#50)). Two consecutive main runs failed only the docs job because .gitignore excluded bun.lock, so bun install --frozen-lockfile on clean runners had no lock to freeze and resolved malformed ts-syntax-highlighter bundles. Both published 0.2.1 and 0.2.4 fail Bun 1.3.14 parsing with undeclared/exported bindings; 0.2.0 is parser-valid and API-compatible with the current 24-page build.

The fix tracks the text bun.lock and records an exact 0.2.0 override. Clean local reproduction passed: remove node_modules, bun install --frozen-lockfile, parse/bundle the highlighter, and bun run docs:build (24 pages). This makes the existing frozen-lock CI step real rather than nominal. The new main run will be monitored before this slice is handed off.

CI follow-up: the docs job now passes on the pushed lockfile/override fix in commit 5100a966 (run: https://github.com/zig-utils/zig-js/actions/runs/29449252934). This confirms the clean bun install --frozen-lockfile path and the full 24-page docs build on GitHub Actions. The remainder of the runtime/TSan matrix is still in progress and will be monitored separately.

Completed in 0ff34466. The focused publication gate passed 17/17 checks; the accepted 1,540-row report reproduces exactly from its TSV; README regeneration is idempotent; and an older run with different zig-gc/zig-regex revisions is rejected as non-comparable. The generated README scorecard is now on main, and the manual workflow retains generated scorecard/history evidence without making hosted timing an ordinary CI gate.