Bench: validate independent suites and additional engines out of tree

Parent: #459 Depends on: #460, #462 Related: #461, #503

Goal

Validate zig-js performance against independent, pinned, out-of-tree benchmark suites and additional engines without making any suite or engine a build/runtime dependency.

Why

#460 defines rigorous repository-owned application-shaped workloads and anti-specialization variants. Broad leadership claims also need independent workloads whose structure was not designed around zig-js, plus controls showing whether a result is specific to system JSC.

Scope

• Inventory candidate language/runtime suites by license, host API requirements, determinism, workload coverage, run duration, maintenance, and compatibility with a non-browser embedder.
• Select pinned script/Wasm subsets that can run unchanged or through a minimal documented host adapter; classify excluded browser/Node APIs rather than silently patching semantics.
• Run suites from explicit temporary or sibling checkouts with exact revision/checksum. Do not vendor them, resolve them during ordinary builds, or make them package dependencies.
• Add optional comparison adapters for system JSC and independently installed V8, SpiderMonkey, and QuickJS where exact programming models/timed boundaries are possible.
• Preserve official scores where valid, plus per-test wall/CPU/RSS/tier/GC data and exact failures/skips.
• Keep engines in separate executables/processes so zig-js's JSC-shaped exports cannot interpose.
• Publish suite-specific results and geometric aggregates only after every included row, exclusion, patch, and host difference is visible.

No-workaround rules

• No editing workload logic to favor zig-js, unsupported test deletion, dependency bundled into zig-js, browser score claimed by a non-browser subset, fake public-API equivalence, or “win” based only on an aggregate hiding losses.

Acceptance

• A machine-readable inventory records suite/engine pins, licenses, checksums, adapters, applicable rows, exclusions, and reasons.
• Acquisition and execution are explicit, offline-after-fetch, checksum-verified, and quarantined from normal build/runtime dependencies by #462.
• Results retain every applicable row, failure, skip, raw sample, dispersion, checksum/output validation, and exact timed boundary.
• Anti-specialization mutations and tier attribution show no suite/source recognizer.
• Broad performance wording distinguishes repository-owned, independent-suite, JSC, and multi-engine evidence.

Foundation

The symmetric cross-engine process/timing protocol is established by #44/#45/#50 and tools/benchmark-comparison.py; its owned replacement is tracked by #497.

The first independent-suite/engine inventory slice landed on main in f3767f87.

zig-js-independent-suite-inventory-v1 now freezes:

• exact repository commit, tree, and SHA-256 file pins for Octane 2 and the audited JetStream 3 alpha metadata/CLI boundary;
• maintenance, determinism, workload coverage, unmeasured-duration, host-API, top-level/per-row license, adapter, and claim boundaries;
• all 17 upstream Octane result names, with six clearly licensed minimal-shell results marked as candidates and every other result retained as excluded with a row-specific license, provenance, host, or output-validation reason;
• JetStream 3 alpha as inventory-only and excluded: its exact subtests, mixed per-workload licenses, npm preparation, compressed assets, and shell boundary remain unresolved rather than being silently patched or partially scored;
• per-run executable hash/version/revision/argv/environment pin contracts for zig-js, system JSC, V8, SpiderMonkey, and QuickJS;
• a required evidence shape that retains every row, failure, skip, raw sample, dispersion value, output validation, and timed boundary.

The new zig build independent-suite-audit is a dependency-free offline Zig gate with malformed-pin and missing-exclusion tests, and it now runs inside benchmark-comparison-test in CI. dependency-audit confirms the change introduced no network-capable repository tool, ordinary-build fetch, package, link, or runtime dependency.

This is candidate classification, not execution or performance evidence. #504 remains open for the explicit checksum-verified acquisition path, owned minimal shell adapter, exact-output execution, raw repeated results, anti-specialization/tier checks, and additional-engine corroboration.

The explicit acquisition/offline-verification boundary followed in 77939f13.

After an operator performs the documented exact-repository clone and detached checkout outside zig-js, zig build independent-suite-audit -Dindependent-suite-id=... -Dindependent-suite-checkout=... now verifies, offline:

• the checkout resolves outside the zig-js repository, including through symlinks;
• origin is the exact inventoried repository;
• HEAD and its full tree match the frozen commit/tree pair;
• the checkout has no tracked or untracked changes;
• every inventoried license, harness, workload, and data file exists and matches its SHA-256.

Live verification passed the exact Octane checkout with 22 pinned files and the sparse JetStream metadata checkout with five pinned files. A wrong-suite/wrong-origin probe and an intentional untracked-file probe both failed closed; the probe file was removed and the exact Octane checkout passed again.

Only the documented operator git clone uses the network. The verifier itself never fetches and remains outside ordinary build/runtime dependencies. This advances the explicit, offline-after-fetch, checksum-verified acquisition acceptance. The owned shell adapter and retained execution/result schema remain next.

The owned zig-js Octane shell adapter landed on main in ee31e4db.

zig build independent-suite-zig-js now performs the exact out-of-tree checkout audit before starting a separate zig-js process. The runner then:

• repeats SHA-256 validation for the pinned base.js and selected workload bytes;
• adds only load and print to the normal embedding environment and globalThis, with host functions allocated as real managed cells;
• accepts only the six-result candidate set (richards, regexp, splay, navier_stokes, and box2d) and rejects excluded/unknown rows;
• preserves upstream source, timing, inputs, warmup, iteration counts, results, errors, score strings, and auxiliary prints without transformation;
• rejects a dirty zig-js worktree, a supplied revision other than exact HEAD, or environment drift from TZ=UTC, LC_ALL=C, and LANG=C;
• emits one schema-1 child containing engine path/hash/revision/argv/environment, exact loaded sources, status/failure/skip, every output, raw wall/CPU/peak-RSS sample, explicit single-sample dispersion, validation, both timing boundaries, bytecode admissions, heap counters, and process isolation.

Scoring and attribution are deliberately separate. score leaves execution/allocation/pause instrumentation disabled and writes not_measured_scored_path plus null, never fake zeros. attribution enables the complete named tier, compilation/deoptimization, allocation, GC, and pause inventory; its timing/score is diagnostic rather than scored evidence.

The exact post-commit command passed the 22-file checkout audit and a clean-revision Richards score sample. Development compatibility probes also demonstrated that failures remain rows: Richards, Splay/SplayLatency, and Box2D completed, while RegExp retained SyntaxError: UnexpectedToken and NavierStokes retained RangeError: evaluation step budget exceeded. Those probes are not publication evidence and no source or limit was changed to turn them into scores.

Validation passed benchmark-comparison-test, the two-mode adapter self-test, dependency and independent-suite audits, the 100-file docs link gate, and the 49-page docs build. The change flips 0 test262 cases (benchmark/tooling/docs only).

#504 remains open for repeated raw-sample collection/dispersion and durable artifacts, anti-specialization variants, system JSC plus independently installed V8/SpiderMonkey/QuickJS adapters, and publication wording after every row/failure/skip is retained across engines.

The repeated-sample/durable-artifact slice landed on main in 06e4ad5d.

zig build independent-suite-collect now runs every applicable Octane row in fresh isolated processes, with uninstrumented score rounds and separate attribution samples. The schema-1 collection:

• atomically checkpoints to a required external path after every child, so each later child can still prove the zig-js source worktree is clean and exact;
• rotates row order by score-sample index;
• retains raw stdout, parsed child JSON when available, stderr, exit/timeout state, and contract validation for every child, including malformed transports and upstream failures;
• validates exact suite/tree/file/license/result names, runner executable identity, source revision, argv/environment, timing boundary, and score-vs-attribution instrumentation boundary across the collection;
• computes median/range/mean/sample-RSD only from contract-valid passed score children while listing every excluded failed/invalid sample index;
• refuses the selected six-result non-browser geometric aggregate unless the collection is complete and every applicable row passes;
• marks fewer than seven score samples, non-quiet hosts, non-AC power, incomplete matrices, and row failures as diagnostic rather than publishable evidence.

The exact post-commit two-score/one-attribution diagnostic passed the 22-file checkout audit and retained all 15 children in /tmp/zig-js-issue504-collector-06e4ad5d.json (SHA-256 a31e017b97bf3de3976550b7bed4c6c920275400d2ad103488a687e9840d8f61). Richards, Splay/SplayLatency, and Box2D each retained two valid score samples and one valid attribution sample. RegExp retained three valid failed children with SyntaxError: UnexpectedToken; NavierStokes retained three valid failed children with RangeError: evaluation step budget exceeded. Both had zero valid score samples, so the artifact explicitly reports the aggregate as unavailable and the command exits nonzero after writing it.

Validation also passed benchmark-comparison-test, dependency-audit (14 classified edges, zero migration targets, 60 repository tools), independent-suite-audit, the 100-file docs link gate, and the 49-page docs build. This flips 0 test262 cases.

#504 remains open for the underlying RegExp/NavierStokes compatibility work, anti-specialization variants, system JSC and additional-engine adapters, and publication wording after a complete all-passing evidence matrix exists.

The required-when-available system-JavaScriptCore adapter landed on main in ff0b132c.

zig build independent-suite-jsc now builds and runs a separate macOS executable linked only to the real platform JavaScriptCore framework. It shares one frozen Octane path/checksum/license/result table with the zig-js adapter, while independently owning context creation, load/print callbacks, exact source evaluation, protocol capture, output validation, and schema-1 JSON serialization. The executable never links zig-js, so zig-js's JSC-shaped exports cannot interpose on the framework.

Each child records the adapter executable path/hash and exact adapter source revision, plus framework bundle version, macOS build, argv/environment, source identities, every upstream/auxiliary output, process wall/CPU/peak RSS, validation, and timing boundaries. This macOS supplies JavaScriptCore code through the dyld shared cache rather than a standalone framework image; the artifact records that boundary and leaves the unavailable binary path/hash null. Exact per-context tier/compilation/deoptimization and GC/allocation/pause counters are likewise unavailable_public_api, never zero.

Exact post-commit diagnostics passed the 22-file checkout audit and all five applicable rows unchanged: Richards, RegExp, Splay/SplayLatency, NavierStokes, and Box2D. All processes reported adapter SHA-256 2f37264e3aaa90adc3e07454754ddcbc0c643c08934ac18990364e83d9837686, system JavaScriptCore framework 22625.1.24.11.2, and macOS build 26A5388g. These were single-sample compatibility diagnostics, not repeated or publishable performance evidence, so their scores are not promoted.

Validation passed both engine-adapter self-tests, the shared frozen-table test, full benchmark-comparison-test, dependency-audit (14 classified edges, zero migration targets, 60 repository tools), independent-suite-audit, the 100-file docs link gate, and the 49-page docs build. This flips 0 test262 cases.

#504 remains open for repeated cross-engine collection, optional independently installed V8/SpiderMonkey/QuickJS adapters, anti-specialization variants, the zig-js RegExp/NavierStokes compatibility gaps, and publication wording after complete repeated evidence exists.

The first zig-js Octane RegExp blocker is removed by zig-regex 7196733, with the exact ECMAScript boundary locked downstream in zig-js e8432679. The unchanged pinned regexp.js now parses and executes instead of failing UnexpectedToken. A fresh isolated diagnostic runner (b00deb1327ae47f47e94a119c43e9f24abd0aac8a29337f169a4dd8ffdde672d) reaches a distinct runtime failure, TypeError: value is not a function; the row therefore remains failed and no score is publishable. I am continuing with that runtime defect, not masking it or promoting the partial result.

The zig-js RegExp row now passes unchanged in both isolated modes. Failure-stack retention 2e849880 plus structured Error stacks 9633ca81 localized the second failure to Exec calling .exec on the first literal. The engine fix 72745298 separates literal/internal RegExp allocation from constructor new.target, eliminating the wrong-prototype object.

A fresh clean-revision runner (SHA-256 9622d7034cc0d61808efe2981ff98e3ec350f9d83ca1ac48e32de2a80756741c) verified the exact pinned base.js and regexp.js, retained the expected RegExp result and one final score, and reported status: passed in both uninstrumented score and separately instrumented attribution processes. These are single compatibility diagnostics, not repeated/publication evidence, so no score is promoted. The remaining zig-js row blocker is NavierStokes; #504 also remains open for dependency provenance, repeated cross-engine collection, anti-specialization, optional engines, and publication wording.

The exact unchanged Navier-Stokes row now passes end-to-end in both score and separate attribution modes at zig-js revision 6ac8040.

Root cause was a harness/resource-boundary mismatch: the finite pinned workload reached the ordinary 500,000,000-step embedder runaway guard before producing output. Commit https://github.com/zig-utils/zig-js/commit/6ac804046a1a2c7d288236ad7d3554a00b6685b6 assigns termination uniformly for every selected row and mode to the collector process timeout, sets the isolated harness evaluation budget to u64 maximum, and retains/validates both facts in schema-1 evidence. Source bytes, inputs, iteration counts, timing logic, and expected outputs remain unchanged.

Validation: full benchmark-comparison-test passed; independent-suite inventory and collector drift gates passed; docs link/build gates passed; score mode produced the required NavierStokes result and one final score; attribution mode produced the same required output contract with tier/GC counters retained. Both runs are single diagnostic samples and their numeric scores are not publication evidence.

Closed the remaining linked-source provenance gap with https://github.com/zig-utils/zig-js/commit/0382848502ed5a56d95f73b1c31d7344d4289306.

The zig-js adapter now rejects dirty or noncanonical local path dependencies and retains the resolved path, canonical repository, exact revision, and clean status for both zig-regex and zig-gc. The collector validates the inventory and requires one stable dependency identity across the collection.

An exact clean Richards diagnostic at 03828485 passed and retained zig-regex 7196733 plus zig-gc c89a10f. Full benchmark-comparison-test, inventory, collector, docs-link, and docs-build gates passed. The single numeric score remains diagnostic and is not publication evidence.

The first complete all-passing repeated zig-js collection is now validated at revision 0382848.

External artifact: /tmp/zig-js-issue504-collector-03828485.json (332,271 bytes, SHA-256 e17ffe7d041ca85ed3fd27cdacdb35523f1431de6c9243e2239f863137d22430). All 15 fresh processes passed contract validation: Richards, RegExp, Splay/SplayLatency, NavierStokes, and Box2D each retain two valid uninstrumented score children plus one separate valid attribution child, with no failed/invalid indexes. The artifact retains stable zig-js executable/source identity and exact clean zig-regex 71967331 / zig-gc c89a10f inputs.

The selected-subset aggregate is structurally available now that every applicable row passes. Publication status remains diagnostic_not_publishable exactly as intended: this run has fewer than seven score samples per row and used a diagnostic host class. No numeric score is promoted.

Remaining #504 acceptance work is anti-specialization/source-recognizer evidence, additional-engine availability/adapters beyond the completed system-JSC path, and final evidence-backed wording.

Anti-specialization/source-recognizer acceptance is now implemented by https://github.com/zig-utils/zig-js/commit/3adb9817168a0e04c78c838ba13e8f6905895105 and validated on the exact clean revision.

The scored path remains byte-for-byte upstream. The new diagnostic path verifies each pinned source SHA-256 first, then prepends a deterministic block comment so the evaluated source hash and offsets change without changing the AST/workload. A static audit also rejects the Octane revision/tree, all selected source hashes, and distinctive suite identifiers anywhere under src/.

External recognizer artifact: /tmp/zig-js-issue504-recognizer-3adb981.json (219,564 bytes, SHA-256 6fb7935d89c4fd4cdf1765993aa0dd5051b7d255af8a01caacbee80d72287dee). All ten isolated attribution children passed. Richards, RegExp, Splay, NavierStokes, and Box2D exact/mutated pairs each retained changed evaluated hashes, identical required output contracts, and identical nonzero execution-tier plus bytecode-admission sets. The artifact is explicitly diagnostic_not_performance_evidence; no timing or score is promoted.

Validation also passed the full benchmark-comparison-test, inventory/adapter/collector/recognizer self-tests, docs-link-check, and manifest-checked docs build. Remaining #504 work is optional additional-engine availability/adapters and final scoped wording.

The first additional independently installed engine control landed in https://github.com/zig-utils/zig-js/commit/855601b83892053c66a808a5154c80e81f011808.

This is labeled node-v8 deliberately: a fresh node:vm context in Node v24.18.0 embedding V8 13.6.233.17-node.50, not a standalone d8 shell and not a browser. Each child retains Node executable path/SHA-256 9fcf926400e3997bc0a3be0c0b00f95df95db9b58b5754d5ca9d02e7fed866f9, adapter path/SHA-256 46310908dcf77b8711ec23302f4d13672ac6d418edc54e2661f00bed0a31d377/revision, exact sources, argv/environment, raw process sample, outputs, validation, and timed boundaries. Exact per-context V8 tier/GC fields are unavailable_public_api rather than zero.

Fresh clean-revision diagnostics passed all five applicable rows unchanged with the required Richards, RegExp, Splay/SplayLatency, NavierStokes, and Box2D outputs and one final score each. These are single compatibility diagnostics; no numeric score is promoted. The optional Node/V8 step does not become an ordinary dependency, and standalone d8, SpiderMonkey, and QuickJS remain distinct planned adapters rather than being misrepresented by this control.

Validation passed the Node/V8 adapter self-test, six-engine inventory audit, docs-link gate, and manifest-checked docs build.

Acceptance is complete on main. Final slices include all-passing repeated zig-js evidence with complete dependency identity (03828485), exact/source-hash-mutated recognizer equivalence (3adb981), and an explicitly labeled independently installed Node/V8 control (855601b) alongside the separate system-JSC adapter. All scored upstream sources remain unchanged; diagnostic runs are not promoted as publication evidence. Standalone d8, SpiderMonkey, and QuickJS were unavailable on this host and remain distinct optional planned adapters, not silently substituted.