ReviewOS

zig-utils/zig-js

Performance evidence: support explicit direct-binary provenance without dummy overlays

#914
Closed glennmichael123 opened this 8 days ago ·
8 days ago

Parents: #460, #496. Blocking complete modern evidence publication for #913.

Current behavior

tools/exact-parent-regression.ts defaults to attribution schema v3. Its v3 branch always calls validateSharedMeasurementOverlay, requiring each binary revision to be a one-commit child of its logical engine revision with a non-empty symmetric changed-path set. A binary built directly from the named logical parent/candidate therefore fails before collection. The existing validateDirectBinaryRevisions helper is reachable only for older schemas.

The frozen v3 contract explicitly describes identical measurement overlays, and the artifact validator requires a non-empty overlay inventory. That contract must not silently change. The ordinary direct-parent command shown in docs/benchmarks.md also omits the now-required binary provenance options.

Required change

Introduce a versioned first-class direct-binary provenance mode alongside the existing overlay mode. Direct mode must attest exact equality of logical and binary revisions, an empty overlay inventory, identical workload source, exact first-parent relation, clean source/dependencies and binary/source hashes. Overlay mode must retain the current strict symmetric-path/blob checks. Reject mixed modes, ambiguous or stale revisions, undeclared changes, missing metadata, and attempts to label an overlay as direct.

Keep v1/v2/v3 artifacts and validators byte-compatible. Do not create dummy overlay commits, downgrade a new experiment merely to bypass provenance, weaken checksum/occupancy/thermal/dispersion guards, or invent performance results. Add negative fixtures and update the owned collector/growth contracts and documented direct-parent invocation. Then collect #913's actual parent/candidate evidence using the matching mode.

8 days ago

Validation update: the direct/overlay negative fixture exposed a root-commit diagnostic edge; the v4 collector now explicitly rejects both mixed modes before walking overlay ancestry, with one-sided negative fixtures. The full 18-validator gate also correctly refused the changed collector under frozen representative matrix v33. The in-progress change now adds v34, hash-inheriting v33 and re-pinning only the versioned evidence integration; workload, jobs, checksums, native runners, attribution counters, and quality gates are unchanged. Full validation is being rerun through Home under the shared machine lock with a 3 GB ceiling. Nothing in this tooling batch is committed or claimed passing yet. #913 performance publication remains pending.

8 days ago

The v34 full benchmark-harness startup reached its 1200-second bound before any test-body marker; a live stack sample showed Home ts_driver.compileSource/checkSourceFile, with 32.4 MB physical footprint. This is recorded as an incomplete gate, not a passing run. Four focused validator entry points are now queued serially through Home to localize validation. The independent docs-link-check passed: 328 files, all links resolve. The docs renderer was missing at the configured path; the nearby BunPress checkout is version 0.1.11, while CI pins 226414a. Restoring that exact owned dependency is queued under the same machine lock, preserving the old checkout. No schema/matrix source from earlier versions was edited, and no unverified tooling changes have been committed.

8 days ago

Landed tooling and documentation in 650efda (pushed to main). Attribution/growth v4 now distinguish explicit direct builds from strict shared overlays; matrix v34 preserves every scored row and historic contract. All four affected Home self-tests passed, including representative matrix 75/75 and legacy artifact/report checks. Docs link check passed for 328 files; CI-pinned BunPress 0.2.14 checked all 50 rendered pages. The manifest also reconciles already-committed ABI prose from 0e3e248. No engine behavior or performance claim changed. The full 18-suite inventory previously timed out during Home compilation at 1200s; a fresh run is now active with a 3600s runtime bound and the same 3 GiB/exclusive-machine guard. Keeping this issue open for that result and the actual #913 direct-build collection.

8 days ago

Full benchmark harness inventory completed successfully: 18/18 suites passed (exit 0), under the shared machine lock and unchanged 3072 MB guard. Observed peak process-tree footprint was 35 MB; host low-water 44%. Log: /private/tmp/zig-js-provenance-full-inventory-retry.log. This completes the pending full tooling validation; actual #913 direct-build performance collection remains outstanding, so this issue stays open. No new throughput claim is being made.

8 days ago

The complete direct-build collection and owned growth/report validation are now published in be59a31. See the final #913 evidence update for all six rows, 84 preserved samples, controls, diagnostic-only claim limits, and passed docs gates. Combined with the recorded 18/18 full tooling-validator result, this completes the remaining requirement. No historical contract or README speed claim was changed.

Sign in to comment on this issue.