ReviewOS

zig-utils/zig-js

Bench: add exact algorithmic-growth evidence profiles

#670
Closed chris opened this 9 days ago ·
9 days ago

Parents: #461, #493, #494

Blocks evidence closure for: #615, #616, #618, #619, #621

Problem

The exact-parent A/B profile correctly requires wall time, retired instructions, cycles, and energy to remain within 5% RSD on a quiet reference host. Compiler classifier fixes can reduce a candidate boundary from quadratic work to a few milliseconds, below the energy and cycle meters' stable resolution. Multiplying identical jobs enough to stabilize the candidate makes the exact quadratic parent prohibitively long at the 4K growth point.

This is not permission to weaken the full-efficiency profile or publish noisy wall/energy results. We need a distinct algorithmic-growth artifact whose claim boundary is normalized compiler work across input sizes, while full efficiency remains a separate evidence class.

Observed exact #615 attempts on the quiet reference host, with exact parent e9412de and candidate 8da47d8, Zig 0.17.0-dev.1441, seven order-balanced pairs, nominal thermals, and identical checksums:

  • 10 jobs at 1K: candidate wall 4.000 ms; candidate energy RSD 105%, correctly blocked.
  • 100 jobs at 1K: candidate wall 37.237 ms; candidate wall/cycles/energy RSD 25.1%/11.4%/9.3%, correctly blocked.
  • Retired instructions were stable in both attempts (candidate RSD 0.09% and 0.14%) and exact allocation replay was invariant.

Raw rejected artifacts remain outside the repository in the active #615 evidence workspace until this contract is implemented.

Scope

  • Add a versioned algorithmic_growth artifact/profile alongside, not inside, the current full-efficiency exact-parent profile.
  • Score at least three frozen input widths using exact parent/candidate revisions, identical generated sources, admission/result checksums, structural checksums, allocation replay, and normalized retired instructions per logical job.
  • Require stable instruction samples at every width and declare wall/cycles/energy diagnostic or unavailable with explicit provenance; never encode missing or noisy counters as zero.
  • Report fitted/adjacent growth ratios with raw rows so bounded near-linear candidate growth and superlinear parent growth are auditable without a throughput claim.
  • Keep one separately sized full-efficiency row when a representative boundary can satisfy the existing wall/instruction/cycle/energy policy.
  • Preserve fail-closed competing-process, cleanliness, exact-first-parent, toolchain, thermal, power, and checksum guards.
  • Add schema/tool self-tests for short candidate boundaries, nonlinear parents, unavailable/noisy metrics, normalization, and rejected publication attempts.

No-workaround rules

  • No threshold relaxation, selective sample removal, source rewriting, asymmetric parent/candidate work, heuristic extrapolation, wall-time claim from a diagnostic row, or replacement of the existing full-efficiency profile.
  • No stacks, queues, jobs-runtime, Promise, Worker, or Map/Set iterator changes.

Acceptance

  • A versioned algorithmic-growth schema and validator reject identity, checksum, normalization, width, sample, or stability drift.
  • #615's frozen 1K/2K/4K witnesses produce exact-parent growth evidence without claiming noisy wall/energy performance.
  • At least one unchanged full-efficiency fixture still blocks on unstable energy, proving the existing policy was not weakened.
  • Raw commands, exact revisions/toolchains/dependencies, samples, normalized metrics, and rendered methodology are checked in and docs/link gates pass.
9 days ago

Landed the additive algorithmic-growth evidence contract in fabf6c7c (tools(perf): add algorithmic growth evidence (#670)).

The profile consumes complete validated exact_parent_ab artifacts across at least three widths, replays process-quality checks, reconstructs every ordinary A/B summary, requires <=5% retired-instruction RSD, and retains exact allocation/byte counters (including valid zero-allocation results). Wall time, CPU, RSS, cycles, energy, and thermals remain embedded diagnostics and cannot support throughput, latency, energy, or full-efficiency claims. The existing V19 exact-parent collector and full-efficiency gate remain byte-for-byte unchanged.

Validation: benchmark-comparison-test 32/32 steps with 4/4 tests; dependency audit 62 repository tools; docs links 269/269 files; BunPress 50 pages; 0 Test262 flips.

I have started the #615 evidence collection, but the fail-closed competing-process guard currently sees unrelated Home/Bun test builds, so no measurement has been accepted yet.

9 days ago

Reconciled two acceptance boxes against the landed contract in fabf6c7c. tools/algorithmic-growth.ts validates schema/profile identity, exact parent/candidate and dependency identities, width ordering/count, sample stability, checksums, normalization, and allocation replay. Its self-test also explicitly requires the unchanged ordinary summary to remain blocked_efficiency_evidence, proving the additive profile did not weaken the full-efficiency policy.

The two evidence-dependent boxes remain open: no 1K/2K/4K #615 artifact has been accepted, and therefore no raw commands/samples/rendered methodology are checked in. The host is currently occupied by unrelated Zig builds, so I am not starting a measurement that the competing-process guard would correctly reject.

9 days ago

The AC requirement is now satisfied (100% on AC). The next exact algorithmic-growth row, TDZ-clear 2K, was correctly rejected before sampling because a Home Zig link began after the quiet preflight. No artifact or performance claim was retained. The exact-parent pipeline is enforcing the competing-process gate; work resumes when the host provides an uninterrupted window.

9 days ago

Box 2 satisfied: #615's frozen 1K/2K/4K witnesses produced exact-parent growth evidence in 0abebd1a — normalized retired instructions scored across three widths with parent exponent 1.989 vs candidate 1.004, allocation replay exact, all wall/cycle/energy observations retained as diagnostics with ordinary rows still `blocked_efficiency_evidence` (the ordinary profile was not weakened). Remaining open scope on this issue is the same evidence class for the sibling classifier issues (#616/#618/#619/#621).

9 days ago

Sibling evidence update: #621 is complete in 1a0cf6e2 (evidence(compiler): land binding-inventory growth rows (#621)). Its frozen 1K/2K/4K exact-parent rows validate under zig-js-algorithmic-growth-v1: both variants remain linear (first-to-last exponent 1.003 vs 1.003), while the candidate uses about 0.94x retired instructions at every width and strictly fewer allocations/bytes per job. Wall/cycle/energy rows remain diagnostic and the ordinary efficiency profile was not weakened. Remaining sibling evidence on this issue: #616, #618, and #619.

9 days ago

Documentation accuracy follow-up: eae28741 / #733 corrects the checked #615 and #621 exact/aggregate boundary metadata from “twenty warmup jobs” to the two untimed one-job warmups the frozen runners actually execute. Both aggregates were regenerated and validate with new source/embedded hashes; every measured sample, checksum, counter, allocation replay, and growth conclusion is unchanged. Offline links pass 277/277 files and BunPress builds the checked 50-page manifest.

9 days ago

Final reconciliation: the remaining evidence/publication contract is satisfied by 0abebd1a and its accuracy correction eae28741. docs/.data/algorithmic-growth-tdz-clear-2026-08-23.json losslessly embeds all three exact-parent rows and records logical revisions, first-parent identity, dependency revisions, binary/source hashes, Zig/OS/hardware/power identities, every paired sample, exact checksums, allocation replay, normalized retired instructions, and the non-throughput claim boundary. docs/benchmarks.md checks in the collection/validation commands and rendered methodology. The recorded final gates are 277/277 offline-linked files and a successful 50-page BunPress build. All four acceptance items are now proven; closing #670. This closes the evidence-profile infrastructure only—the remaining compiler children retain their own evidence gates.

9 days ago

The additive algorithmic-growth profile now has a second compiler-classifier publication using the current v3 binary-overlay provenance: #619 is closed by 89c8a2c4.

Its 1K/2K/4K deferred-class frame rows retain every ordinary A/B decision while scoring only stable normalized instructions and exact allocation replay. The exact parent grows 14.9477x (exponent 1.951) and the candidate 4.1586x (exponent 1.028); no wall-time or full-efficiency claim is made. This is a concrete reuse/cross-link of #670's fail-closed evidence boundary.

9 days ago

A third compiler-classifier publication now reuses the v3 additive growth boundary: #618 is closed by f47cd07c.

Its 1K/2K/4K repeated-body rows retain every ordinary A/B decision while scoring only stable normalized instructions and exact allocation replay. The exact parent grows 14.6568x (exponent 1.937) and the candidate 4.0318x (exponent 1.006); no wall-time or full-efficiency claim is made. This cross-links #670's fail-closed profile with #618 and the related #616 loop-head boundary.

9 days ago

The related loop-head classifier publication is now complete under the v3 additive growth boundary: #616 is closed by be1478fd.

Its 1K/2K/4K rows retain every ordinary A/B decision while scoring only stable normalized instructions and exact allocation replay. The exact parent grows 14.8145x (exponent 1.944) and the candidate 4.0486x (exponent 1.009); no wall-time or full-efficiency claim is made. This completes the #616/#618 loop-classifier cross-link under #670.

Sign in to comment on this issue.