ReviewOS

zig-utils/zig-js

Bench: add a duration-qualified arithmetic publication tier

#843
Closed chris opened this 11 days ago ·
11 days ago

Problem

The frozen required-bytecode/no-JIT arithmetic profile uses 100 jobs for smoke validation and 10,000 jobs for its full structural row. On the reference Apple Silicon host, the stable-Number full row completes in roughly 7.5 ms. proc_pid_rusage(RUSAGE_INFO_V6).ri_energy_nj is quantized at that boundary: a clean seven-pair AC run reported exact/stable instructions and cycles but process-energy deltas alternating between zero and coarse nonzero values. The exact-parent collector correctly blocked publication because energy RSD exceeded the frozen 5% limit.

This is a benchmark-contract defect, not permission to discard energy, weaken the efficiency gate, repeat until favorable, or publish a diagnostic row. The existing quick/full rows are useful and must remain byte-for-byte comparable for smoke, checksum, attribution, and allocation validation, but the scored publication boundary needs enough identical logical work to make every required efficiency counter measurable.

Scope

  • Version the representative matrix and preserve the existing 100-job quick and 10,000-job full contracts unchanged.
  • Add an explicit duration-qualified publication tier to the four required-bytecode/no-JIT arithmetic workloads.
  • Freeze 500,000-job checksums and require the exact-parent collector to use that tier for public efficiency evidence. Seven-pair sizing probes rejected 100,000 jobs because the coercion control remained at 6.2-6.8% process-energy RSD and rejected 200,000 jobs because its parent remained at 6.7%; all four parent/candidate rows are below 2.1% at 500,000 jobs.
  • Document that the publication tier changes only measurement duration, not the workload source, Context options, required-bytecode/no-native boundary, warmup policy, attribution semantics, or control coverage.
  • Extend matrix validation and rejection tests so missing, shortened, or checksum-drifting publication tiers fail closed.

Acceptance

  • Matrix version advances without rewriting historical manifests.
  • 100-job quick and 10,000-job full jobs/checksums remain unchanged.
  • All four 500,000-job publication checksums match exact parent and candidate binaries.
  • Matrix validator and structural mutation suite reject publication-tier drift.
  • Docs state why sub-counter-resolution rows cannot support public energy/efficiency claims.
  • The #734 exact-parent batch passes all stable-Number and control rows without weakening the 5% instruction/cycle/energy or nominal-thermal gates.

Related: #734, #503, #842.

11 days ago

Completed in:

V32 preserves the existing 100-job smoke and 10,000-job structural jobs/checksums. It adds only a 500,000-job publication tier for the four required-bytecode/no-JIT arithmetic witnesses, with exact parent/candidate checksums and the existing 5% instruction/cycle/process-energy, nominal-thermal, AC-power, no-discard, and provenance gates unchanged.

Sizing was fail-closed: 100,000 and 200,000 jobs were rejected because the shortest control still exceeded the process-energy RSD limit; seven-pair 500,000-job probes put every parent/candidate energy series below 2.1%. The final four artifacts all have stable efficiency evidence and no publication block. Matrix validation and 70/70 structural mutations, the 18/18 benchmark harness, 313-file link audit, dependency audit, and checked 50-page docs build pass.

Remote main is verified at 4d347739524d44fd1f4ae64096f2e9a786a20c22; all #843 acceptance items are complete.

Sign in to comment on this issue.