ReviewOS

zig-utils/zig-js

Bench: gate exact-parent samples on measured-boundary CPU occupancy

#768
Closed chris opened this 11 days ago ·
11 days ago

Parents: #461, #670 Blocks evidence closure for: #767 Related: #493, #614

Problem

tools/exact-parent-regression.ts applies the 60% CPU-occupancy gate to /usr/bin/time -l totals for the complete benchmark process, while its scored wall/instruction/cycle/energy counters bracket only the production invocation. Untimed context startup, warmup, teardown, thermal queries, and allocation replay can dominate a fast optimized candidate. A valid short boundary is then rejected even when no competing build exists; multiplying jobs enough to amortize that untimed work makes a quadratic parent prohibitively long, which is exactly the evidence problem #670 was created to solve.

Observed on the frozen #767 ASCII-1024 full contract (10 jobs, one lane): the candidate scored boundary was 13.456 ms with the frozen checksum and 625,799,311 retired instructions, while complete-process accounting was 0.42 s real / 0.03 s user / 0.00 s system. The current process-wide gate therefore cannot distinguish untimed runner overhead from descheduling inside the scored boundary. No sample was retained.

Scope

  • Add Darwin user and system CPU-time deltas to the same native snapshots that already bracket the exact measured invocation in both production comparison runners.
  • Parse and identity-check those deltas in the exact-parent collector.
  • Apply the unchanged 60% occupancy threshold to measured-boundary CPU time divided by measured-boundary elapsed time.
  • Retain complete-process wall/user/system/occupancy as diagnostic provenance, not as a scored-boundary quality proxy.
  • Preserve exact parent/candidate identity, checksums, balanced sampling, thermal checks, cleanliness, and fail-closed before/after competing-process guards.
  • Keep historical artifacts readable and make new publication artifacts explicitly identify the boundary-occupancy policy.
  • Cover missing/malformed telemetry, short candidates, genuinely descheduled boundaries, multi-threaded boundaries, and legacy artifact validation.

No-workaround rules

  • No threshold reduction, fixed-duration exemption, discarded sample, asymmetric job count, source rewrite, heuristic cycle-to-time conversion, or wall-time claim from algorithmic-growth evidence.
  • No stacks, queues, jobs-runtime, Promise, Worker, or Map/Set iterator changes.

Acceptance

  • Native runner telemetry reports exact nonnegative measured-boundary user/system CPU deltas with identity checks.
  • New collections require at least 60% measured-boundary occupancy and reject a synthetic descheduled boundary below it.
  • Existing process-quality, exact-revision, checksum, cleanliness, thermal, and no-competing-job guards remain fail-closed.
  • Tool self-tests, benchmark contract tests, dependency audit, docs links/build, and focused runner probes pass.
  • #767 exact growth evidence collects without increasing its frozen logical job counts or weakening any stability rule.
11 days ago

Landed a5975dbb (fix(bench): gate exact samples on measured CPU time (#768)).

Both native runners now snapshot getrusage(RUSAGE_SELF) user/system CPU time around the exact scored invocation and emit a separate identity-bound telemetry row. The exact-parent v2 collector applies the unchanged 60% gate to that boundary and retains whole-process occupancy only as diagnostic provenance. Attribution v1 and algorithmic-growth v1 remain byte-identical; v2 plus representative matrix V24 record the incompatible quality-scope change.

Native probes after the final implementation:

  • string indexing: 13.705 ms CPU / 13.734 ms elapsed (~99.8%) with frozen checksum 23,880,132, while complete-process occupancy was only about 9.7%;
  • frontend capture classifier: 0.389 ms CPU / 0.375 ms elapsed (capped at 100%), checksum and exact allocation replay present.

Validation: exact-parent, algorithmic-growth, instrumentation, attribution, V24, and 52/52 matrix mutation self-tests pass; both ReleaseFast native runners build; docs links 277/277; dependency audit 16 edges / 0 migrations / 63 tools; BunPress 50 pages. Flips 0 Test262 cases. The issue remains open only until the frozen #767 exact growth rows collect through this v2 path.

11 days ago

Landed 7831d3df (fix(bench): attest exact binary overlays (#768)). Exact-parent v3 artifacts now distinguish logical engine revisions from binary revisions and fail closed unless both binaries are one-commit children of their logical revisions, change the fixed V24 measurement-overlay inventory, and contain byte-identical blobs on every overlay path. The generic attribution validator also rejects malformed revision IDs and unsafe/non-canonical overlay paths.\n\nValidation: exact-parent self-test passed; algorithmic-growth self-test passed; performance-attribution verified its schema failures and 168 legacy rows; representative matrix V25 passed 52/52 structural mutations and validates 18 implemented families with 0 deferred; dependency audit passed 16 edges / 0 migrations / 63 tools; all 277 documentation files resolve; BunPress manifest check built 50 pages. Flips 0 Test262 cases.\n\nThe #767 collection will name logical parent 5bca24a359b2b571010547bdf096b3321f8d22d7, logical candidate c65729e5db60041ef68bf0bc7ae5f3d880a368f7, parent binary revision 7d82c96e05ca32a64e8d885ae98e7a935795ab3c, and candidate binary revision a5975dbb504e0d3eb9fbce1df6bc99e632e05996.

11 days ago

Schema-v3 measured-boundary gating has now produced a checked live artifact in 4e4531ec: two balanced ASCII-1024 pairs passed the >=60% measured-boundary occupancy requirement with attested overlay/binary hashes and exact checksums. The broader #767 batch still correctly fails closed whenever unrelated Home jobs appear during collector startup, so the final multi-row evidence acceptance remains open.

11 days ago

Completed by 33734e97 — bench(strings): publish guarded UTF-16 growth (#767).

The #767 collection kept the frozen contract unchanged: 10 logical jobs, one lane, two pair samples, single_no_jit, the exact workload checksum, and the 60% measured-boundary occupancy floor. Across all 72 timed process observations, minimum boundary occupancy was 99.67%; no thermal transition drifted and no competing benchmark child was allowed.

One initial astral-2048 parent pair exceeded the existing 5% wall RSD rule. It was backed up and recollected under the identical revisions/binaries/jobs/checksum instead of weakening or editing the rule. The accepted replacement records parent/candidate RSD 1.29%/0.91%, and all 18 final rows are stable.

Sign in to comment on this issue.