ReviewOS

zig-utils/zig-js

Bench: reject competing jobs during exact-parent evidence

#617
Closed chris opened this 9 days ago ·
9 days ago

Parents: #461, #492 Related: #614, #615, #616

Problem

tools/exact-parent-regression.ts could collect and write a valid-looking exact-parent artifact while an unrelated Zig build/test or Home test executable overlapped a parent/candidate sample. Process-local instruction counters may remain stable, but wall/cycle/energy attribution and order balance are contaminated.

Two live #615 attempts exposed both races:

  • a Home zig test began during a row and remained alive for the after-snapshot;
  • /private/tmp/home-url-final began and exited entirely inside a row, evading both snapshots while inflating a normally millisecond invocation to roughly ten seconds.

Scope

  • Define the exact evidence competitors explicitly from process-table rows.
  • Reject unrelated zig and maker processes plus known repository test/corpus executables before and after every parent/candidate invocation.
  • Exclude the collector and its own runner ancestry/children without PID guessing.
  • Measure complete-process wall and CPU time; reject CPU occupancy below 60% so a transient competitor cannot hide between snapshots.
  • Persist the full-process wall time, occupancy, and threshold in every raw sample and rendered report.
  • Record no artifact when overlap is detected; partial output paths must not masquerade as complete evidence.
  • Add deterministic fixtures for ancestry exclusion, competitor classification, unrelated process tolerance, before/after failure behavior, occupancy calculation, policy drift, and below-threshold failure.

No-workaround rules

  • No sleeps, retries that hide an overlapped sample, process killing, bare name substring matching, or accepting stable counters as proof of an uncontended host.
  • Do not treat browser/system background activity as a named compiler/test competitor; host-class/RSD/thermal policy continues to govern ordinary noise.
  • Never delete or modify the competing user-owned process.

Acceptance

  • Self-tests prove exact competitor classification and own-runner exclusion.
  • Every measured invocation fails closed on a competitor present before or after it and writes no complete artifact.
  • Complete-process CPU occupancy rejects transient overlap that begins and ends between snapshots.
  • The occupancy observation and minimum threshold are validated and persisted in raw and Markdown artifacts.
  • Existing exact-parent identity, checksum, cleanliness, pairing, and schema gates remain green.
  • #615/#616 evidence is collected only after this guard lands and passes.
9 days ago

Pushed bcf53f57 (fix(bench): reject competing exact-parent jobs (#617)).

The collector now checks before and after every parent/candidate invocation. Deterministic fixtures cover collector ancestry, owned runner descendants, sibling Zig/maker processes, Home test executables, zig-js unit/corpus runners, another frontend benchmark, and unrelated-process tolerance. Existing exact-parent identity, checksum, pairing, cleanliness, schema, regression, and thermal/counter tests remain green; dependency audit remains 15 edges / 0 migrations / 60 tools.

A live fixture then exercised the exact observed race: a Home test began between checks. The collector rejected the row after the invocation with PID/command evidence and produced neither the requested JSON nor Markdown output. No process was killed or modified.

The issue stays open only until #615/#616 artifacts are successfully collected behind this landed guard.

9 days ago

Follow-up pushed in 16cd30f8 (fix(bench): reject transient exact-parent overlap (#617)).

The first guard correctly caught persistent jobs, but a second live attempt exposed the remaining interval race: /private/tmp/home-url-final began and exited entirely inside one invocation. The before/after snapshots therefore saw no PID even though a normally millisecond row took roughly ten seconds.

Every invocation now records its complete /usr/bin/time -l wall, user, and system times, computes process CPU occupancy, and fails below 60%. The raw sample and Markdown report preserve the wall time, occupancy, and threshold; artifact-specific validation rejects missing, below-threshold, over-100%, or policy-drifted quality data before either output is written. Snapshots remain in place for exact PID/command attribution when a competitor persists.

Validation: exact-parent self-test passes; dependency audit remains 15 classified edges / 0 migrations / 60 tools. This flips 0 Test262 cases. #615/#616 evidence still remains intentionally uncollected while the live Home job is active.

9 days ago

The #617 fail-closed process/occupancy policy is now shared with the focused-build collector in d8fc8c97, linked from #614. Exact-parent’s existing self-test remains green after extracting the common classifier and occupancy calculation. This is a policy-preserving refactor for exact-parent evidence and adds the same protection to V2 build-feedback artifacts; the remaining #617 acceptance item is still the guarded #615/#616 collection.

9 days ago

Related fail-closed runner child #620 is complete in 34f5c8c6: CLI mistakes can no longer silently launch or leave a full Test262 driver tree. The final uncontaminated valid-path smoke passed 38/38 with empty before/after process snapshots.

9 days ago

Live fail-closed validation during #621: a seven-pair exact-parent collection began on a clear host, then a new Home maker/TypeScript conformance/home-url-final cycle appeared between invocations. The collector stopped with competing build/test process detected before benchmark invocation; neither requested JSON nor Markdown output existed afterward, and no process was killed or modified. This is additional real-host evidence for the completed guard behavior. The final #615/#616 successful guarded collections remain the only unchecked acceptance item.

9 days ago

The final acceptance item is now satisfied: a complete successful guarded collection ran end-to-end in 0abebd1a. During the hunt, the collector rejected multiple live attempts fail-closed — competing Home `ts_checker`/`maker` builds and transient `home-url-final` processes between before/after snapshots — writing no artifact for any rejected attempt; the three retained widths (7 pairs each, 42 guarded invocations) all passed pre/post competitor checks with >=60% CPU occupancy. Closing.

Sign in to comment on this issue.