ReviewOS

zig-utils/zig-js

Benchmark: reject competing processes in JSON JSC collector

#1021
Open glennmichael123 opened this 8 days ago ·
8 days ago

Parent evidence lane: #1019

Measured problem

tools/json-pipeline-benchmark.ts collected two 126-sample direct-JSC triplets while unrelated compiler/test work started mid-run. The second run appeared to narrow both target gaps, but its shallow baseline/candidate/JSC RSD reached 30.01%/28.52%/41.56%. Raw paired candidate/baseline ratios ranged from 0.51x to 1.46x.

The contemporaneous host evidence identifies the interference: another Codex task launched a Zig test process, it crashed at 02:15:04, and macOS ReportCrash then consumed a core. The crash report is outside the repository and is not evidence to check in; all benchmark samples and the rejection rationale are preserved in c6082df7.

The exact-parent collector already calls competingEvidenceProcesses before and after every benchmark invocation. The JSON/JSC collector did not perform an equivalent check, so a short process could contaminate a complete matrix without causing collection to fail.

Scope

  • Reuse the shared tools/evidence-processes.ts classification rather than creating a second process parser.
  • Reject competing build/test/corpus/crash-report work before and after every zig-js, baseline, and JSC invocation in ordinary, gap-reduction, and tier-residency modes.
  • Exclude the collector and its direct child runner from self-detection exactly as the shared classifier requires.
  • Keep raw sample order, checksums, runner arguments, and report schema unchanged.

Acceptance

  • A deterministic self-test proves recognized competitors fail both pre- and post-invocation checks while a clean fixture passes.
  • The collector self-test and repository tool gate pass.
  • A contaminated run fails before publishing either artifact.
  • A clean seven-triplet #1019 run completes with the same 126-sample schema and exact checksums.
8 days ago

Progress landed directly on main:

  • e0e5550 freezes the shared competing-process policy in matrix V42.
  • 4550396 brackets every JSON/JSC, gap-reduction, and tier-residency invocation and refreshes the generated docs manifest.
  • ff8d3b0 freezes the guarded collector in matrix V43.
  • 388a01f distinguishes the permanent idle ReportCrash agent from active crash processing and re-pins both collectors in V44.

Verified:

  • exact-parent classifier self-test: pass
  • JSON pipeline self-test: pass
  • representative matrix V44: 18 implemented families, 0 deferred
  • structural mutations: 82/82 pass
  • docs build: 51 pages pass
  • live contaminated probe: exits 1 before sample 1 and writes neither raw JSON nor Markdown

The umbrella benchmark-harness self-test is currently blocked before assertions by an unrelated Home TypeScript-checker panic (start index 7 is larger than end index 6). The final clean seven-triplet / 126-sample run is also still pending because unrelated command-bearing Bun jobs are active on the host. I am leaving this issue open until those two acceptance items are honestly complete.

8 days ago

Blocker audit after the guarded commits:

  • The authoritative benchmark-comparison-test Home-driven command still aborts before executing its inventory. Running its imports individually proves benchmark-comparison, benchmark-publication, and test_representative_matrix pass; compiling the next module, tools/representative-benchmark.ts, panics Home's TypeScript checker with start index 7 is larger than end index 6. That zig-js tool has not changed since 224cd48 (2026-09-29), so this is an upstream compiler failure rather than a failed benchmark assertion. No syntax workaround was added.
  • The live host remains intentionally ineligible for the final 126-sample matrix: the unrelated wildloop Bun watch children have been resident for more than 16 hours. The idle macOS ReportCrash agent is correctly ignored by V44, while the command-bearing Bun watchers remain competitors because they can wake during a sample.

No clean matrix was started and no artifact was written. The issue remains open for the unchanged acceptance items: an upstream-green repository tool gate and one genuinely clean seven-triplet run.

8 days ago

Filed the independently reproducible upstream checker blocker as home-lang/home#834: /home-lang/home/issues/834

A fresh bounded single-module run of tools/representative-benchmark.ts --self-test exits 134 with the same start index 7 is larger than end index 6 panic before any validator runs. Peak tree footprint was 138 MB with 37% host low-water, ruling out memory pressure. The dirty Home checkout was not rebuilt or modified, and no zig-js syntax workaround was added.

8 days ago

The repository tool-gate blocker is now diagnosed upstream, not merely reproduced: home-lang/home#834 comment. Home records a diagnostic-array index, recursively checks a member receiver that may remove earlier diagnostics, then slices from the stale index. The logical upstream fix is to query the actual TS2684 diagnostic after the nested check. No zig-js syntax workaround has been added. The authoritative tool gate and clean 126-sample matrix remain pending that upstream fix and an eligible benchmark host.

8 days ago

The upstream gate blocker is cleared: Home #834 is fixed, representative matrix V47 re-pins the shared runner in 9940ddca, and zig build benchmark-comparison-test -Dhome-tool=<current-home-tool> --summary all now passes all 19 Home validators plus every native audit leg.

A fresh live preflight on 2026-10-07 still correctly refuses before sample 1 and writes neither output artifact. The classified competitors are an active ReportCrash daemon and the command-bearing ts-cloud ... dashboard:serve Bun process. I did not terminate unrelated user processes or publish contaminated evidence.

Acceptance status: the collector self-test and repository tool gate are complete. The sole remaining item is the genuinely clean seven-triplet / 126-sample run, which remains host-state blocked.

Sign in to comment on this issue.