ReviewOS

home-lang/home

fix(bench): align the container harness with pinned TS 6 and TS 7

#464
Closed chris opened this 8 days ago ·
8 days ago

Follow-up to #416 and the compiler-version correction. The local Python harness now pins and verifies JavaScript TS 6.0.3 and native TS 7.0.2 in separate prefixes. The unused legacy bench/vs_tsgo/Dockerfile still installs global TypeScript 5.6.3, downloads an unpinned latest tsgo binary with a success-looking fallback, calls pantry without installing it, and documents a build context that does not contain the repository/compiler. It is not the environment used for the published Apple M3 Pro measurements.

Acceptance criteria:

  • Use the same corpus.toml pins and exact-version preflight as the local runner; count native TS 7 only once.
  • Provide a working, explicit repository build context and release Home binary build/mount strategy.
  • Fail when required tools are unavailable; remove the unpinned download/fallback.
  • Test the documented Linux container commands, preserve identical projects/options/negative controls, and publish environment-qualified results rather than claiming byte-for-byte timing reproducibility.
  • Keep the existing macOS results and caveats intact; do not imply unmeasured platforms are covered.
8 days ago

Starting this checkpoint from bf64a67. Audit confirms the committed container path still has every issue-body failure: global TS 5.6.3, unpinned latest tsgo with a success-looking fallback, wrong build context, and an unavailable pantry command. Docker is not installed on the local Apple host, so no Linux number will be inferred locally. I am replacing it with a fail-closed root-context image and a manual Ubuntu benchmark workflow that builds the exact source with the pinned Zig toolchain, runs the same run.py setup/admission/timing path inside the container, retains the complete artifact, and publishes only environment-qualified Linux results after the workflow actually succeeds.

8 days ago

Completed on origin/main via 351bc66, 82853c6, bbaa4bc, 5aa0234, and 924da0b. The image uses the repository root context with a 666-byte narrowed context, a native Linux ReleaseFast Home mount, pinned Node 24.18.0 Bookworm and Hyperfine 1.20.0 artifacts, and the same corpus.toml setup/admission path as the local runner. Missing tools and wrong compiler versions fail closed; TS 6.0.3 and the single native TS 7.0.2 competitor are installed separately. The documented command was exercised with Docker 29.7.2 rootless in an Ubuntu 26.04 ARM64 Lima/VZ VM using a native ext4 workspace. Result 20260829T035150Z retains 600 round files and 1,800 samples and records lower Home means on 20/20 workloads. The narrowest rows remain explicit: CheckJS 1.02x with 19/30 paired wins, null-safe access 1.03x, and CommonJS 1.05x. The macOS table remains intact, and VirtioFS results are explicitly ineligible as Linux performance evidence. GitHub Actions dispatch was unavailable for this repository, so the published checkpoint is the tested local Linux container run, not an inferred Actions result.

Sign in to comment on this issue.