ReviewOS

home-lang/home

test(ts): record and verify benchmark executable provenance

#544
Closed chris opened this 9 days ago ·
9 days ago

Follow-up under #416 discovered while publishing #536. Full-run metadata currently records reported versions but not exact command paths or content identities; all Home builds report0.1.0. Current checkpoint documentation and paired before/after scripts retain frozen paths and SHA-256 manually, but full-run automation should capture provenance itself. Record actual launch commands and resolved launchers, identify the loaded TS6 compiler payload/Node runtime and TS7 native executable rather than mistaking a JavaScript wrapper hash for a compiler hash, and hash the frozen Home executable. Detect changes between preflight/admission and measurement completion without putting hashing inside timed commands. Keep legacy reports explicitly distinguishable, add tests for symlink/wrapper paths and changed binaries, and retain invalid/partial runs rather than publish them as verified. Do not substitute self-reported version strings for artifact identity.

9 days ago

Checkpoint-local provenance.py now records both frozen Home binaries, Node runtime, TS6 launcher/shim/_tsc.js payload, TS7 launcher/wrapper/getExePath.js and the resolved native executable. All10 artifacts retain identical resolved paths and SHA-256 hashes across full measurement20260828T234350Z and confirmation20260828T234818Z. Evidence commonjs-query.hyU20d/provenance-before.json, provenance-after.json, provenance-confirmation.json. This is manual checkpoint evidence; the harness-wide automation requested here remains open. Docs707045c73: https://github.com/home-lang/home/blob/main/docs/docs/TS_PERFORMANCE.md#prepared-query-checkpoint-performance

9 days ago

Starting the automated provenance implementation from bf64a67 as a prerequisite for the Linux/container checkpoint #464. The design records requested and resolved launcher paths, TS 6 lib/_tsc.js plus the Node executable, the platform-specific native TS 7 lib/tsc payload, Home executable, Hyperfine executable, commands, sizes, SHA-256 identities, and runtime versions outside timed commands. It compares snapshots after admission and after timing. Result metadata starts in a non-reportable measuring state; incomplete or changed-artifact directories remain on disk but compare.py rejects them. Legacy reports remain readable and clearly lack the new provenance schema.

9 days ago

Completed on origin/main via 2e4bc05, with qualified result documentation in 5aa0234 and 924da0b. The harness now records requested and resolved launchers, the TS 6.0.3 lib/_tsc.js payload and Node runtime, the platform-native TS 7.0.2 executable, Home, Hyperfine, and Python; it hashes snapshots before admission and after measurement and rejects incomplete or changed provenance. Regression coverage includes symlink/wrapper resolution, payload selection, changed binaries, and non-reportable partial results. The final Linux ARM64 result 20260829T035150Z reports verified unchanged provenance across all 1,800 timing samples. Native TS 7 resolved to @typescript/typescript-linux-arm64/lib/tsc, not a wrapper, and Home SHA-256 is 9512c02505248c031beb75ab304e65bde9e018bf93d340adf56bd164a5eb31bb. The full 93-test harness suite passes.

Sign in to comment on this issue.