ReviewOS

home-lang/home

Benchmark checkpoints must preserve partial round samples

#880
Closed chris opened this 8 days ago ·
8 days ago

Track rigorous recovery planning for #416 after the landed-source matrix stopped at disk guard exit123. Original run 20261011T224925Z retains 150 admissions,124 complete five-image rounds and one partial reexport round with a successful Bun sample. No complete ranking is available.

Acceptance: validate complete selection, runs, rotating compiler order, source/native/tool/corpus provenance and original admission before producing a plan. Preserve original raw files unchanged. A planner may identify only unrecorded compiler slots; it must reject duplicate, reordered, invalid, non-prefix or changed-provenance data. Never rerun a recorded successful sample, synthesize timings, drop failed/partial evidence or turn a selected subset into a complete report. Execution/checkpoint lineage and statistical caveats require explicit evidence before any resumed matrix can be ranked. Guard floors/limits remain unchanged, and unrelated/shared data stays untouched.

First implementation is a pure tested checkpoint planner, not automatic resume or new timing. Then determine whether a no-repeat continuation can satisfy the full protocol; if not, keep the failed run unranked and document the limitation. Link commits and retain all raw evidence.

8 days ago

Implemented pure read-only checkpoint planner plus14 integrity tests; complete harness161/161 passes. Planner executes no compiler/timer and never mutates raw matrix data or qualifies a ranking. It validates schema5 admission hash/fullcoverage, exact current source/native/tool/corpus provenance snapshot, host/metadata, rotating order, contiguous complete-round prefix and onlyone final partial compiler prefix. Duplicate/reordered/invalid/failed/empty/non-prefix/changed-artifact evidence is rejected rather than discarded. Plans identify only unrecorded compiler slots and retain every existing round hash. Applying read-only to actual stopped matrix after recomputing unchanged measurement-runner provenance; no new timing/repeats or guard changes. Source uncommitted until actual-data plan verification.

8 days ago

Pure planner verified against the real stopped matrix with freshly recomputed identical provenance:124 complete rounds +one successful Bun sample=621 recorded successful samples,2379 unrecorded slots, original raw files unchanged, unranked. Complete harness161/161 and pickier/diff checks pass. Only checkpoint.py/test_checkpoint.py staged for a small conventional Chris/no-coauthor commit, but gitcommit failed writing worktree index with Out of diskspace. Tested source and all raw evidence/images remain preserved; no execution continuation or measurement repeats started. Task-owned generated cache cleanup reclaimed230254390 bytes and verified1145 protected hashes unchanged, but disk immediately collapsed again from1.3GB to135MB/ENOSPC due external state. Shared/unrelated data remains untouched. Publishing and runtime validation now require real capacity recovery.

8 days ago

Capacity recheck still about201MB, all verified task caches absent, no stale Git index lock, installed adoptedbb4c00dd image unchanged. No identical failed commit or compiler/timer retry. Read-only review found one legitimate integrity gap: symlinked evidence paths could leave the checkpoint. Added real-directory/file checks and three tests; complete harness now164/164 passes, pickier/diff checks clean. Existing161-result remains historical. New source is preserved atop staged planner but cannot publish under current ENOSPC condition; no measurements or continuation execution started. All partial raw matrix/621 recorded samples remain unchanged and unranked; external disk capacity is still needed beyond disposed task artifacts.

8 days ago

Disk recovered to7.1GB; tested read-only planner landed in9d713c01639eae8a14d2f9110dc2aee00407f1bd on origin/main asChris/no coauthors after164/164 harness. Concurrent runtime commit30ab0d71b was preserved by clean fast-forward before benchmark-only work. Added independent immutable-lineage verifier: every original metadata/admission/raw round hash and recorded sample must be retained; supplemental identities cannot overlap; assembled result dictionaries must exactly equal retained raw dictionaries; full standard matrix verifier still required. Changed selection/host/provenance, missing coverage, path escape/symlink, failed/invalid/raw alteration and replaced-original samples rejected. Complete harness177/177 passes, pickier/diff clean. No executor or new measurements yet; partial matrix remainsunranked. Verifier is a separate small issue-linked source chunk, not a shortcut to report partial data.

8 days ago

Small benchmark-integrity chunks now on origin/main as Chris/no coauthors: planner9d713c016, immutable-lineage verifier5072003e6, no-repeat executor eb912f9. Concurrent runtime code/docs30ab0d71b/302b02c preserved by non-overlapping rebase; no forced push. Complete harness186/186 passes with simulated timers: original partial compiler prefix never invoked, original bytes unchanged, mutated plans/provenance/host/paths rejected, invalid raw kept, repeated phase forbidden, untouched workloads keep original warmup count, final lineage plus standard complete-matrix verification required. Actual controller selects unchanged original measured runner/cwd/pins and exactbb4c00dd image, with a new output20261011T232901Z-resumed and explicit pause/partial-round lineage; no original files overwritten. Initialization under unchangedguard queued original12722 behind unrelated live compiler64411; no timing or missing slot started. Source matrix remains unranked until full600round/3000sample/150admission/provenance/lineage verification.

8 days ago

Authoritative original initialization12722 terminalexit123: refused beforestart at446MB vs unchanged1024MB admission. No outputdirectory, compiler or missingtiming started; all621originalsamples and bb4c00d frozen/installed hashes intact. Localstate still saysqueued because even tiny guardrecord apply_patch failed with disk exhaustion; it was not updated and no file was created. Currentdisk about116MB, taskgenerated caches absent, no stale Gitlock, unrelated guarded task67206 remains untouched. Treat original toolterminal as authoritative, do not pollclosed12722 or truststale queuedstate. No retries/guardoverride/ranking. Pure planner/lineage/executor186tested commits remain onoriginmain; publishing/runtime continuation now requires meaningful capacity recovery or explicitly authorized otherdisposable paths. Protectedsource/raw/retained/shared data unchanged.

8 days ago

Pushed 88b3fef to origin/main as Chris Breuer, no coauthors: ordinary reporting now verifies resumed raw lineage and discloses the measurement pause/split round; archive publication retains every original, supplemental, phase and warmup byte with checksums rather than dropping lineage or normalizing bytes covered by its hashes. All 191 harness tests pass, Pickier and diff checks pass. Task-generated .zig-cache directories remain absent; source/raw evidence/retained compiler hashes are preserved. Disk recovered to 4.9 GiB at admission attempt; initialization session94100 is queued on the unchanged shared machine lock, not yet running measurements. Original refused session12722 is recorded separately and never polled/restarted. No performance ranking or completion claim.

8 days ago

Documentation and complete interrupted checkpoint archive are now published on main (docs commit follows 88b3fef). README has an explicit 621/3000 incomplete/unranked table row linked to the detailed report. Archive SHA256 4dfadb5918eb6ad855d30918f8f59ff932e9bb3e46cf953fb134515924c067ca has all135 payload checksums independently verified, and every original checkpoint byte matches. Includes150 preflight admissions,125 original roundfiles (124 complete plus Bun-only partial), source/metadata, both guard terminals and continuation implementation/controller. No successful recorded sample rerun or selective ranking. Initialization94100 remains queued on the original shared lock; unrelated1800s build81275 is untouched. Source/raw/retained binaries preserved; task-generated Zig caches remain absent. Guard floors/limits unchanged.

8 days ago

Pushed 91e8a70 to origin/main as Chris Breuer, no coauthors.193/193 harness tests pass; Pickier and diff checks pass. Fairness hardening rejects removed/altered pause and original-checkpoint descriptors; presence of retained lineage independently triggers publication verification, so deleting the resumption marker cannot conceal interruption or bypass report/archive checks. Original checkpoint and native hashes remain protected. Existing initialization94100 is still live (guard pid83177), waiting behind unrelated81275. No timing started or restarted; unchanged guard retained.

8 days ago

Initialization94100 completed exit0 under unchanged guard (15MB peak,44% host low-water). Original provenance/host match; independent byte comparison confirms all125 original rawrounds,metadata and admission copied unchanged.621 successful recorded samples remain immutable;2379 missing slots planned. Reexport continuation guard40200 is now queued behind unrelated Program suite94022; its first segment contains onlyTS6/TS7/Home/Rust, not the already recorded Bun sample. No new timing yet or complete ranking. Prepared a sequential remaining-phase controller that admits each untouched workload under unchanged600s/3840MB guard, preserves terminal/log evidence, stops any refusal/failure and never retries an attempted phase.

8 days ago

Actual continuation progress: reexports, TSX, generic calls, control flow and type predicates completed with guardexit0. Snapshot270 top-level rounds retains1350 successful samples; every original125 rawhash stillmatches. Reexports first supplemental round contains exactlyTS6/TS7/Home/Rust and preserves original Bun raw, no repeats. Remaining-controller34167 is live. Correction: historical waitlog named debugbuild27061, but ps now confirms it finished; large-predicate guard28349 has acquired the lock and its phase marker is running. All guardlogs/terminalJSON and rawsegments remain underlineage;15 execution-context sources independently byte-copied. No partial ranking, failedphase retry, limit change or unrelatedjob/cache mutation. Full600round/3000sample lineage verification stillrequired.

8 days ago

Completed in bd0d82e on origin/main, authored as Chris Breuer with no coauthors. Full paused matrix:600 rounds/3000 successful samples/150 original admissions, exactunchanged provenance/host,621 original+2379 missing slots, zero repeated recorded samples. All16 phaseguards and finalizationexit0 under unchanged limits. Complete rawlineage and ordinaryprotocol verified; all1502 archivepayloads and retainedbytes verified.193 harness tests pass; every20 detailed/10 README row and ratio verified against raw medians. Home has1/20 lowestmedians and9/20 fasterTS7, descriptive only; pause/sharedhost/preexisting failures explicitlyretained, not causal/universalwins. README/fullreport/rawarchive published; original interrupted evidence remainsseparate and unranked. Checkpoint issue acceptance is achieved; #416 universalperformance/platforms and #548 Zod remain open.

Sign in to comment on this issue.