ReviewOS

home-lang/home

Replace adapter corpus coverage with faithful native discovery and execution

#703
Open chris opened this 8 days ago ·
8 days ago

Parent: #66. Related: #202, #702. Logical parity is incomplete; an adapter-generated green count must not be treated as suite completion.

Current publication (2026-09-08)

The native-only discovery/dispatch remains in place: 4,754 original pre-exclusion entries, 1,863 test-runner and 2,891 script modes, with synthetic corpus rewriting/bootstrap fallback removed. Inventory is not passing coverage.

The launch contract is now ported in fcd8ccb97 and corrected in 0785d795a, with final evidence 07b32b554, all pushed to origin/main as Chris Breuer without coauthors. 23/23 native/harness build steps and 9/9 harness tests pass. Five focused launch units pass. An independent selector comparison matches 19,016 rows across all 4,754 paths and four CI/local/ASAN contexts against pinned runner functions. It does not execute those corpus files. Baseline, final evidence.

The real launcher now selects original ordinary/integration test deadlines and per-test defaults, separate Node deadlines and script bounds, applies CI GC/integrity/cache settings and validation rules, creates per-file temp/cache storage, and routes command aliases to Home. Native child/cleanup tests and a paired Home/Bun cache control verify allocation ownership, initially empty caches and actual child execution. The initial 6/9 harness result (three allocator crashes) and Home cache failure remain preserved; both causes are fixed and retested.

Grouped selected originals: Home 71 pass / 1 fail; pinned Bun 72 pass, plus two successful Node process checks and one comment-only file with no passing cases. Seven native private files pass. Original sources/snapshots remain unchanged. The only selected original failure is the known impossible audit snapshot edge in #721, still reported as failure.

Remaining durable selection/start/completion, native JUnit retention and separate skip/TODO outcomes are split into #722. Full intended CI execution, service/vendor setup, platform/ASAN execution, default CLI ownership and independent native builds remain unfinished under #66/#709/#202. This issue remains open. Historical selected results and lost-log caveats remain in #66; no earlier interrupted aggregate is reclassified as complete.

Historical discovery and dispatch gap (corrected in the native-only migration)

A read-only audit during #702 extracted Home's exact production classification/routing/mode predicates into a standalone Zig program and ran them over the unchanged tracked manifest. In the HTML candidate based on main f491cacd7ae34d4c8f49c1fb02464699e073afa6, 4,708 classified files comprise 175 native test-runner files, 266 native script files and 4,267 bootstrap-adapter files. These are route counts, not passing-test evidence. Published main before the five-file HTML activation has five fewer native test-runner routes.

Separately, the exact getTests, isTest, isNodeTest, isClusterTest, JavaScript and hidden-path predicates extracted from scripts/runner.node.mjs at Bun pin 4982b91e3702094330f3be3883354c52b8c01323 discover 4,754 tracked paths in the mirror, with zero untracked discovered paths, under local Darwin arm64 context. This is discovery before expectations, include/exclude filters, sharding, integration setup and vendor selection, not a final CI execution count.

Compared with Home's manifest classifier:

  • 78 upstream-discovered paths are absent: 73 in js/bun/test/parallel, two TypeScript cluster scripts, js/bun/http/http-spec.ts, and two .test.mts files in Node HTTP/HTTP2.
  • 32 Home-only entries include helper/negative fixtures, standalone N-API children and jsonwebtoken/test-utils.js. Their coverage must be attributed through their original owning tests; indiscriminately treating them as independent successful files is incorrect.
  • Four current native mode differences: test-buffer-isencoding.js and test-buffer-tojson.js use Home's test runner where upstream uses run; test-module-isBuiltin.js and test-url-is-url-internal.js use Home's script mode where upstream uses test.
  • A single native recursive test command is not sufficient: its scanner's suffix rules differ from the CI runner's Node-script discovery and mode selection.

Required work

  1. Establish a reproducible pinned upstream execution inventory, including modes, original config/startup environment, integration/vendor scope, platform applicability and original negative-fixture ownership. Preserve source bytes, workloads, deadlines and the distinction between upstream skips and implementation gaps.
  2. Correct Home's missing/extra discovery and native mode/context mismatches. Verify negative controls so omitted tests, zero-test runs, process failures and helper fixtures cannot produce false passing coverage.
  3. Migrate the remaining corpus to actual Home runtime and actual child-process execution in substantial batches. Remove semantic substitutes and fallback successes; do not preserve an adapter's historical green status at the expense of real behavior. Truthfully report newly exposed failures and split missing features into linked implementation issues.
  4. Remove the bootstrap corpus fallback once native execution owns the intended inventory. Audit remaining synthetic outputs/RSS and filename-driven behavior rather than merely bypassing them in a small whitelist.
  5. Run the intended complete upstream suite through Home and retain per-file/test results, platform skips, failures and source-integrity evidence. Native build ownership and platform completeness remain separate required work under #66.

Local audit evidence: /tmp/home-702-routing-inventory.zig, /tmp/home-702-routing-inventory.log, /tmp/home-702-routing-counts.json, /tmp/home-702-upstream-discovery.mjs, /tmp/home-702-upstream-discovery.json, /tmp/home-702-discovery-comparison.json. The audit executed only discovery/predicates; it did not execute or claim passes for the 4,754 files.

Further source inspection: the upstream Node runner passes bunfig.node-test.toml, which preloads both test/js/node/harness.ts and test/preload.ts and disables auto-install. That config was absent before this migration and is now restored from the pinned upstream Git object. Mode differences require semantic review, not a blind table update: upstream's textual node:test check also matches the module-name assertion in test-module-isBuiltin.js and a comment in test-url-is-url-internal.js. The latter file's entire test body is commented out and must receive no implemented-feature credit. Preserve actual assertions and report zero-body/zero-test cases truthfully while reconciling context.

8 days ago

The concurrently integrated scripts/vm-corpus-scan.sh is a triage tool, not completion evidence for #66/#703. It selects only *.test.* executable suffixes, invokes every file in test-runner mode, and automatically re-runs timeout-only failures with --timeout 60000. Those runs are labelled slow, not passes, but its comment that this is "not a parity gap" is not justified for the requested unchanged-deadline acceptance criterion. Its resource-bound confirmations also overwrite the first run log/status.

The faithful native migration must preserve original per-test deadlines and retain initial outcomes. Timeout overrides or a diagnostic success must not erase an original failure or establish logical parity. The final inventory must also preserve script modes, owning fixtures and original Node configuration. This scanner is not being used as an acceptance gate for the current HTML batch; the ordinary route checks retain the original test deadlines and report any observed failures.

8 days ago

Published 6448d628a on origin/main as Chris Breuer without coauthors: the triage scanner no longer overrides original per-test deadlines or retries timeout-only failures to reclassify them as slow. A deliberately timed-out fixture under pinned Bun ran exactly once and remained fail; this is scanner validation, not Home feature coverage. Shell syntax, scoped lint and whitespace checks pass.

The earlier source audit remains valid: the scanner's discovery/mode selection is still incomplete, resource-bound confirmations still need full first-outcome retention, and the bootstrap fallback still owns 4,267 classified files. #703 and #66 remain open. The integrated HTML-native build and its final Home verification are still running under #702.

8 days ago

Published native-only corpus migration: 1d62a395b3cb1f467e84c3d2b295d33b5f92f417 on origin/main, as Chris Breuer without coauthors. It includes the shared runtime fixes through 5e9a3a049.

  • Removed the corpus's synthetic JavaScript prelude, source-rewrite engine, fixture-specific outputs and adapter-only test infrastructure (about 160,000 lines). Every selected original file now executes in Home's production runtime; the former 4,267-file bootstrap fallback is gone.
  • Discovery matches 4,754 pinned pre-exclusion CI entries: 1,863 test-runner entries and 2,891 scripts, with zero missing/extra paths or mode differences. A deeper check of spawnBunTest corrected the two cluster entries to script mode. The audit separately identifies 23 comment-only sources, which receive no passing-feature credit.
  • Restored the exact upstream Node config, preserved explicit CLI config, retained real native execution/workload matrices, and removed the old N-API concurrency override. Added real child-failure, Node-assertion, static-ESM and inventory controls.
  • Standalone discovery tests pass 11/11. The optimized build and focused native controls are now running; new native runtime pass counts are not yet claimed. The earlier compiler processes were superseded to correct the cluster modes before testing the final source.

The full CI execution profile remains explicit work: Bun's launcher sets its own test/file deadlines and reporter options, while ordinary Home test deadlines remain unchanged; native capture still has its existing 120-second outer file bound. CI/service/vendor setup, complete outcome retention, broad native feature failures, other platforms and independent native build ownership remain unfinished. #66, #202 and #703 remain open. The previous historical adapter green result is not valid completion evidence.

Audit: scripts/audit-bun-corpus-discovery.py, docs/docs/bun-corpus-discovery-audit.json; the previous inventory is preserved in docs/docs/bun-corpus-discovery-audit-before-native.json. Durable build/test logs: zig-out/bun-port-703/.

8 days ago

Native-only verification update for #703 / #66:

Published config correction 82156e60d as Chris Breuer, without coauthors. The initial native-only production build passed 46/46 steps, but the focused harness was 5/6, not green. Its Node assertion control exposed an incorrect split --config path argument. The launcher now emits the exact pinned CI form --config=path; direct Home and pinned Bun controls reach and fail the intended Node assertion. The corrected optimized and focused builds are running.

Retained ordinary native routes passed: HTML 145/one original skip, Blob 65, body/stream 9,467/four original skips, Headers/Response 170, Request 24, FormData boundary 5, microtasks 2. A separate HTML native aggregate matched 530 assertions and 145/one skip. Source audit: 12,990 regular files and five symlinks match pinned bytes/modes; only existing expectations metadata differs. Scoped lint and format checks pass.

All 4,754 discovered entries still route natively with zero path/mode differences. The comparison plan now retains each individual Home and pinned Bun result for all 78 formerly missed files; it runs after the corrected build completes. This is an ordinary execution baseline, not full CI launch-profile equivalence. Inactive Docker guards receive no Docker coverage credit. Full parity, #701/#694, CI profiles, outcome accounting, service/vendor/platform setup and independent build ownership remain unfinished.

8 days ago

Progress under #703 / #66:

The config-fixed runtime 82156e60d passed 46/46 production steps and all six focused controls. Its complete 78-file comparison retained 156 first-attempt logs: 69 files exit zero in both runtimes, four Docker guards are inactive, four fail only in Home, and one certificate-expiry failure occurs in both. The per-file baseline and log hashes are published in docs/docs/bun-native-newly-discovered-baseline.json.

The Home accounting/extension corrections are published in 1e2f39847: manual-assertion process checks are separate from registered test counts, failed processes retain their actual registered results, original TAP skips/comment-only files have separate counters, and the CLI accepts every discovered JavaScript/TypeScript extension. The audit now checks the CLI's actual target classifier too; all 4,754 entries classify as files. Corrected native verification is still pending.

A host restart interrupted the first build. Two later supervised attempts hit false tree-memory limits: a real two-process probe showed the guard was adding the aggregate Summary Footprint to the individual process totals. Fixed in ab44979a9 on main; regression tests and live positive/negative controls pass. The default limit and host-pressure checks remain enabled. Production and focused builds are now running sequentially with correct measurement. No test deadline was extended, and no incomplete build is reported as passing.

The missing native create command is tracked in #709; its initial native activation and filesystem/process API port are prepared in an isolated checkout, pending compilation and integration. The shared expired TLS fixture is tracked in #710; its original validation and pinned bytes remain intact. All completed publications are as Chris Breuer without coauthors. Full Bun parity remains incomplete.

8 days ago

Pushed 66a6d6e444ed9c6be2817f1b2901b67aacb06f4b to origin/main as Chris Breuer, without coauthors. Production build passed 46/46 steps with the complete native create command activated (measured peak tree footprint 13,319 MB). Runtime SHA256: 435b57f0b492e01ef40deb0254fdf79b0ea6409b45eeb7ff077714189933d743.

The original cli/install/bun-create.test.ts reports 13 passed, 0 failed. The unchanged rsbuild integration now completes its real native create, 89-package install, production build, and all assertions, with no system Bun delegation. It is correctly reported as one process check, zero registered test passes. The original HTTP batch passes 29 registered tests plus the separate http-spec.ts process check, whose original output confirms 40/40 custom checks.

The negative accounting control initially lacked its private tracked-file manifest and therefore did not reach execution. That invalid setup outcome is retained. After adding the control manifest, Home correctly exits 1 with the real Node assertion diagnostic, zero registered passes, one successful process check, one failed file, one comment-only file. No pinned test source changed.

JSX is still failing. Home's first original JSX file run is 1 pass / 12 failures: leading --only-missing caused install to be treated as a script. A general native dependency-command classification fix for leading flags is now rebuilding together with the six focused corpus harness tests. The pinned Bun JSX control also failed (3 pass / 10 fail), from snapshot differences and original 5-second deadline failures; those do not excuse the Home dispatch defect. No snapshots or deadlines were changed.

Evidence is retained under zig-out/bun-port-709/, and a compact per-result report with hashes is prepared at docs/docs/bun-native-create-accounting-baseline.json. The ordinary baseline is not full upstream CI-profile equivalence or 4,754-file coverage. Successful child stderr retention is still incomplete, so the template gate's 13 registered passes alone cannot independently exclude its existing GitHub rate-limit early return. Explicit API-doc generation also remains to be verified. #66, #703, #709 and #438 remain open.

8 days ago

Everything from this batch is committed and pushed to origin/main through 9e1656f7c3cd4b76200819e88a630f09b028e5c7, as Chris Breuer without coauthors. The newer native HTTP Request change was merged without dropping work.

Verified runtime remains 66a6d6e444ed9c6be2817f1b2901b67aacb06f4b, SHA256 435b57f0b492e01ef40deb0254fdf79b0ea6409b45eeb7ff077714189933d743:

  • Production build: 46/46 steps.
  • Original native create templates: 13 registered passes; unchanged rsbuild create/install/build succeeds as one real process check.
  • Original HTTP accounting cases: 29 registered passes plus the script's actual 40/40 manual checks. The corrected private negative control fails correctly with its real Node assertion.
  • Retained regressions: 9,878 registered passes, five original skips/todos, zero unexpected failures across HTML, Blob, body/stream, Headers/Response, Request, FormData boundary and microtasks. The alternate whole-VM HTML route also passes its same 145 cases plus one original skip; it is not additional distinct coverage.
  • Discovery audit: all 4,754 pinned entries match, with no missing, extra, CLI-routing or native-mode differences. Integrity audit still finds 12,990 identical regular files and five identical symlinks; only the known expectations metadata differs.

Newer source is pending full runtime verification. 62d020e94 fixes leading native dependency flags. 6775d0ba9 streams each completed file's stdout, stderr and process status for every CLI gate, avoiding discarded successful stderr and repeated aggregate-buffer copying; its regression extends the existing focused harness with success-after-failure output and sink-error checks. Syntax, formatting and discovery checks passed. No claim that those new runtime tests have run yet.

The compiler was stopped through its supervisor when disk space fell below 350 MB; the volume then filled completely and briefly prevented Git writes. Only this task's inactive generated artifacts were reclaimed. 643c20934 now guards actual working/temp-volume free space; all six live disk/scanner controls pass, including refusal, cleanup, measurement failure and distinguishing a child's own exit 123. Disk stops remain incomplete outcomes, never feature passes.

The latest full build now safely refuses admission: 1,464 MB free, 8,192 MB required, exit 123 (zig-out/bun-port-709/final-full-build-admission.log). The user has been asked to free at least 10 GB. Once space is available, the next batch is the full rebuild and six focused harness tests, offline leading-flag dependency controls, original JSX tests, and explicit API-doc generation. The earlier JSX results remain Home 1 pass / 12 fail and Bun 3 pass / 10 fail; snapshots and original deadlines remain unchanged. Full CI-profile, platform and 100% suite parity remain unfinished.

Compact verified evidence and hashes: docs/docs/bun-native-create-accounting-baseline.json. Detailed local logs and continuation state are under zig-out/bun-port-709/. #66, #703, #709 and #438 stay open.

8 days ago

Verified the production fix at 6053c6c. Build: 46/46 steps. Home executable SHA-256: fbc3cfe5af5e1c020e4aad2e6d4bae74584afbd87de970b1134e3ac653da9e15.

Native frozen prerelease regression passes all release/short-tag/long-tag/build-tag cases, with real dependency-removal rejection controls. Native create regression also passes. The original semver corpus reports 25 registered passes and zero failures.

The exact native frozen corpus provisioning succeeded. Both pinned lockfiles remain byte-identical. All 5,945 installed AWS CDK 2.148.0 regular files match the verified pinned tarball, with zero missing or different files. The subsequent corpus source audit retains 12,996 tracked entries: 12,990 identical regular files, five identical symlinks, and only the known expectations.txt metadata difference; no missing or mode differences.

Full logs, first failures, hashes, and the updated baseline are pushed in 65d0e916e. This completed the existing partially populated dependency tree; it is not a cold-install benchmark. The first pinned-Bun AWS extraction failure remains retained and is not replaced by Home's successful outcome.

Closing #714 and #715 on these concrete results. Full parity remains open: #713 local-dependency duplication, #438 JSX pipelines, full inventory execution and upstream CI/platform coverage, and independent native build ownership still need work. All changes are on origin/main as Chris, without coauthors.

8 days ago

Audit batch checkpoint 3ff5c660a is pushed, with verified native runtime 952744932. Next broad work remains the real corpus launch profile.

Revalidated against pinned 4982b91e3702094330f3be3883354c52b8c01323 runner.node.mjs:

  • Ordinary Bun test files: 180 s; integration/3rd_party/docker/bun-install-registry/bun-security-scanner-matrix/v8/bundler_compile/tonic/test[native separator]napi: 300 s. CLI per-test default is half the file bound. ASAN multiplies file by 2 and per-test by 3. Non-test scripts: 30 s.
  • The separate Node runner uses 20 s in CI, 60 s for test-dns, test-cluster-, -docker-, test-stdin-pipe-large, local runs, or an ASAN step. It does not use spawnBunTest's CLI reporter/timeout arguments. Preserve this distinction.
  • Fresh per-file TMPDIR/BUN_TMPDIR/TEST_TMPDIR/install cache; GC 1, integrity audit 1.0, transpiler cache disabled, real executable directory/alias on PATH, internal bindings and quiet logs. Ordinary files force color and GitHub annotations; Node files disable color. Windows has separate TEMP/Path handling.
  • ASAN/local exception/leak validation respects the original skip lists and suppression file. ASAN ordinary test children also use no-orphans. Full outcomes need retained selection/start/completion and exact script/test/skip/todo accounting.

Current Home capture still has a shared 120 s bound and conditional GC 0 default. This is a confirmed launch-contract gap, not a proposal to extend failing test deadlines. No new launch code or full-inventory pass is claimed. Full CI launch and execution remain open in #703, with parent #66 and #709/#202. Original assertion deadlines, snapshots and source bytes must remain intact.

8 days ago

Pushed fcd8ccb97 as Chris Breuer without coauthors. The production corpus now passes a per-file launch contract to the native capture layer: ordinary/integration test deadlines and CLI defaults, the separate Node CI deadline path, 30-second scripts, GC 1/integrity audit 1.0, disabled transpiler cache, isolated per-file temp/install cache, Node/ordinary color distinctions, validation skip-list handling, and real executable aliases for nested commands. Timeout diagnostics retain the selected bound. Temporary storage is cleaned on both successful and failed launch paths.

Four focused launch unit tests pass. A real native regression now checks startup settings, both command aliases' actual child execution, distinct storage across files, and cleanup preserving the project. The native/harness build is running; no passing production artifact is claimed yet. A separate pinned-runner comparison across the complete discovery inventory and CI/local/ASAN contexts is being prepared. Full inventory execution, durable case-level outcomes/JUnit, integration/service/platform coverage and independent build ownership remain unfinished. Refs #66, #709, #202.

Build storage maintenance preserved source archives and inventories before removing stale generated create fixtures and obsolete/aged compiler cache outputs. No source changes, installed binaries or active compiler outputs were deleted.

8 days ago

Pushed corrections 0785d795a and evidence 1f1c9423d as Chris Breuer without coauthors. All 19,016 launch-rule selections match pinned Bun across 4,754 discovered paths and four CI/local/ASAN contexts. This verifies rule selection, not passing corpus files. Five focused launch units pass.

The first full build compiled Home but exposed three allocator crashes in nine harness tests (six passed; 21/23 build steps): a sentinel-terminated realpath allocation was returned as a plain slice and freed with the wrong length. A separate real generated control also found aliases inside the nominally empty install cache (Home fails, pinned Bun passes). Both failures are retained. The fix preserves the allocation's ownership and keeps alias storage beside the fresh cache. A native regression now explicitly checks cache emptiness as well as actual aliases, startup settings and cleanup.

The corrected native/harness build is currently running. Grouped original-file and private regressions are prepared and await that result; no corrected production pass or whole-suite completion is claimed. Evidence manifest. Refs #66, #703, #709, #202. Full corpus execution, durable case/JUnit accounting, integration/platform coverage and independent build ownership remain unfinished.

Sign in to comment on this issue.