ReviewOS

home-lang/home

Native test CLI: honor test-name filters and stop discarding options

#481
Closed chris opened this 8 days ago ·
8 days ago

Status: implemented and verified on codex/runtime-port, pending integration

Published 1340cebd5, authored and committed by Chris Breuer without coauthor trailers. Not merged into main. The native entry point now calls the real Arguments parser and bunfig loader. The manual flag-skipping loop and preload extractor are removed. Fatal diagnostics flush and exit instead of panicking; direct/buffered output routing, newline handling, CI boolean detection, and process-title storage are corrected.

Verification

  • Native build and unchanged cached rebuild: 16/16 steps. All 20 native regression files pass, including the expanded execution-sensitive CLI suite. The prior installed binary fails that suite because its excluded poison body still executes. Shared CLI controls pass under Bun 1.3.14.
  • Original snapshot-filter reproduction: 3 pass / 31 filtered out / 0 fail / 7 assertions. Excluded test bodies do not execute.
  • Unchanged upstream bunfig suite: 5 pass / 17 assertions / 1 snapshot; path ignores: 10 pass / 38 assertions; retry/repeat hooks: 12 pass / 7 assertions; immediate concurrency: 2 pass / 8 assertions / 2 snapshots.
  • Ordinary concurrency, run from the original Bun checkout to preserve literal snapshot paths: 6 pass / 15 assertions / 3 snapshots. Full coverage: 12 pass / 27 assertions / 10 snapshots. No snapshot updates or relaxed assertions.
  • These seven native commands total 50 passing tests / 119 assertions. Filtered cases are not passes. Matching Bun controls for config, path ignores, coverage, and ordinary concurrency pass.
  • Focused Pickier, diff whitespace, and CLI/output Zig format checks pass. Existing enum-builtin formatter migrations in the two shared root/crash-handler files were not swept into this change.

Separate remaining work

The isolation failure exposed by this CLI work is implemented and verified in 6e092777a: #484 now passes 19/19 unchanged upstream tests / 67 assertions. Parallel execution #485 remains incomplete (0 pass / 3 fail / 24 filtered out in its original reproduction; Bun 3 pass). Fake-clock semantics #490 and deferred imports #491 are separate remaining work. The unchanged worker eval deadline remains #482. Full Bun parity is still incomplete under #66; parsing a flag is not proof its entire subsystem works.

Original reproduction and diagnosis

Follow-up to #66 and discovered while verifying the #471 snapshot work on codex/runtime-port.

HOME_NATIVE_VM=1 HOME_CORPUS_FULL_VM=1 ./zig-out/bin/home-debug test packages/runtime/test/bun-corpus/js/node/worker_threads/worker_threads.test.ts -t getHeapSnapshot runs all 34 tests instead of the three matching snapshot tests. The same command shape with Bun 1.3.14 runs 3 tests / 31 filtered out / 0 fail. All three snapshot tests pass in Home too, but the unfiltered Home file reports 33 pass / 1 timeout / 1 between-test error; this is not a green filtered run.

Root cause: src/main.zig's runTestsViaVM constructs a default Command.Context and manually copies a small subset of CLI options. Its loop skips every other dash-prefixed flag, leaving -t unparsed and its value as a positional. It never initializes ctx.test_options.test_filter_pattern or test_filter_regex. The existing native Arguments parser implements -t, --test-name-pattern, and --grep, but this entry point bypasses it.

Port the actual test-option semantics into the native Home test entry point rather than silently discarding flags. Cover aliases, regex matching including nested descriptions, inline/separate values, missing/invalid values, and coexistence with file targets and other supported options. Audit the other skipped test flags as part of that work. Regression controls must prove excluded test bodies do not execute; filtered-out cases are not completed parity.

The eval-source deadline failure discovered in the full file is separate from snapshot cancellation and filtering. Preserve its original workload and deadline in parity reporting.

The independent eval-source deadline is tracked in #482.

8 days ago

Published 1340cebd5 on codex/runtime-port, authored and committed as Chris Breuer without coauthors; not merged into main.

The native test command now uses the real argument and bunfig parsers. Filter aliases and regexes control actual body execution; invalid CLI/config input exits with diagnostics. The audit also fixed process-title storage, newline handling, CI boolean detection, and the buffered/direct writer routing required by unchanged output snapshots.

Final verification: build and cached rebuild 16/16, 20/20 native regression files, and 50 passing unchanged upstream tests / 119 assertions across seven native commands. The snapshot filter is now 3 pass / 31 filtered out / 0 fail. Full coverage is 12 pass / 10 snapshots, and ordinary concurrency 6 pass / 3 snapshots. No assertion, snapshot, workload, or original deadline was relaxed. Bun controls pass. Filtered tests are not counted as passed.

The broader audit exposed separate real failures: #484 for the parked isolated-global module-record bridge and #485 for the unimplemented parallel coordinator/worker IPC. Their unchanged selections fail in Home and pass in Bun. #482 still tracks the worker eval deadline. The bodies of #66 and #481 now reflect the published checkpoint and these limits. Full logical Bun parity remains incomplete.

8 days ago

Published isolation follow-up 6e092777a: #484 now passes all 19 unchanged tests / 67 assertions, including three repeat runs. All 20 native regression files, the 50-test / 119-assertion CLI set, and the 17/17-step native/cached builds remain green. Separate gaps: parallel #485, fake clocks #490, deferred imports #491, and worker eval timing #482. This does not establish full Bun parity under #66.

8 days ago

Integration status reconciled on current main: both the test-CLI implementation 1340cebd5dfa6534f9cc83dbc7bac7e674e646d8 and isolation follow-up 6e092777af9a3ea44907ba653e7cf42be3a000f0 are ancestors of ff9dc16be5503b066bf5accc3af1071ffd358094.

Fresh current-main validation:

  • tests/runtime/native-node-core-cli.test.mjs: pass, including filter aliases, excluded-body poison controls, invalid patterns/options, and isolation checks
  • no upstream test or snapshot was changed

The integration condition recorded in this issue is satisfied. Parallel coordination and other test-runner features remain separate trackers under #66.

Sign in to comment on this issue.