ReviewOS

home-lang/home

Native test parallelism: port the coordinator, worker IPC, and lifecycle

#485
Closed chris opened this 9 days ago ·
9 days ago

Follow-up to #66 and native test CLI option integration in #481. Parsing --parallel now exposes the parked native implementation rather than silently running serial tests.

Reproduction

HOME_NATIVE_VM=1 HOME_CORPUS_FULL_VM=1 ./zig-out/bin/home-debug test packages/runtime/test/bun-corpus/cli/test/parallel.test.ts -t 'unique JEST_WORKER_ID|runs files across workers and aggregates|forwards -t to workers'

The byte-identical upstream selection reports 0 pass / 3 fail / 24 filtered out. Child test commands return ParallelTestCoordinatorNotImplemented from packages/runtime/src/runtime/cli/test/ParallelRunner.zig. Its coordinator/worker entry points, event reporting, and Worker exit callbacks are still parked implementations even though the real parallel/ sources are present.

Bun 1.3.14 passes the same selection: 3 pass / 24 filtered out / 0 fail / 1 snapshot / 24 assertions. Source pin: 4982b91e3702094330f3be3883354c52b8c01323.

Required work

Port the real coordinator and worker IPC, subprocess ownership and exit handling, worker identifiers, test-name filter forwarding, aggregation, bail, output ordering, coverage, and cleanup. Validate the complete upstream parallel suite, not only the three selected cases. Native child commands must remain Home and execute their assigned test bodies; no serial substitute, fabricated worker output, or skipped workload.

This depends on working isolated test execution (#484). Generic worker task reclamation (#465) and producer quiescence (#468) remain separate lifecycle work.

Reproduced during the published CLI checkpoint 1340cebd5. The CLI integration is complete on the branch; this subsystem is still failing and open. See #481 for the complete verification boundary.

9 days ago

Completed and published on main in 8c56f61c6 (feat(runtime): activate native parallel test workers).

The parked facade now exports the existing pinned coordinator, worker, and fd 3 channel implementation. Native home test --parallel uses real isolated Home subprocesses, unique JEST_WORKER_ID / BUN_TEST_WORKER_ID values, directory-local partitioning and work stealing, lazy scale-up, test-filter/runtime-option forwarding, live result frames, atomic console output, bail/crash accounting, snapshots, JUnit XML, LCOV coverage, and worker exit ownership. Current Zig signal, writer, sentinel-path, and coverage types are adapted without changing the wire protocol or corpus expectations.

The complete unchanged pinned test/cli/test/parallel.test.ts initially improved from the parked 0/3 reproduction to 25/27. The two remaining cases exposed real integration defects:

  • The 68,000,000-character result line spent the unchanged 60-second inner deadline repeatedly shifting the remaining 64 MiB IPC buffer. The channel now advances an unsent offset, making backpressure linear while retaining the pinned 64 MiB limit and truncation marker.
  • SIGTERM woke neither an idle coordinator loop nor a real worker process group. The signal handler now preserves errno and wakes the existing loop; Home's POSIX spawn adapter now maps new_process_group to POSIX_SPAWN_SETPGROUP, so the coordinator's group termination reaches workers and their grandchildren.

Final verification on Darwin:

  • Complete unchanged parallel file: 27 pass / 0 fail / 185 assertions / 1 snapshot.
  • Original issue selection: 3 pass / 24 filtered out / 24 assertions / 1 snapshot. Filtered cases are excluded from pass counts.
  • Permanent native CLI regression proves two files execute in two distinct Home subprocesses with two distinct worker IDs.
  • Native node-core CLI regression: pass.
  • Focused home_rt: 1,815 pass / 19 skip / 0 fail, 20/20 build steps.
  • JSC-enabled build: 18/18; JSC-disabled build: 5/5.
  • The unchanged parallel file is byte-identical to Bun pin 4982b91e3702094330f3be3883354c52b8c01323; scoped Pickier, Zig formatting, and whitespace checks pass.

The repository-wide umbrella test is not claimed green: this run reported an unrelated parser allocation leak and was interrupted after its remaining steps stayed silent; the previously known unrelated TypeScript conformance baseline failures also remain outside this checkpoint. Linux/Windows runtime verification, generic worker teardown issues, and the full Bun-suite ledger remain under #66.

Sign in to comment on this issue.