ReviewOS

also looking at this

zig-utils/zig-test-framework

Unify discovery and programmatic test execution

#23
Merged glennmichael123 wants to merge codex/issue-5-unified-test-pipeline into main
12 files +656 -169
docs/execution-pipeline.mdadded+62-0
Changes to docs/execution-pipeline.md
@@ -0,0 +1,62 @@
1# Unified execution pipeline
2
3The framework has two plan producers and two executor implementations, but one
4execution contract.
5
61. Programmatic registration or file discovery produces a `TestPlan`.
72. A registered-test or external-process executor consumes that plan.
83. The executor emits typed `LifecycleEvent` values through `EventStream`.
94. A selected reporter consumes the events and a shared `TestResults` summary.
10
11`PlanItem.registered` retains the suite and test-case pointers needed for hooks
12and in-process execution. `PlanItem.discovered` retains the discovered file used
13by the external `zig test` executor. `PlanItem.identity()` gives extensions a
14common stable identity without erasing executor-specific information.
15
16## Shared lifecycle and results
17
18Both executors emit run, suite, and test start/end events. Discovered files are
19represented as one synthetic suite and test result per external process. This
20means spec, dot, JSON, TAP, and JUnit reporters receive the same lifecycle and
21summary model in either mode. `ReporterSet` is the single factory for built-in
22reporters, including the configured JUnit output path.
23
24`ExecutionPolicy` owns common filtering, bail, and timeout decisions:
25
26- Registered tests match filters directly by test name. Discovery forwards the
27 same filter to `zig test`, where Zig applies it to names inside each file.
28- Bail stops scheduling after the shared result model records a failure. Work
29 already started by a parallel batch cannot be cancelled.
30- A global timeout compares elapsed execution time with the same policy in both
31 executors. It marks an over-budget result failed after control returns; it
32 does not yet terminate an in-process function or external child process.
33- Summaries always come from `TestResults`, rather than executor-local counters.
34
35## Extension points
36
37New plan producers should append `PlanItem` values. New executors should emit
38`LifecycleEvent` values and update `TestResults`. New reporters can continue to
39implement `Reporter.VTable`; `EventStream` adapts the typed lifecycle to that
40stable callback interface.
41
42The event types are intentionally internal Zig structures. The separately
43versioned JSON wire protocol is built on this layer rather than making this API
44depend on one serialization format.
45
46## Migration
47
48Existing programmatic APIs remain source compatible:
49
50- `TestRunner.init`, `runTests`, and `runTestsWithOptions` are unchanged.
51- `RunnerOptions` only gains optional `junit_output`, `timeout_ms`, and
52 `reporter_writer` fields.
53- `runDiscoveredTests` and `LoaderOptions` remain available; optional reporter,
54 JUnit path, timeout, and writer fields default to the previous spec behavior.
55
56CLI hosts can inject a stdout writer with `reporter_writer`. Embedded callers
57may leave it unset; reporter output then uses stderr so it cannot corrupt Zig's
58stdout-based test protocol.
59
60Code that implements a custom execution path can migrate incrementally by first
61building a `TestPlan`, then routing callbacks through `EventStream`. Existing
62custom reporters require no migration.