ReviewOS

also looking at this

zig-utils/zig-test-framework

Unify discovery and programmatic test execution

#23
Merged glennmichael123 wants to merge codex/issue-5-unified-test-pipeline into main
12 files +656 -169

Review threads live on the whole diff, not on one commit, so none are shown here - a thread's line means something in the branch's final form, and painting it into an intermediate step would put it on code it is not about.

README.mdmodified+12-5
Changes to README.md
@@ -95,15 +95,20 @@ This will automatically discover and run all `*.test.zig` files in the `tests` d
9595- `--no-recursive` - Disable recursive directory search
9696- `--bail` - Stop on first failure
9797- `--filter <text>` / `--grep <text>` - Run only Zig tests whose names contain the text
98- `--reporter <name>` - Use spec, dot, JSON, TAP, or JUnit output
99- `--junit-output <file>` - Set the JUnit XML path
100- `--timeout <ms>` - Mark files that exceed the global time budget as failed
98101- `--shard-index <N>` - Run one one-based file shard
99102- `--shard-count <N>` - Set the total number of file shards
100103- `--no-color` - Disable color in child Zig test processes
101104- `--verbose` - Show detailed output
102105
103Discovery mode rejects options that it cannot honor yet instead of silently
104ignoring them. Reporter selection, quiet output, timeouts, parallel jobs,
105snapshot updates, memory profiling, JUnit output, and the web UI require the
106shared execution pipeline planned for future releases.
106Discovery and programmatic execution share the same test-plan, lifecycle,
107result, and reporter model. Discovery still rejects options it cannot honor
108instead of silently ignoring them: quiet output, parallel jobs, snapshot
109updates, memory profiling, and the web UI remain programmatic-only. See
110[the execution pipeline architecture](docs/execution-pipeline.md) for extension
111points, timeout semantics, and migration guidance.
107112
108113**Examples:**
109114
@@ -634,7 +639,9 @@ zig-test --no-color
634639| `--version` | `-v` | Show version information |
635640| `--bail` | `-b` | Stop on first failure |
636641| `--filter <pattern>` | | Run only tests matching pattern |
637| `--reporter <name>` | | Set reporter (spec, dot, json) |
642| `--reporter <name>` | | Set reporter (spec, dot, json, tap, junit) |
643| `--junit-output <file>` | | Set the JUnit XML output path |
644| `--timeout <ms>` | | Set the global test timeout |
638645| `--verbose` | | Enable verbose output |
639646| `--quiet` | `-q` | Minimal output |
640647| `--no-color` | | Disable colored output |
docs/execution-pipeline.mdadded+62-0
Changes to docs/execution-pipeline.md
@@ -0,0 +1,62 @@
1# Unified execution pipeline
2
3The framework has two plan producers and two executor implementations, but one
4execution contract.
5
61. Programmatic registration or file discovery produces a `TestPlan`.
72. A registered-test or external-process executor consumes that plan.
83. The executor emits typed `LifecycleEvent` values through `EventStream`.
94. A selected reporter consumes the events and a shared `TestResults` summary.
10
11`PlanItem.registered` retains the suite and test-case pointers needed for hooks
12and in-process execution. `PlanItem.discovered` retains the discovered file used
13by the external `zig test` executor. `PlanItem.identity()` gives extensions a
14common stable identity without erasing executor-specific information.
15
16## Shared lifecycle and results
17
18Both executors emit run, suite, and test start/end events. Discovered files are
19represented as one synthetic suite and test result per external process. This
20means spec, dot, JSON, TAP, and JUnit reporters receive the same lifecycle and
21summary model in either mode. `ReporterSet` is the single factory for built-in
22reporters, including the configured JUnit output path.
23
24`ExecutionPolicy` owns common filtering, bail, and timeout decisions:
25
26- Registered tests match filters directly by test name. Discovery forwards the
27 same filter to `zig test`, where Zig applies it to names inside each file.
28- Bail stops scheduling after the shared result model records a failure. Work
29 already started by a parallel batch cannot be cancelled.
30- A global timeout compares elapsed execution time with the same policy in both
31 executors. It marks an over-budget result failed after control returns; it
32 does not yet terminate an in-process function or external child process.
33- Summaries always come from `TestResults`, rather than executor-local counters.
34
35## Extension points
36
37New plan producers should append `PlanItem` values. New executors should emit
38`LifecycleEvent` values and update `TestResults`. New reporters can continue to
39implement `Reporter.VTable`; `EventStream` adapts the typed lifecycle to that
40stable callback interface.
41
42The event types are intentionally internal Zig structures. The separately
43versioned JSON wire protocol is built on this layer rather than making this API
44depend on one serialization format.
45
46## Migration
47
48Existing programmatic APIs remain source compatible:
49
50- `TestRunner.init`, `runTests`, and `runTestsWithOptions` are unchanged.
51- `RunnerOptions` only gains optional `junit_output`, `timeout_ms`, and
52 `reporter_writer` fields.
53- `runDiscoveredTests` and `LoaderOptions` remain available; optional reporter,
54 JUnit path, timeout, and writer fields default to the previous spec behavior.
55
56CLI hosts can inject a stdout writer with `reporter_writer`. Embedded callers
57may leave it unset; reporter output then uses stderr so it cannot corrupt Zig's
58stdout-based test protocol.
59
60Code that implements a custom execution path can migrate incrementally by first
61building a `TestPlan`, then routing callbacks through `EventStream`. Existing
62custom reporters require no migration.