also looking at this
Unify discovery and programmatic test execution
#23
12 files
+656
-169
Review threads live on the whole diff, not on one commit, so none are shown here - a thread's line means something in the branch's final form, and painting it into an intermediate step would put it on code it is not about.
| @@ -95,15 +95,20 @@ This will automatically discover and run all `*.test.zig` files in the `tests` d | ||
| 95 | 95 | - `--no-recursive` - Disable recursive directory search |
| 96 | 96 | - `--bail` - Stop on first failure |
| 97 | 97 | - `--filter <text>` / `--grep <text>` - Run only Zig tests whose names contain the text |
| 98 | - `--reporter <name>` - Use spec, dot, JSON, TAP, or JUnit output | |
| 99 | - `--junit-output <file>` - Set the JUnit XML path | |
| 100 | - `--timeout <ms>` - Mark files that exceed the global time budget as failed | |
| 98 | 101 | - `--shard-index <N>` - Run one one-based file shard |
| 99 | 102 | - `--shard-count <N>` - Set the total number of file shards |
| 100 | 103 | - `--no-color` - Disable color in child Zig test processes |
| 101 | 104 | - `--verbose` - Show detailed output |
| 102 | 105 | |
| 103 | Discovery mode rejects options that it cannot honor yet instead of silently | |
| 104 | ignoring them. Reporter selection, quiet output, timeouts, parallel jobs, | |
| 105 | snapshot updates, memory profiling, JUnit output, and the web UI require the | |
| 106 | shared execution pipeline planned for future releases. | |
| 106 | Discovery and programmatic execution share the same test-plan, lifecycle, | |
| 107 | result, and reporter model. Discovery still rejects options it cannot honor | |
| 108 | instead of silently ignoring them: quiet output, parallel jobs, snapshot | |
| 109 | updates, memory profiling, and the web UI remain programmatic-only. See | |
| 110 | [the execution pipeline architecture](docs/execution-pipeline.md) for extension | |
| 111 | points, timeout semantics, and migration guidance. | |
| 107 | 112 | |
| 108 | 113 | **Examples:** |
| 109 | 114 | |
| @@ -634,7 +639,9 @@ zig-test --no-color | ||
| 634 | 639 | | `--version` | `-v` | Show version information | |
| 635 | 640 | | `--bail` | `-b` | Stop on first failure | |
| 636 | 641 | | `--filter <pattern>` | | Run only tests matching pattern | |
| 637 | | `--reporter <name>` | | Set reporter (spec, dot, json) | | |
| 642 | | `--reporter <name>` | | Set reporter (spec, dot, json, tap, junit) | | |
| 643 | | `--junit-output <file>` | | Set the JUnit XML output path | | |
| 644 | | `--timeout <ms>` | | Set the global test timeout | | |
| 638 | 645 | | `--verbose` | | Enable verbose output | |
| 639 | 646 | | `--quiet` | `-q` | Minimal output | |
| 640 | 647 | | `--no-color` | | Disable colored output | |
| @@ -0,0 +1,62 @@ | ||
| 1 | # Unified execution pipeline | |
| 2 | ||
| 3 | The framework has two plan producers and two executor implementations, but one | |
| 4 | execution contract. | |
| 5 | ||
| 6 | 1. Programmatic registration or file discovery produces a `TestPlan`. | |
| 7 | 2. A registered-test or external-process executor consumes that plan. | |
| 8 | 3. The executor emits typed `LifecycleEvent` values through `EventStream`. | |
| 9 | 4. A selected reporter consumes the events and a shared `TestResults` summary. | |
| 10 | ||
| 11 | `PlanItem.registered` retains the suite and test-case pointers needed for hooks | |
| 12 | and in-process execution. `PlanItem.discovered` retains the discovered file used | |
| 13 | by the external `zig test` executor. `PlanItem.identity()` gives extensions a | |
| 14 | common stable identity without erasing executor-specific information. | |
| 15 | ||
| 16 | ## Shared lifecycle and results | |
| 17 | ||
| 18 | Both executors emit run, suite, and test start/end events. Discovered files are | |
| 19 | represented as one synthetic suite and test result per external process. This | |
| 20 | means spec, dot, JSON, TAP, and JUnit reporters receive the same lifecycle and | |
| 21 | summary model in either mode. `ReporterSet` is the single factory for built-in | |
| 22 | reporters, including the configured JUnit output path. | |
| 23 | ||
| 24 | `ExecutionPolicy` owns common filtering, bail, and timeout decisions: | |
| 25 | ||
| 26 | - Registered tests match filters directly by test name. Discovery forwards the | |
| 27 | same filter to `zig test`, where Zig applies it to names inside each file. | |
| 28 | - Bail stops scheduling after the shared result model records a failure. Work | |
| 29 | already started by a parallel batch cannot be cancelled. | |
| 30 | - A global timeout compares elapsed execution time with the same policy in both | |
| 31 | executors. It marks an over-budget result failed after control returns; it | |
| 32 | does not yet terminate an in-process function or external child process. | |
| 33 | - Summaries always come from `TestResults`, rather than executor-local counters. | |
| 34 | ||
| 35 | ## Extension points | |
| 36 | ||
| 37 | New plan producers should append `PlanItem` values. New executors should emit | |
| 38 | `LifecycleEvent` values and update `TestResults`. New reporters can continue to | |
| 39 | implement `Reporter.VTable`; `EventStream` adapts the typed lifecycle to that | |
| 40 | stable callback interface. | |
| 41 | ||
| 42 | The event types are intentionally internal Zig structures. The separately | |
| 43 | versioned JSON wire protocol is built on this layer rather than making this API | |
| 44 | depend on one serialization format. | |
| 45 | ||
| 46 | ## Migration | |
| 47 | ||
| 48 | Existing programmatic APIs remain source compatible: | |
| 49 | ||
| 50 | - `TestRunner.init`, `runTests`, and `runTestsWithOptions` are unchanged. | |
| 51 | - `RunnerOptions` only gains optional `junit_output`, `timeout_ms`, and | |
| 52 | `reporter_writer` fields. | |
| 53 | - `runDiscoveredTests` and `LoaderOptions` remain available; optional reporter, | |
| 54 | JUnit path, timeout, and writer fields default to the previous spec behavior. | |
| 55 | ||
| 56 | CLI hosts can inject a stdout writer with `reporter_writer`. Embedded callers | |
| 57 | may leave it unset; reporter output then uses stderr so it cannot corrupt Zig's | |
| 58 | stdout-based test protocol. | |
| 59 | ||
| 60 | Code that implements a custom execution path can migrate incrementally by first | |
| 61 | building a `TestPlan`, then routing callbacks through `EventStream`. Existing | |
| 62 | custom reporters require no migration. | |