also looking at this
Unify discovery and programmatic test execution
#23
12 files
+656
-169
| @@ -0,0 +1,62 @@ | ||
| 1 | # Unified execution pipeline | |
| 2 | ||
| 3 | The framework has two plan producers and two executor implementations, but one | |
| 4 | execution contract. | |
| 5 | ||
| 6 | 1. Programmatic registration or file discovery produces a `TestPlan`. | |
| 7 | 2. A registered-test or external-process executor consumes that plan. | |
| 8 | 3. The executor emits typed `LifecycleEvent` values through `EventStream`. | |
| 9 | 4. A selected reporter consumes the events and a shared `TestResults` summary. | |
| 10 | ||
| 11 | `PlanItem.registered` retains the suite and test-case pointers needed for hooks | |
| 12 | and in-process execution. `PlanItem.discovered` retains the discovered file used | |
| 13 | by the external `zig test` executor. `PlanItem.identity()` gives extensions a | |
| 14 | common stable identity without erasing executor-specific information. | |
| 15 | ||
| 16 | ## Shared lifecycle and results | |
| 17 | ||
| 18 | Both executors emit run, suite, and test start/end events. Discovered files are | |
| 19 | represented as one synthetic suite and test result per external process. This | |
| 20 | means spec, dot, JSON, TAP, and JUnit reporters receive the same lifecycle and | |
| 21 | summary model in either mode. `ReporterSet` is the single factory for built-in | |
| 22 | reporters, including the configured JUnit output path. | |
| 23 | ||
| 24 | `ExecutionPolicy` owns common filtering, bail, and timeout decisions: | |
| 25 | ||
| 26 | - Registered tests match filters directly by test name. Discovery forwards the | |
| 27 | same filter to `zig test`, where Zig applies it to names inside each file. | |
| 28 | - Bail stops scheduling after the shared result model records a failure. Work | |
| 29 | already started by a parallel batch cannot be cancelled. | |
| 30 | - A global timeout compares elapsed execution time with the same policy in both | |
| 31 | executors. It marks an over-budget result failed after control returns; it | |
| 32 | does not yet terminate an in-process function or external child process. | |
| 33 | - Summaries always come from `TestResults`, rather than executor-local counters. | |
| 34 | ||
| 35 | ## Extension points | |
| 36 | ||
| 37 | New plan producers should append `PlanItem` values. New executors should emit | |
| 38 | `LifecycleEvent` values and update `TestResults`. New reporters can continue to | |
| 39 | implement `Reporter.VTable`; `EventStream` adapts the typed lifecycle to that | |
| 40 | stable callback interface. | |
| 41 | ||
| 42 | The event types are intentionally internal Zig structures. The separately | |
| 43 | versioned JSON wire protocol is built on this layer rather than making this API | |
| 44 | depend on one serialization format. | |
| 45 | ||
| 46 | ## Migration | |
| 47 | ||
| 48 | Existing programmatic APIs remain source compatible: | |
| 49 | ||
| 50 | - `TestRunner.init`, `runTests`, and `runTestsWithOptions` are unchanged. | |
| 51 | - `RunnerOptions` only gains optional `junit_output`, `timeout_ms`, and | |
| 52 | `reporter_writer` fields. | |
| 53 | - `runDiscoveredTests` and `LoaderOptions` remain available; optional reporter, | |
| 54 | JUnit path, timeout, and writer fields default to the previous spec behavior. | |
| 55 | ||
| 56 | CLI hosts can inject a stdout writer with `reporter_writer`. Embedded callers | |
| 57 | may leave it unset; reporter output then uses stderr so it cannot corrupt Zig's | |
| 58 | stdout-based test protocol. | |
| 59 | ||
| 60 | Code that implements a custom execution path can migrate incrementally by first | |
| 61 | building a `TestPlan`, then routing callbacks through `EventStream`. Existing | |
| 62 | custom reporters require no migration. | |