also looking at this
Add retries, repeated runs, and flaky-test reporting
#25
15 files
+719
-53
| @@ -26,10 +26,14 @@ reporters, including the configured JUnit output path. | ||
| 26 | 26 | - Registered tests match filters directly by test name. Discovery forwards the |
| 27 | 27 | same filter to `zig test`, where Zig applies it to names inside each file. |
| 28 | 28 | - Bail stops scheduling after the shared result model records a failure. Work |
| 29 | already started by a parallel batch cannot be cancelled. | |
| 29 | already started by a parallel batch cannot be cancelled. With retries, bail | |
| 30 | is evaluated only after the retry budget is exhausted. | |
| 30 | 31 | - A global timeout compares elapsed execution time with the same policy in both |
| 31 | executors. It marks an over-budget result failed after control returns; it | |
| 32 | does not yet terminate an in-process function or external child process. | |
| 32 | executors on every attempt. It marks an over-budget attempt failed after | |
| 33 | control returns; it does not yet terminate an in-process function or external | |
| 34 | child process. | |
| 35 | - Every logical result owns ordered attempt history. A failure followed by a | |
| 36 | successful retry is classified as `flaky`, distinct from an ordinary pass. | |
| 33 | 37 | - Summaries always come from `TestResults`, rather than executor-local counters. |
| 34 | 38 | |
| 35 | 39 | ## Extension points |
| @@ -48,10 +52,11 @@ depend on one serialization format. | ||
| 48 | 52 | Existing programmatic APIs remain source compatible: |
| 49 | 53 | |
| 50 | 54 | - `TestRunner.init`, `runTests`, and `runTestsWithOptions` are unchanged. |
| 51 | - `RunnerOptions` only gains optional `junit_output`, `timeout_ms`, and | |
| 52 | `reporter_writer` fields. | |
| 55 | - `RunnerOptions` gains optional reporter, timeout, retry, repeat, and flaky-exit | |
| 56 | fields with backward-compatible defaults. | |
| 53 | 57 | - `runDiscoveredTests` and `LoaderOptions` remain available; optional reporter, |
| 54 | JUnit path, timeout, and writer fields default to the previous spec behavior. | |
| 58 | JUnit path, timeout, retry, repeat, flaky-exit, and writer fields default to | |
| 59 | the previous spec behavior. | |
| 55 | 60 | |
| 56 | 61 | CLI hosts can inject a stdout writer with `reporter_writer`. Embedded callers |
| 57 | 62 | may leave it unset; reporter output then uses stderr so it cannot corrupt Zig's |