also looking at this
Add retries, repeated runs, and flaky-test reporting
#25
15 files
+719
-53
Review threads live on the whole diff, not on one commit, so none are shown here - a thread's line means something in the branch's final form, and painting it into an intermediate step would put it on code it is not about.
| @@ -16,6 +16,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 | ||
| 16 | 16 | |
| 17 | 17 | ## [Unreleased] |
| 18 | 18 | |
| 19 | ### Added | |
| 20 | ||
| 21 | - Retry failed tests and discovered test files with complete attempt history. | |
| 22 | - Repeat mode for stress-running selected tests. | |
| 23 | - Flaky outcomes, optional fail-on-flaky exit behavior, and JSON/JUnit attempt data. | |
| 24 | ||
| 19 | 25 | ### Planned Features |
| 20 | 26 | |
| 21 | 27 | - Snapshot testing |
| @@ -8,6 +8,7 @@ A modern, feature-rich testing framework for Zig inspired by Jest, Vitest, and B | ||
| 8 | 8 | - **Code Coverage** - Line, branch, and function coverage with HTML reports (via kcov/grindcov) |
| 9 | 9 | - **Async Test Support** - Full async test execution with concurrent and sequential modes |
| 10 | 10 | - **Timeout Handling** - Configurable timeouts at test, suite, and global levels with extension support |
| 11 | - **Retries & Repeats** - Re-run failures, stress-run selected tests, and report flaky outcomes | |
| 11 | 12 | - **Familiar API** - Describe/it syntax similar to Jest and Vitest |
| 12 | 13 | - **Rich Assertions** - Comprehensive assertion library with `.expect()` and matchers |
| 13 | 14 | - **Error Assertions** - `toThrow()` and `toThrowError()` for testing error handling |
| @@ -98,6 +99,9 @@ This will automatically discover and run all `*.test.zig` files in the `tests` d | ||
| 98 | 99 | - `--reporter <name>` - Use spec, dot, JSON, TAP, or JUnit output |
| 99 | 100 | - `--junit-output <file>` - Set the JUnit XML path |
| 100 | 101 | - `--timeout <ms>` - Mark files that exceed the global time budget as failed |
| 102 | - `--retry <N>` - Retry a failed test or discovered test file up to N times | |
| 103 | - `--repeat <N>` - Require N successful repetitions of every selected test | |
| 104 | - `--fail-on-flaky` - Return a failing exit status when a retry recovers a failure | |
| 101 | 105 | - `--shard-index <N>` - Run one one-based file shard |
| 102 | 106 | - `--shard-count <N>` - Set the total number of file shards |
| 103 | 107 | - `--no-color` - Disable color in child Zig test processes |
| @@ -125,6 +129,12 @@ zig-test --test-dir tests --no-recursive | ||
| 125 | 129 | # Stop on first failure |
| 126 | 130 | zig-test --test-dir tests --bail |
| 127 | 131 | |
| 132 | # Retry failures twice and fail CI if any test is flaky | |
| 133 | zig-test --test-dir tests --retry 2 --fail-on-flaky | |
| 134 | ||
| 135 | # Stress-run matching tests 100 times | |
| 136 | zig-test --test-dir tests --filter parser --repeat 100 | |
| 137 | ||
| 128 | 138 | # Run the second of four deterministic file shards |
| 129 | 139 | zig-test --test-dir tests --shard-index 2 --shard-count 4 |
| 130 | 140 | ``` |
| @@ -168,7 +178,10 @@ precedence over configured values. | ||
| 168 | 178 | "test_dir": "tests", |
| 169 | 179 | "pattern": "*.test.zig", |
| 170 | 180 | "recursive": true, |
| 171 | "filter": "database" | |
| 181 | "filter": "database", | |
| 182 | "retries": 2, | |
| 183 | "repeat": 10, | |
| 184 | "fail_on_flaky": true | |
| 172 | 185 | }, |
| 173 | 186 | "sharding": { |
| 174 | 187 | "index": 1, |
| @@ -185,6 +198,9 @@ The accepted top-level sections are `test`, `sharding`, `parallel`, `reporter`, | ||
| 185 | 198 | `snapshot`, `watch`, `memory`, `ui`, and `coverage`. Options retain the same execution-mode |
| 186 | 199 | limitations as their CLI equivalents; unsupported combinations fail explicitly. |
| 187 | 200 | |
| 201 | See [Retries, repeats, and flaky tests](docs/retries-and-repeats.md) for attempt | |
| 202 | semantics, exit codes, reporter output, and programmatic configuration. | |
| 203 | ||
| 188 | 204 | ### Parallel execution semantics |
| 189 | 205 | |
| 190 | 206 | Programmatic runners can enable bounded execution with `RunnerOptions.parallel` |
| @@ -642,6 +658,9 @@ zig-test --no-color | ||
| 642 | 658 | | `--reporter <name>` | | Set reporter (spec, dot, json, tap, junit) | |
| 643 | 659 | | `--junit-output <file>` | | Set the JUnit XML output path | |
| 644 | 660 | | `--timeout <ms>` | | Set the global test timeout | |
| 661 | | `--retry <N>` | `--retries <N>` | Retry each failed test up to N times | | |
| 662 | | `--repeat <N>` | | Require N successful runs of each selected test | | |
| 663 | | `--fail-on-flaky` | | Fail the run when a retry recovers a failure | | |
| 645 | 664 | | `--verbose` | | Enable verbose output | |
| 646 | 665 | | `--quiet` | `-q` | Minimal output | |
| 647 | 666 | | `--no-color` | | Disable colored output | |
| @@ -921,6 +940,7 @@ MIT License - see [LICENSE](LICENSE) for details. | ||
| 921 | 940 | - [x] Memory profiling |
| 922 | 941 | - [x] Strict JSON configuration files |
| 923 | 942 | - [x] Parallel test execution |
| 943 | - [x] Retries, repeated runs, and flaky-test reporting | |
| 924 | 944 | - [ ] Parameterized tests (it.each) |
| 925 | 945 | - [ ] Property-based testing |
| 926 | 946 | - [ ] IDE integration |
| @@ -293,6 +293,21 @@ pub fn build(b: *std.Build) void { | ||
| 293 | 293 | ); |
| 294 | 294 | test_step.dependOn(&run_discovery_tap.step); |
| 295 | 295 | |
| 296 | const run_discovery_repeat = b.addRunArtifact(exe); | |
| 297 | run_discovery_repeat.addArgs(&.{ | |
| 298 | "--test-dir", | |
| 299 | "tests", | |
| 300 | "--pattern", | |
| 301 | "sample.test.zig", | |
| 302 | "--retry", | |
| 303 | "1", | |
| 304 | "--repeat", | |
| 305 | "2", | |
| 306 | "--fail-on-flaky", | |
| 307 | "--no-color", | |
| 308 | }); | |
| 309 | test_step.dependOn(&run_discovery_repeat.step); | |
| 310 | ||
| 296 | 311 | const run_version = b.addRunArtifact(exe); |
| 297 | 312 | run_version.addArg("--version"); |
| 298 | 313 | run_version.expectStdErrEqual(b.fmt("Zig Test Framework v{s}\n", .{package_version})); |
| @@ -26,10 +26,14 @@ reporters, including the configured JUnit output path. | ||
| 26 | 26 | - Registered tests match filters directly by test name. Discovery forwards the |
| 27 | 27 | same filter to `zig test`, where Zig applies it to names inside each file. |
| 28 | 28 | - Bail stops scheduling after the shared result model records a failure. Work |
| 29 | already started by a parallel batch cannot be cancelled. | |
| 29 | already started by a parallel batch cannot be cancelled. With retries, bail | |
| 30 | is evaluated only after the retry budget is exhausted. | |
| 30 | 31 | - A global timeout compares elapsed execution time with the same policy in both |
| 31 | executors. It marks an over-budget result failed after control returns; it | |
| 32 | does not yet terminate an in-process function or external child process. | |
| 32 | executors on every attempt. It marks an over-budget attempt failed after | |
| 33 | control returns; it does not yet terminate an in-process function or external | |
| 34 | child process. | |
| 35 | - Every logical result owns ordered attempt history. A failure followed by a | |
| 36 | successful retry is classified as `flaky`, distinct from an ordinary pass. | |
| 33 | 37 | - Summaries always come from `TestResults`, rather than executor-local counters. |
| 34 | 38 | |
| 35 | 39 | ## Extension points |
| @@ -48,10 +52,11 @@ depend on one serialization format. | ||
| 48 | 52 | Existing programmatic APIs remain source compatible: |
| 49 | 53 | |
| 50 | 54 | - `TestRunner.init`, `runTests`, and `runTestsWithOptions` are unchanged. |
| 51 | - `RunnerOptions` only gains optional `junit_output`, `timeout_ms`, and | |
| 52 | `reporter_writer` fields. | |
| 55 | - `RunnerOptions` gains optional reporter, timeout, retry, repeat, and flaky-exit | |
| 56 | fields with backward-compatible defaults. | |
| 53 | 57 | - `runDiscoveredTests` and `LoaderOptions` remain available; optional reporter, |
| 54 | JUnit path, timeout, and writer fields default to the previous spec behavior. | |
| 58 | JUnit path, timeout, retry, repeat, flaky-exit, and writer fields default to | |
| 59 | the previous spec behavior. | |
| 55 | 60 | |
| 56 | 61 | CLI hosts can inject a stdout writer with `reporter_writer`. Embedded callers |
| 57 | 62 | may leave it unset; reporter output then uses stderr so it cannot corrupt Zig's |
| @@ -0,0 +1,92 @@ | ||
| 1 | # Retries, repeats, and flaky tests | |
| 2 | ||
| 3 | The runner records each execution of a logical test as an attempt. An attempt | |
| 4 | contains its one-based number, repetition number, status, duration, and error | |
| 5 | message. Registered tests and discovered test files use the same model. | |
| 6 | ||
| 7 | ## Command-line use | |
| 8 | ||
| 9 | ```bash | |
| 10 | # One initial attempt plus as many as two retries after failure | |
| 11 | zig-test --test-dir tests --retry 2 | |
| 12 | ||
| 13 | # Require every selected test to pass 100 times | |
| 14 | zig-test --test-dir tests --filter parser --repeat 100 | |
| 15 | ||
| 16 | # Make a recovered failure fail the overall run in CI | |
| 17 | zig-test --test-dir tests --retry 2 --fail-on-flaky | |
| 18 | ``` | |
| 19 | ||
| 20 | `--retries` is an alias for `--retry`. A retry count is the number of | |
| 21 | *additional* attempts after a failure. `--repeat N` requires N successful | |
| 22 | repetitions; each repetition receives its own retry budget. Attempt history is | |
| 23 | ordered across all repetitions. | |
| 24 | ||
| 25 | Discovery operates at test-file granularity because each file is an external | |
| 26 | `zig test` process. Programmatic mode operates at individual-test granularity. | |
| 27 | ||
| 28 | ## Outcomes and exit status | |
| 29 | ||
| 30 | - `passed`: every attempt passed without a retry. | |
| 31 | - `flaky`: at least one attempt failed, but a retry recovered and all required | |
| 32 | repetitions eventually passed. | |
| 33 | - `failed`: one repetition exhausted its retry budget. | |
| 34 | - `skipped`: the test was not selected for execution. | |
| 35 | ||
| 36 | Failed tests always make the process exit unsuccessfully. Flaky tests succeed | |
| 37 | by default so retries can recover an intermittent failure. `--fail-on-flaky` | |
| 38 | changes that policy and returns an unsuccessful exit status when any test is | |
| 39 | flaky. | |
| 40 | ||
| 41 | `--bail` is evaluated after retries, so one failed attempt does not stop the | |
| 42 | run. The runner stops scheduling only after a logical test exhausts its retry | |
| 43 | budget. The timeout is evaluated independently for every attempt; a timed-out | |
| 44 | attempt may be retried. | |
| 45 | ||
| 46 | When retries or repeats are configured for registered tests, execution is | |
| 47 | sequential even if parallel mode was requested. This preserves deterministic | |
| 48 | attempt ordering and hook behavior. | |
| 49 | ||
| 50 | ## Programmatic configuration | |
| 51 | ||
| 52 | Set defaults for a registered-test run with `RunnerOptions`: | |
| 53 | ||
| 54 | ```zig | |
| 55 | const options = ztf.RunnerOptions{ | |
| 56 | .retries = 2, | |
| 57 | .repeat = 10, | |
| 58 | .fail_on_flaky = true, | |
| 59 | }; | |
| 60 | const passed = try ztf.runTestsWithOptions(allocator, registry, options); | |
| 61 | ``` | |
| 62 | ||
| 63 | An individual registered test can override the run-level retry count: | |
| 64 | ||
| 65 | ```zig | |
| 66 | try suite.addTest( | |
| 67 | ztf.TestCase.init("eventually consistent", testEventuallyConsistent) | |
| 68 | .withRetries(5), | |
| 69 | ); | |
| 70 | ``` | |
| 71 | ||
| 72 | `LoaderOptions` exposes the same `retries`, `repeat`, and `fail_on_flaky` | |
| 73 | fields for callers that run discovered files directly. Strict JSON | |
| 74 | configuration uses the corresponding fields in the `test` object: | |
| 75 | ||
| 76 | ```json | |
| 77 | { | |
| 78 | "test": { | |
| 79 | "retries": 2, | |
| 80 | "repeat": 10, | |
| 81 | "fail_on_flaky": true | |
| 82 | } | |
| 83 | } | |
| 84 | ``` | |
| 85 | ||
| 86 | ## Reporter data | |
| 87 | ||
| 88 | The spec and dot reporters show flaky results separately from passes. JSON test | |
| 89 | objects include an `attempts` array with `number`, `repetition`, `status`, and | |
| 90 | `durationNs` fields plus an optional `error`. The JSON summary includes a | |
| 91 | `flaky` count. JUnit output includes the flaky count on `<testsuites>` and emits | |
| 92 | ordered attempt data in each retried test case's `<system-out>` element. | |
| @@ -244,6 +244,9 @@ test "ConfigLoader loads supported JSON and rejects unknown keys" { | ||
| 244 | 244 | try std.testing.expectEqualStrings("tests/fixtures", config.test_options.test_dir); |
| 245 | 245 | try std.testing.expectEqualStrings("selected test passes", config.test_options.filter.?); |
| 246 | 246 | try std.testing.expect(!config.test_options.recursive); |
| 247 | try std.testing.expectEqual(@as(usize, 1), config.test_options.retries); | |
| 248 | try std.testing.expectEqual(@as(usize, 2), config.test_options.repeat); | |
| 249 | try std.testing.expect(!config.test_options.fail_on_flaky); | |
| 247 | 250 | try std.testing.expectEqual(@as(?usize, 1), config.sharding.index); |
| 248 | 251 | try std.testing.expectEqual(@as(?usize, 1), config.sharding.count); |
| 249 | 252 | |
| @@ -3,7 +3,10 @@ | ||
| 3 | 3 | "test_dir": "tests/fixtures", |
| 4 | 4 | "pattern": "filter_fixture.zig", |
| 5 | 5 | "recursive": false, |
| 6 | "filter": "selected test passes" | |
| 6 | "filter": "selected test passes", | |
| 7 | "retries": 1, | |
| 8 | "repeat": 2, | |
| 9 | "fail_on_flaky": false | |
| 7 | 10 | }, |
| 8 | 11 | "sharding": { |
| 9 | 12 | "index": 1, |