also looking at this
Add retries, repeated runs, and flaky-test reporting
#25
15 files
+719
-53
| @@ -8,6 +8,7 @@ A modern, feature-rich testing framework for Zig inspired by Jest, Vitest, and B | ||
| 8 | 8 | - **Code Coverage** - Line, branch, and function coverage with HTML reports (via kcov/grindcov) |
| 9 | 9 | - **Async Test Support** - Full async test execution with concurrent and sequential modes |
| 10 | 10 | - **Timeout Handling** - Configurable timeouts at test, suite, and global levels with extension support |
| 11 | - **Retries & Repeats** - Re-run failures, stress-run selected tests, and report flaky outcomes | |
| 11 | 12 | - **Familiar API** - Describe/it syntax similar to Jest and Vitest |
| 12 | 13 | - **Rich Assertions** - Comprehensive assertion library with `.expect()` and matchers |
| 13 | 14 | - **Error Assertions** - `toThrow()` and `toThrowError()` for testing error handling |
| @@ -98,6 +99,9 @@ This will automatically discover and run all `*.test.zig` files in the `tests` d | ||
| 98 | 99 | - `--reporter <name>` - Use spec, dot, JSON, TAP, or JUnit output |
| 99 | 100 | - `--junit-output <file>` - Set the JUnit XML path |
| 100 | 101 | - `--timeout <ms>` - Mark files that exceed the global time budget as failed |
| 102 | - `--retry <N>` - Retry a failed test or discovered test file up to N times | |
| 103 | - `--repeat <N>` - Require N successful repetitions of every selected test | |
| 104 | - `--fail-on-flaky` - Return a failing exit status when a retry recovers a failure | |
| 101 | 105 | - `--shard-index <N>` - Run one one-based file shard |
| 102 | 106 | - `--shard-count <N>` - Set the total number of file shards |
| 103 | 107 | - `--no-color` - Disable color in child Zig test processes |
| @@ -125,6 +129,12 @@ zig-test --test-dir tests --no-recursive | ||
| 125 | 129 | # Stop on first failure |
| 126 | 130 | zig-test --test-dir tests --bail |
| 127 | 131 | |
| 132 | # Retry failures twice and fail CI if any test is flaky | |
| 133 | zig-test --test-dir tests --retry 2 --fail-on-flaky | |
| 134 | ||
| 135 | # Stress-run matching tests 100 times | |
| 136 | zig-test --test-dir tests --filter parser --repeat 100 | |
| 137 | ||
| 128 | 138 | # Run the second of four deterministic file shards |
| 129 | 139 | zig-test --test-dir tests --shard-index 2 --shard-count 4 |
| 130 | 140 | ``` |
| @@ -168,7 +178,10 @@ precedence over configured values. | ||
| 168 | 178 | "test_dir": "tests", |
| 169 | 179 | "pattern": "*.test.zig", |
| 170 | 180 | "recursive": true, |
| 171 | "filter": "database" | |
| 181 | "filter": "database", | |
| 182 | "retries": 2, | |
| 183 | "repeat": 10, | |
| 184 | "fail_on_flaky": true | |
| 172 | 185 | }, |
| 173 | 186 | "sharding": { |
| 174 | 187 | "index": 1, |
| @@ -185,6 +198,9 @@ The accepted top-level sections are `test`, `sharding`, `parallel`, `reporter`, | ||
| 185 | 198 | `snapshot`, `watch`, `memory`, `ui`, and `coverage`. Options retain the same execution-mode |
| 186 | 199 | limitations as their CLI equivalents; unsupported combinations fail explicitly. |
| 187 | 200 | |
| 201 | See [Retries, repeats, and flaky tests](docs/retries-and-repeats.md) for attempt | |
| 202 | semantics, exit codes, reporter output, and programmatic configuration. | |
| 203 | ||
| 188 | 204 | ### Parallel execution semantics |
| 189 | 205 | |
| 190 | 206 | Programmatic runners can enable bounded execution with `RunnerOptions.parallel` |
| @@ -642,6 +658,9 @@ zig-test --no-color | ||
| 642 | 658 | | `--reporter <name>` | | Set reporter (spec, dot, json, tap, junit) | |
| 643 | 659 | | `--junit-output <file>` | | Set the JUnit XML output path | |
| 644 | 660 | | `--timeout <ms>` | | Set the global test timeout | |
| 661 | | `--retry <N>` | `--retries <N>` | Retry each failed test up to N times | | |
| 662 | | `--repeat <N>` | | Require N successful runs of each selected test | | |
| 663 | | `--fail-on-flaky` | | Fail the run when a retry recovers a failure | | |
| 645 | 664 | | `--verbose` | | Enable verbose output | |
| 646 | 665 | | `--quiet` | `-q` | Minimal output | |
| 647 | 666 | | `--no-color` | | Disable colored output | |
| @@ -921,6 +940,7 @@ MIT License - see [LICENSE](LICENSE) for details. | ||
| 921 | 940 | - [x] Memory profiling |
| 922 | 941 | - [x] Strict JSON configuration files |
| 923 | 942 | - [x] Parallel test execution |
| 943 | - [x] Retries, repeated runs, and flaky-test reporting | |
| 924 | 944 | - [ ] Parameterized tests (it.each) |
| 925 | 945 | - [ ] Property-based testing |
| 926 | 946 | - [ ] IDE integration |