also looking at this
Add reproducible randomized test order
#29
18 files
+487
-42
| @@ -9,6 +9,7 @@ A modern, feature-rich testing framework for Zig inspired by Jest, Vitest, and B | ||
| 9 | 9 | - **Async Test Support** - Full async test execution with concurrent and sequential modes |
| 10 | 10 | - **Timeout Handling** - Configurable timeouts at test, suite, and global levels with extension support |
| 11 | 11 | - **Retries & Repeats** - Re-run failures, stress-run selected tests, and report flaky outcomes |
| 12 | - **Reproducible Randomization** - Shuffle suites and tests with replayable seeds | |
| 12 | 13 | - **Familiar API** - Describe/it syntax similar to Jest and Vitest |
| 13 | 14 | - **Rich Assertions** - Comprehensive assertion library with `.expect()` and matchers |
| 14 | 15 | - **Error Assertions** - `toThrow()` and `toThrowError()` for testing error handling |
| @@ -102,6 +103,8 @@ This will automatically discover and run all `*.test.zig` files in the `tests` d | ||
| 102 | 103 | - `--retry <N>` - Retry a failed test or discovered test file up to N times |
| 103 | 104 | - `--repeat <N>` - Require N successful repetitions of every selected test |
| 104 | 105 | - `--fail-on-flaky` - Return a failing exit status when a retry recovers a failure |
| 106 | - `--shuffle` - Randomize suites, tests, or discovered files with a generated seed | |
| 107 | - `--seed <N>` - Replay a randomized order; supplying a seed implies `--shuffle` | |
| 105 | 108 | - `--shard-index <N>` - Run one one-based file shard |
| 106 | 109 | - `--shard-count <N>` - Set the total number of file shards |
| 107 | 110 | - `--no-color` - Disable color in child Zig test processes |
| @@ -139,6 +142,10 @@ zig-test --test-dir tests --retry 2 --fail-on-flaky | ||
| 139 | 142 | # Stress-run matching tests 100 times |
| 140 | 143 | zig-test --test-dir tests --filter parser --repeat 100 |
| 141 | 144 | |
| 145 | # Expose order dependencies, then replay the exact order with the printed seed | |
| 146 | zig-test --test-dir tests --shuffle | |
| 147 | zig-test --test-dir tests --seed 8675309 | |
| 148 | ||
| 142 | 149 | # Run the second of four deterministic file shards |
| 143 | 150 | zig-test --test-dir tests --shard-index 2 --shard-count 4 |
| 144 | 151 | ``` |
| @@ -185,7 +192,9 @@ precedence over configured values. | ||
| 185 | 192 | "filter": "database", |
| 186 | 193 | "retries": 2, |
| 187 | 194 | "repeat": 10, |
| 188 | "fail_on_flaky": true | |
| 195 | "fail_on_flaky": true, | |
| 196 | "shuffle": true, | |
| 197 | "seed": 8675309 | |
| 189 | 198 | }, |
| 190 | 199 | "sharding": { |
| 191 | 200 | "index": 1, |
| @@ -204,6 +213,8 @@ limitations as their CLI equivalents; unsupported combinations fail explicitly. | ||
| 204 | 213 | |
| 205 | 214 | See [Retries, repeats, and flaky tests](docs/retries-and-repeats.md) for attempt |
| 206 | 215 | semantics, exit codes, reporter output, and programmatic configuration. |
| 216 | See [Randomized test order](docs/randomized-order.md) for replay and ordering | |
| 217 | guarantees. | |
| 207 | 218 | |
| 208 | 219 | ### Parallel execution semantics |
| 209 | 220 | |
| @@ -212,7 +223,8 @@ and set `n_jobs`. A configured value is the exact maximum number of active test | ||
| 212 | 223 | workers; omitting it uses the host's logical CPU count, with a one-worker |
| 213 | 224 | fallback when detection is unavailable. |
| 214 | 225 | |
| 215 | Suites and nested suites are visited in declaration order. `beforeAll` and | |
| 226 | Suites and nested suites are visited in the selected order: declaration order | |
| 227 | by default, or deterministic seed order when randomization is enabled. `beforeAll` and | |
| 216 | 228 | `afterAll` run once on the coordinating thread around that suite and its nested |
| 217 | 229 | suites. Direct tests within a suite may overlap. Their inherited `beforeEach` |
| 218 | 230 | and `afterEach` hooks run on the same worker as the test, so per-test hook order |
| @@ -220,7 +232,7 @@ is preserved but hooks from different tests may overlap. Hook state and the | ||
| 220 | 232 | allocator passed to tests must therefore be thread-safe. |
| 221 | 233 | |
| 222 | 234 | Reporter callbacks never run concurrently: test results are emitted in |
| 223 | declaration order after each parallel batch, making output and final totals | |
| 235 | the selected order after each parallel batch, making output and final totals | |
| 224 | 236 | deterministic. A parallel batch is already scheduled as a unit, so `bail` cannot |
| 225 | 237 | cancel tests that have started. |
| 226 | 238 | |
| @@ -665,6 +677,8 @@ zig-test --no-color | ||
| 665 | 677 | | `--retry <N>` | `--retries <N>` | Retry each failed test up to N times | |
| 666 | 678 | | `--repeat <N>` | | Require N successful runs of each selected test | |
| 667 | 679 | | `--fail-on-flaky` | | Fail the run when a retry recovers a failure | |
| 680 | | `--shuffle` | | Randomize suite, test, and discovered-file order | | |
| 681 | | `--seed <N>` | | Replay a randomized order; implies `--shuffle` | | |
| 668 | 682 | | `--verbose` | | Enable verbose output | |
| 669 | 683 | | `--quiet` | `-q` | Minimal output | |
| 670 | 684 | | `--no-color` | | Disable colored output | |