also looking at this
Add reproducible randomized test order
#29
18 files
+487
-42
| @@ -0,0 +1,58 @@ | ||
| 1 | # Randomized test order | |
| 2 | ||
| 3 | Randomized order is opt-in. Use `--shuffle` to generate a seed for the run: | |
| 4 | ||
| 5 | ```bash | |
| 6 | zig-test --shuffle | |
| 7 | ``` | |
| 8 | ||
| 9 | The selected reporter prints the generated seed at run start. If a test fails, | |
| 10 | the human-readable reporters print a replay command as well. Pass that seed to | |
| 11 | reproduce the order; `--seed` implies `--shuffle`: | |
| 12 | ||
| 13 | ```bash | |
| 14 | zig-test --seed 8675309 | |
| 15 | ``` | |
| 16 | ||
| 17 | Keep every other order-affecting option the same when replaying a run, including | |
| 18 | the test set, filter, shard, execution mode, and suite structure. | |
| 19 | ||
| 20 | ## Ordering guarantees | |
| 21 | ||
| 22 | Without `--shuffle` or `--seed`, execution retains declaration or discovery | |
| 23 | order. | |
| 24 | ||
| 25 | In sequential programmatic execution, a seed deterministically orders root | |
| 26 | suites, each suite's direct tests, and its nested suites. Lifecycle hooks keep | |
| 27 | their normal scope: `beforeAll` and `afterAll` still wrap a suite, while | |
| 28 | `beforeEach` and `afterEach` still wrap their selected test. | |
| 29 | ||
| 30 | In parallel programmatic execution, the seed determines suite traversal, each | |
| 31 | worker queue, and serialized reporter output. Workers claim tests from that | |
| 32 | queue in order, but operating-system scheduling means wall-clock start and | |
| 33 | completion timing can differ between runs. Reporter order remains reproducible. | |
| 34 | ||
| 35 | In discovery mode, sharding selects files first and randomization orders the | |
| 36 | files inside the selected shard. The seed does not change shard membership. | |
| 37 | Retries and repetitions stay attached to their logical test or file and do not | |
| 38 | trigger another shuffle. | |
| 39 | ||
| 40 | ## Metadata | |
| 41 | ||
| 42 | The seed is stored as `random_seed` in JSON `run_start` and `run_end` events, | |
| 43 | live UI events, and test-history files. JUnit output records it as a | |
| 44 | `random_seed` property on each test suite. A non-randomized run reports a null | |
| 45 | seed in JSON-compatible formats and omits the JUnit property. | |
| 46 | ||
| 47 | Programmatic callers can use `RunnerOptions.shuffle` and `RunnerOptions.seed`, | |
| 48 | or the matching `LoaderOptions` fields. JSON configuration accepts the same | |
| 49 | values under `test`: | |
| 50 | ||
| 51 | ```json | |
| 52 | { | |
| 53 | "test": { | |
| 54 | "shuffle": true, | |
| 55 | "seed": 8675309 | |
| 56 | } | |
| 57 | } | |
| 58 | ``` | |