ReviewOS

also looking at this

zig-utils/zig-test-framework

Add reproducible randomized test order

#29
Merged glennmichael123 wants to merge codex/issue-14-random-order into main
18 files +487 -42
docs/randomized-order.mdadded+58-0
Changes to docs/randomized-order.md
@@ -0,0 +1,58 @@
1# Randomized test order
2
3Randomized order is opt-in. Use `--shuffle` to generate a seed for the run:
4
5```bash
6zig-test --shuffle
7```
8
9The selected reporter prints the generated seed at run start. If a test fails,
10the human-readable reporters print a replay command as well. Pass that seed to
11reproduce the order; `--seed` implies `--shuffle`:
12
13```bash
14zig-test --seed 8675309
15```
16
17Keep every other order-affecting option the same when replaying a run, including
18the test set, filter, shard, execution mode, and suite structure.
19
20## Ordering guarantees
21
22Without `--shuffle` or `--seed`, execution retains declaration or discovery
23order.
24
25In sequential programmatic execution, a seed deterministically orders root
26suites, each suite's direct tests, and its nested suites. Lifecycle hooks keep
27their normal scope: `beforeAll` and `afterAll` still wrap a suite, while
28`beforeEach` and `afterEach` still wrap their selected test.
29
30In parallel programmatic execution, the seed determines suite traversal, each
31worker queue, and serialized reporter output. Workers claim tests from that
32queue in order, but operating-system scheduling means wall-clock start and
33completion timing can differ between runs. Reporter order remains reproducible.
34
35In discovery mode, sharding selects files first and randomization orders the
36files inside the selected shard. The seed does not change shard membership.
37Retries and repetitions stay attached to their logical test or file and do not
38trigger another shuffle.
39
40## Metadata
41
42The seed is stored as `random_seed` in JSON `run_start` and `run_end` events,
43live UI events, and test-history files. JUnit output records it as a
44`random_seed` property on each test suite. A non-randomized run reports a null
45seed in JSON-compatible formats and omits the JUnit property.
46
47Programmatic callers can use `RunnerOptions.shuffle` and `RunnerOptions.seed`,
48or the matching `LoaderOptions` fields. JSON configuration accepts the same
49values under `test`:
50
51```json
52{
53 "test": {
54 "shuffle": true,
55 "seed": 8675309
56 }
57}
58```