ReviewOS

also looking at this

zig-utils/zig-test-framework

Add reproducible randomized test order

#29
Merged glennmichael123 wants to merge codex/issue-14-random-order into main
18 files +487 -42
docs/execution-pipeline.mdmodified+6-2
Changes to docs/execution-pipeline.md
@@ -34,6 +34,9 @@ reporters, including the configured JUnit output path.
3434 child process.
3535- Every logical result owns ordered attempt history. A failure followed by a
3636 successful retry is classified as `flaky`, distinct from an ordinary pass.
37- Optional randomized execution uses one run seed across the selected plan.
38 The seed is carried by reporters and `TestResults`; see
39 [Randomized test order](randomized-order.md) for mode-specific guarantees.
3740- Summaries always come from `TestResults`, rather than executor-local counters.
3841
3942## Extension points
@@ -53,10 +56,11 @@ model without coupling the execution API to JSON.
5356Existing programmatic APIs remain source compatible:
5457
5558- `TestRunner.init`, `runTests`, and `runTestsWithOptions` are unchanged.
56- `RunnerOptions` gains optional reporter, timeout, retry, repeat, and flaky-exit
59- `RunnerOptions` gains optional reporter, timeout, retry, repeat, flaky-exit,
60 shuffle, and seed
5761 fields with backward-compatible defaults.
5862- `runDiscoveredTests` and `LoaderOptions` remain available; optional reporter,
59 JUnit path, timeout, retry, repeat, flaky-exit, and writer fields default to
63 JUnit path, timeout, retry, repeat, flaky-exit, shuffle, seed, and writer fields default to
6064 the previous spec behavior.
6165
6266CLI hosts can inject a stdout writer with `reporter_writer`. Embedded callers