ReviewOS

also looking at this

zig-utils/zig-test-framework

Add reproducible randomized test order

#29
Merged glennmichael123 wants to merge codex/issue-14-random-order into main
18 files +487 -42
docs/json-event-protocol.mdmodified+4-2
Changes to docs/json-event-protocol.md
@@ -35,8 +35,8 @@ events are emitted in attempt order immediately before their `test_end`.
3535
3636| Type | Data fields |
3737| --- | --- |
38| `run_start` | `total` |
39| `run_end` | `total`, `passed`, `flaky`, `failed`, `skipped`, `duration_ns` |
38| `run_start` | `total`, optional `random_seed` |
39| `run_end` | `total`, `passed`, `flaky`, `failed`, `skipped`, `duration_ns`, optional `random_seed` |
4040| `suite_start`, `suite_end` | `name` |
4141| `test_start` | `name`, optional `suite` |
4242| `test_end` | `name`, optional `suite`, `status`, `duration_ns`, optional `error_message` |
@@ -51,6 +51,8 @@ Output streams are `stdout` or `stderr`.
5151
5252Optional fields are serialized as `null` when no value is available. Durations
5353are unsigned integer nanoseconds. Percentages are numbers from 0 through 100.
54When randomized ordering is enabled, both run events carry the reproducible
55unsigned integer seed.
5456
5557## Compatibility policy
5658