also looking at this

Add reproducible randomized test order

#29

Merged glennmichael123 wants to merge codex/issue-14-random-order into main

18 files +487 -42

Review threads live on the whole diff, not on one commit, so none are shown here - a thread's line means something in the branch's final form, and painting it into an intermediate step would put it on code it is not about.

README.mdmodified+17-3

Changes to README.md
		@@ -9,6 +9,7 @@ A modern, feature-rich testing framework for Zig inspired by Jest, Vitest, and B
9	9	- Async Test Support - Full async test execution with concurrent and sequential modes
10	10	- Timeout Handling - Configurable timeouts at test, suite, and global levels with extension support
11	11	- Retries & Repeats - Re-run failures, stress-run selected tests, and report flaky outcomes
	12	- Reproducible Randomization - Shuffle suites and tests with replayable seeds
12	13	- Familiar API - Describe/it syntax similar to Jest and Vitest
13	14	- Rich Assertions - Comprehensive assertion library with `.expect()` and matchers
14	15	- Error Assertions - `toThrow()` and `toThrowError()` for testing error handling
		@@ -102,6 +103,8 @@ This will automatically discover and run all `*.test.zig` files in the `tests` d
102	103	- `--retry <N>` - Retry a failed test or discovered test file up to N times
103	104	- `--repeat <N>` - Require N successful repetitions of every selected test
104	105	- `--fail-on-flaky` - Return a failing exit status when a retry recovers a failure
	106	- `--shuffle` - Randomize suites, tests, or discovered files with a generated seed
	107	- `--seed <N>` - Replay a randomized order; supplying a seed implies `--shuffle`
105	108	- `--shard-index <N>` - Run one one-based file shard
106	109	- `--shard-count <N>` - Set the total number of file shards
107	110	- `--no-color` - Disable color in child Zig test processes
		@@ -139,6 +142,10 @@ zig-test --test-dir tests --retry 2 --fail-on-flaky
139	142	# Stress-run matching tests 100 times
140	143	zig-test --test-dir tests --filter parser --repeat 100
141	144
	145	# Expose order dependencies, then replay the exact order with the printed seed
	146	zig-test --test-dir tests --shuffle
	147	zig-test --test-dir tests --seed 8675309
	148
142	149	# Run the second of four deterministic file shards
143	150	zig-test --test-dir tests --shard-index 2 --shard-count 4
144	151	```
		@@ -185,7 +192,9 @@ precedence over configured values.
185	192	"filter": "database",
186	193	"retries": 2,
187	194	"repeat": 10,
188		"fail_on_flaky": true
	195	"fail_on_flaky": true,
	196	"shuffle": true,
	197	"seed": 8675309
189	198	},
190	199	"sharding": {
191	200	"index": 1,
		@@ -204,6 +213,8 @@ limitations as their CLI equivalents; unsupported combinations fail explicitly.
204	213
205	214	See [Retries, repeats, and flaky tests](docs/retries-and-repeats.md) for attempt
206	215	semantics, exit codes, reporter output, and programmatic configuration.
	216	See [Randomized test order](docs/randomized-order.md) for replay and ordering
	217	guarantees.
207	218
208	219	### Parallel execution semantics
209	220
		@@ -212,7 +223,8 @@ and set `n_jobs`. A configured value is the exact maximum number of active test
212	223	workers; omitting it uses the host's logical CPU count, with a one-worker
213	224	fallback when detection is unavailable.
214	225
215		Suites and nested suites are visited in declaration order. `beforeAll` and
	226	Suites and nested suites are visited in the selected order: declaration order
	227	by default, or deterministic seed order when randomization is enabled. `beforeAll` and
216	228	`afterAll` run once on the coordinating thread around that suite and its nested
217	229	suites. Direct tests within a suite may overlap. Their inherited `beforeEach`
218	230	and `afterEach` hooks run on the same worker as the test, so per-test hook order
		@@ -220,7 +232,7 @@ is preserved but hooks from different tests may overlap. Hook state and the
220	232	allocator passed to tests must therefore be thread-safe.
221	233
222	234	Reporter callbacks never run concurrently: test results are emitted in
223		declaration order after each parallel batch, making output and final totals
	235	the selected order after each parallel batch, making output and final totals
224	236	deterministic. A parallel batch is already scheduled as a unit, so `bail` cannot
225	237	cancel tests that have started.
226	238
		@@ -665,6 +677,8 @@ zig-test --no-color
665	677	\| `--retry <N>` \| `--retries <N>` \| Retry each failed test up to N times \|
666	678	\| `--repeat <N>` \| \| Require N successful runs of each selected test \|
667	679	\| `--fail-on-flaky` \| \| Fail the run when a retry recovers a failure \|
	680	\| `--shuffle` \| \| Randomize suite, test, and discovered-file order \|
	681	\| `--seed <N>` \| \| Replay a randomized order; implies `--shuffle` \|
668	682	\| `--verbose` \| \| Enable verbose output \|
669	683	\| `--quiet` \| `-q` \| Minimal output \|
670	684	\| `--no-color` \| \| Disable colored output \|

docs/execution-pipeline.mdmodified+6-2

Changes to docs/execution-pipeline.md
		@@ -34,6 +34,9 @@ reporters, including the configured JUnit output path.
34	34	child process.
35	35	- Every logical result owns ordered attempt history. A failure followed by a
36	36	successful retry is classified as `flaky`, distinct from an ordinary pass.
	37	- Optional randomized execution uses one run seed across the selected plan.
	38	The seed is carried by reporters and `TestResults`; see
	39	[Randomized test order](randomized-order.md) for mode-specific guarantees.
37	40	- Summaries always come from `TestResults`, rather than executor-local counters.
38	41
39	42	## Extension points
		@@ -53,10 +56,11 @@ model without coupling the execution API to JSON.
53	56	Existing programmatic APIs remain source compatible:
54	57
55	58	- `TestRunner.init`, `runTests`, and `runTestsWithOptions` are unchanged.
56		- `RunnerOptions` gains optional reporter, timeout, retry, repeat, and flaky-exit
	59	- `RunnerOptions` gains optional reporter, timeout, retry, repeat, flaky-exit,
	60	shuffle, and seed
57	61	fields with backward-compatible defaults.
58	62	- `runDiscoveredTests` and `LoaderOptions` remain available; optional reporter,
59		JUnit path, timeout, retry, repeat, flaky-exit, and writer fields default to
	63	JUnit path, timeout, retry, repeat, flaky-exit, shuffle, seed, and writer fields default to
60	64	the previous spec behavior.
61	65
62	66	CLI hosts can inject a stdout writer with `reporter_writer`. Embedded callers

docs/json-event-protocol.mdmodified+4-2

Changes to docs/json-event-protocol.md
		@@ -35,8 +35,8 @@ events are emitted in attempt order immediately before their `test_end`.
35	35
36	36	\| Type \| Data fields \|
37	37	\| --- \| --- \|
38		\| `run_start` \| `total` \|
39		\| `run_end` \| `total`, `passed`, `flaky`, `failed`, `skipped`, `duration_ns` \|
	38	\| `run_start` \| `total`, optional `random_seed` \|
	39	\| `run_end` \| `total`, `passed`, `flaky`, `failed`, `skipped`, `duration_ns`, optional `random_seed` \|
40	40	\| `suite_start`, `suite_end` \| `name` \|
41	41	\| `test_start` \| `name`, optional `suite` \|
42	42	\| `test_end` \| `name`, optional `suite`, `status`, `duration_ns`, optional `error_message` \|
		@@ -51,6 +51,8 @@ Output streams are `stdout` or `stderr`.
51	51
52	52	Optional fields are serialized as `null` when no value is available. Durations
53	53	are unsigned integer nanoseconds. Percentages are numbers from 0 through 100.
	54	When randomized ordering is enabled, both run events carry the reproducible
	55	unsigned integer seed.
54	56
55	57	## Compatibility policy
56	58

docs/randomized-order.mdadded+58-0

Changes to docs/randomized-order.md
		@@ -0,0 +1,58 @@
	1	# Randomized test order
	2
	3	Randomized order is opt-in. Use `--shuffle` to generate a seed for the run:
	4
	5	```bash
	6	zig-test --shuffle
	7	```
	8
	9	The selected reporter prints the generated seed at run start. If a test fails,
	10	the human-readable reporters print a replay command as well. Pass that seed to
	11	reproduce the order; `--seed` implies `--shuffle`:
	12
	13	```bash
	14	zig-test --seed 8675309
	15	```
	16
	17	Keep every other order-affecting option the same when replaying a run, including
	18	the test set, filter, shard, execution mode, and suite structure.
	19
	20	## Ordering guarantees
	21
	22	Without `--shuffle` or `--seed`, execution retains declaration or discovery
	23	order.
	24
	25	In sequential programmatic execution, a seed deterministically orders root
	26	suites, each suite's direct tests, and its nested suites. Lifecycle hooks keep
	27	their normal scope: `beforeAll` and `afterAll` still wrap a suite, while
	28	`beforeEach` and `afterEach` still wrap their selected test.
	29
	30	In parallel programmatic execution, the seed determines suite traversal, each
	31	worker queue, and serialized reporter output. Workers claim tests from that
	32	queue in order, but operating-system scheduling means wall-clock start and
	33	completion timing can differ between runs. Reporter order remains reproducible.
	34
	35	In discovery mode, sharding selects files first and randomization orders the
	36	files inside the selected shard. The seed does not change shard membership.
	37	Retries and repetitions stay attached to their logical test or file and do not
	38	trigger another shuffle.
	39
	40	## Metadata
	41
	42	The seed is stored as `random_seed` in JSON `run_start` and `run_end` events,
	43	live UI events, and test-history files. JUnit output records it as a
	44	`random_seed` property on each test suite. A non-randomized run reports a null
	45	seed in JSON-compatible formats and omits the JUnit property.
	46
	47	Programmatic callers can use `RunnerOptions.shuffle` and `RunnerOptions.seed`,
	48	or the matching `LoaderOptions` fields. JSON configuration accepts the same
	49	values under `test`:
	50
	51	```json
	52	{
	53	"test": {
	54	"shuffle": true,
	55	"seed": 8675309
	56	}
	57	}
	58	```