Add retries, repeated runs, and flaky-test reporting

docs/retries-and-repeats.mdadded+92-0

Changes to docs/retries-and-repeats.md

		@@ -0,0 +1,92 @@
	1	+# Retries, repeats, and flaky tests
	2	+
	3	+The runner records each execution of a logical test as an attempt. An attempt
	4	+contains its one-based number, repetition number, status, duration, and error
	5	+message. Registered tests and discovered test files use the same model.
	6	+
	7	+## Command-line use
	8	+
	9	+```bash
	10	+# One initial attempt plus as many as two retries after failure
	11	+zig-test --test-dir tests --retry 2
	12	+
	13	+# Require every selected test to pass 100 times
	14	+zig-test --test-dir tests --filter parser --repeat 100
	15	+
	16	+# Make a recovered failure fail the overall run in CI
	17	+zig-test --test-dir tests --retry 2 --fail-on-flaky
	18	+```
	19	+
	20	+`--retries` is an alias for `--retry`. A retry count is the number of
	21	+*additional* attempts after a failure. `--repeat N` requires N successful
	22	+repetitions; each repetition receives its own retry budget. Attempt history is
	23	+ordered across all repetitions.
	24	+
	25	+Discovery operates at test-file granularity because each file is an external
	26	+`zig test` process. Programmatic mode operates at individual-test granularity.
	27	+
	28	+## Outcomes and exit status
	29	+
	30	+- `passed`: every attempt passed without a retry.
	31	+- `flaky`: at least one attempt failed, but a retry recovered and all required
	32	+ repetitions eventually passed.
	33	+- `failed`: one repetition exhausted its retry budget.
	34	+- `skipped`: the test was not selected for execution.
	35	+
	36	+Failed tests always make the process exit unsuccessfully. Flaky tests succeed
	37	+by default so retries can recover an intermittent failure. `--fail-on-flaky`
	38	+changes that policy and returns an unsuccessful exit status when any test is
	39	+flaky.
	40	+
	41	+`--bail` is evaluated after retries, so one failed attempt does not stop the
	42	+run. The runner stops scheduling only after a logical test exhausts its retry
	43	+budget. The timeout is evaluated independently for every attempt; a timed-out
	44	+attempt may be retried.
	45	+
	46	+When retries or repeats are configured for registered tests, execution is
	47	+sequential even if parallel mode was requested. This preserves deterministic
	48	+attempt ordering and hook behavior.
	49	+
	50	+## Programmatic configuration
	51	+
	52	+Set defaults for a registered-test run with `RunnerOptions`:
	53	+
	54	+```zig
	55	+const options = ztf.RunnerOptions{
	56	+ .retries = 2,
	57	+ .repeat = 10,
	58	+ .fail_on_flaky = true,
	59	+};
	60	+const passed = try ztf.runTestsWithOptions(allocator, registry, options);
	61	+```
	62	+
	63	+An individual registered test can override the run-level retry count:
	64	+
	65	+```zig
	66	+try suite.addTest(
	67	+ ztf.TestCase.init("eventually consistent", testEventuallyConsistent)
	68	+ .withRetries(5),
	69	+);
	70	+```
	71	+
	72	+`LoaderOptions` exposes the same `retries`, `repeat`, and `fail_on_flaky`
	73	+fields for callers that run discovered files directly. Strict JSON
	74	+configuration uses the corresponding fields in the `test` object:
	75	+
	76	+```json
	77	+{
	78	+ "test": {
	79	+ "retries": 2,
	80	+ "repeat": 10,
	81	+ "fail_on_flaky": true
	82	+ }
	83	+}
	84	+```
	85	+
	86	+## Reporter data
	87	+
	88	+The spec and dot reporters show flaky results separately from passes. JSON test
	89	+objects include an `attempts` array with `number`, `repetition`, `status`, and
	90	+`durationNs` fields plus an optional `error`. The JSON summary includes a
	91	+`flaky` count. JUnit output includes the flaky count on `<testsuites>` and emits
	92	+ordered attempt data in each retried test case's `<system-out>` element.