ReviewOS

zig-utils/zig-test-framework

Support randomized test order with reproducible seeds

#14
Closed glennmichael123 opened this 9 days ago ·
9 days ago

Motivation

Randomized execution can expose order-dependent tests, but failures must be reproducible.

Proposed work

  • Add options to shuffle suites and tests.
  • Accept an explicit seed and generate one when omitted.
  • Print the seed at run start and on failure.
  • Preserve the exact ordering for a given seed.
  • Include the seed in JSON, JUnit properties, and test history.

Acceptance criteria

  • Re-running with the printed seed reproduces test order.
  • Sequential and parallel modes document their ordering guarantees.
  • Unit tests verify stable ordering for a fixed seed.
  • Shuffle remains opt-in by default.
9 days ago

Implemented in #29 and rebase-merged as dd87ffe. The pull-request workflow and the post-merge main workflow are both green across documentation, formatting, coverage, macOS, Ubuntu, and Windows.

Sign in to comment on this issue.