ReviewOS

stacks/bun-router

test(perf): replace wall-clock optimizer assertion

#901
Closed chris opened this 8 days ago ·
8 days ago

Problem

The main CI run for release-prep commit 84fc358 failed because test/bun-optimizations.test.ts compares two sub-millisecond wall-clock samples and requires memoization to beat a tiny arithmetic loop in every run.

The failing sample measured the optimized loop at 0.061621 ms and the memoized loop at 0.245548 ms. The implementation revision immediately before it passed the same full suite, and the release-note commit changed only CHANGELOG.md.

Failure: https://github.com/stacksjs/bun-router/actions/runs/35571199259

Expected

Performance tests should verify deterministic optimization behavior or use a statistically sound benchmark outside the correctness suite. A unit test should not fail based on scheduler and timer noise at this scale.

Scope

  • inspect what the optimizer contract actually promises,
  • replace the one-sample timing comparison with deterministic behavior coverage,
  • keep real measurements in the benchmark harness,
  • validate on exact Bun 1.4.2 and link the green main CI run.
8 days ago

Fixed in 6ac8f75.

The test now verifies the optimizer's actual contract: 100 calls spanning 10 argument values return the expected results while invoking the wrapped function exactly 10 times. It no longer compares scheduler-sensitive sub-millisecond samples.

Exact Bun 1.4.2 validation:

  • focused test passed 100 consecutive processes,
  • full suite passed 1,007 tests with 0 failures,
  • standard and registry typechecks passed,
  • pickier lint passed.

I will close this issue after the exact main CI run is green.

8 days ago

The exact fix revision passed the full main CI workflow on Bun 1.4.2: https://github.com/stacksjs/bun-router/actions/runs/35571397050

All 1,007 tests passed in CI, along with lint, both typechecks, the build, and commit publication. The deterministic memoization test is complete.

Sign in to comment on this issue.