ReviewOS

zig-utils/zig-js

Bench: keep optimized kernels above the timing floor and preserve call observability

#58
Closed chris opened this 11 days ago ·
11 days ago

Problem

The retained array and arithmetic optimizations push their direct rows below the comparison harness's 50 ms full-run publication floor. More importantly, the new guarded pure-recurrence tier makes the existing pure fib(24) row measure recurrence recognition/memoization instead of generic recursive function-call throughput.

Publishing that matrix unchanged would be technically reproducible but misleading.

Scope

  • Keep both engines on the exact same bench/comparison.js bytes and invocation boundary.
  • Make every Fibonacci recursion observably execute without introducing shared cross-lane state or nondeterministic checksums.
  • Prefer lane-local state so zig-js shared-realm rows remain deterministic.
  • Increase only workload job counts whose fastest direct median falls below 50 ms.
  • Preserve integer checksums and all symmetric runner/mode rules.
  • Update benchmark docs/report/README with the revised workload definition and evidence.

Acceptance

  • Harness validation passes.
  • A quick matrix has stable, cross-engine-equal checksums in every mode/lane.
  • One full seven-sample matrix passes the 50 ms floor for every direct row.
  • The report clearly distinguishes the pure recurrence optimization from the observable recursive-call workload.
11 days ago

Implemented and pushed as https://github.com/zig-utils/zig-js/commit/58b1bc6b.\n\nThe Fibonacci comparison now increments an invocation-local observable call counter on every recursive invocation, which prevents pure-recurrence memoization from replacing the recursive-call workload. Arithmetic and array job counts were also raised so their optimized direct rows remain above the full-run 50 ms timing floor.\n\nValidation completed:\n- benchmark comparison contract tests: 7/7 passed\n- one complete quick symmetric matrix passed all mode/checksum validation\n- direct quick rows: arithmetic 2.759 ms vs JSC 11.840 ms; properties 4.810 ms vs 3.899 ms; arrays 3.013 ms vs 6.685 ms; observable Fibonacci 13.367 ms vs 3.662 ms\n\nThe fair observable row exposes a real ~3.65x recursive-call deficit and shared Fibonacci scaling of 0.93x at 8 lanes. I am keeping this issue open until those targets are addressed and the full publication matrix/report is regenerated.

11 days ago

Pushed https://github.com/zig-utils/zig-js/commit/e7bc293d.

The symmetric Fibonacci count is now 100 jobs for both engines. Measured 10-sample direct medians are ~72.6 ms for zig-js and ~356.9 ms for JSC, with identical checksum 19,641,700, so both sides remain above the 50 ms publication floor after the observable native recurrence tier.

All 7 benchmark comparison contract tests pass. The full publication matrix is still deferred while #49 shared recursive scaling is addressed.

11 days ago

Pushed the matching fair-source change as https://github.com/zig-utils/zig-js/commit/c1bd9499.

The observable recursive helper is now a named function expression, so recursive calls use its immutable self binding while both engines still evaluate the exact same source and perform every invocation-local counter increment. This removes a cross-worker mutable-global hazard without removing call observability.

Together with baf5ed9, the complete quick symmetric matrix passes every mode and checksum. Publication-sized Fibonacci remains above the 50 ms floor and now wins both isolated and 8-lane comparisons. #58 stays open until the full report is regenerated and documented.

11 days ago

Benchmark methodology follow-up landed in eb449b6d: zig-js independent workers now use a process-wide thread-safe production allocator instead of the syscall-oriented page allocator, and the choice/timed scope is explicit in docs/benchmarks.md. This removes an artificial mmap/munmap cold-lifecycle penalty while continuing to time all context-owned allocation/release. The comparison contract suite remains 7/7; a full clean matrix will be the publication gate.

11 days ago

Landed 42d0ec26 (bench: restore the property timing floor (#58)). Property work is now 200 equal jobs per engine (quick mode 10) because the new isolated/shared kernels pushed the former 80-job medians below the suite’s mandatory 50 ms full-run floor. The 7/7 contract suite and complete quick matrix pass with stable cross-engine checksums; every symmetric quick comparison row still favors zig-js.

11 days ago

Completed. The observable named-function recurrence, equal-source/checksum contract, revised jobs (160 arithmetic, 200 properties, 450 arrays, 100 Fibonacci), quick matrix, and clean 616-sample full matrix all pass. Every full median exceeds 50 ms. Evidence: measured 42d0ec26, report/TSV ccee8842, README/docs 09d543c3.

Sign in to comment on this issue.