ReviewOS

home-lang/home

bench(ts): add null-safe access workload

#451
Closed chris opened this 8 days ago ·
8 days ago

Goal

Add a deterministic TypeScript frontend benchmark for common null-safe access patterns that are not represented by the existing suite:

  • optional property access
  • optional element access
  • optional calls
  • nullish coalescing
  • non-null assertions
  • nested nullable object/tuple reads with typed consumption

Fairness gates

  • The same generated project and tsconfig are passed to Home, JavaScript tsc 6.0.3, and native TypeScript/tsgo 7.0.2.
  • All compilers must accept the valid project silently before timing.
  • A preserved invalid variant must be rejected by all three with the same diagnostic family.
  • The generated source is frozen and hashed before profiling or optimization.
  • No benchmark-name, generated-symbol, or family-count special casing is permitted.
  • Optimization must be general checker/parser work and retain the full ts_checker gate.

Tracked under #416.

8 days ago

Benchmark corpus frozen on main in 56cb9c1 before timing or optimization.

Accepted source:

  • 256 independent nullable-object families
  • 8,192 lines, 245,538 bytes
  • SHA-256 22559101b63ff7129383baa604a006d33a37cd0ba4aa499a849ffd26894ff57c

Validity gate:

  • tsc 6.0.3: valid project accepted silently
  • native TypeScript/tsgo 7.0.2: valid project accepted silently
  • Home 0.1.0: valid project accepted silently
  • preserved invalid tuple string-to-number assignment: all three reject with TS2322

Next: capture the unoptimized interleaved baseline, then profile only general compiler paths.

8 days ago

Pre-baseline full-suite validation caught and corrected an accidental one-line change to the already-frozen type_predicates generator. Commit 12c311b restores its documented SHA-256 8b04ff5b9ae889c940facb652e5c925c7ff2aa6350058292208ca29020192350. The new null_safe_access source remains unchanged at SHA-256 22559101b63ff7129383baa604a006d33a37cd0ba4aa499a849ffd26894ff57c. Both workloads again validate silently on all three compilers. No null-safe timing or compiler optimization occurred before this correction.

8 days ago

Frozen red baseline captured in 20260827T152206Z (5 round-robin interleaved runs after 3 warmups):

  • tsc 6.0.3: 209.2 ± 13.3 ms
  • native TypeScript/tsgo 7.0.2: 62.1 ± 1.6 ms
  • Home 0.1.0: 1166.8 ± 11.0 ms
  • Home is 18.79× slower than the fastest competitor.

The other 15 workloads remain Home wins in the same run. Profiling will use scaled copies of the exact same generated shape; the accepted 256-family source and hash remain unchanged.

8 days ago

Completed and published on main:

  • 56cb9c1 adds the deterministic null-safe-access workload.
  • 12c311b restores the unrelated frozen predicate-generator literal after full-suite preflight caught the accidental edit.
  • fa78423 adds the general lexical/source parameter-recovery guard and regression test.
  • ad94381 publishes the full unfiltered results and historical evidence.

The unchanged accepted workload improves from 1166.8 ± 11.0 ms in red baseline 20260827T152206Z to 51.1 ± 14.1 ms in authoritative 30-run snapshot 20260827T154256Z (22.83× Home improvement, 1.40× faster than native TypeScript 7.0.2 at 71.3 ± 26.9 ms). All 4,192 checker tests pass; valid/invalid gates and the frozen source hash are preserved.

The full snapshot is honestly documented as 15/16 wins, because checked JavaScript has an unfiltered mean loss. Follow-up #452 tracks that remaining margin.

Sign in to comment on this issue.