ReviewOS

home-lang/home

bench(ts): add variadic tuple workload

#447
Closed chris opened this 8 days ago ·
8 days ago

Context

The accepted TypeScript frontend suite now leads across twelve workloads, but tuple coverage is incidental. It does not isolate variadic tuple construction, readonly tuple inference, conditional head/tail extraction, tuple indexing, generic rest parameters, or spread-call checking.

Proposed fair workload

Add one deterministic, dependency-free TypeScript module containing repeated independent tuple families with:

  • readonly literal tuples with distinct element shapes;
  • generic variadic tuple concatenation and inferred tuple results;
  • conditional head and tail extraction with infer and rest elements;
  • fixed and indexed reads from the composed tuple;
  • generic rest-parameter calls fed by tuple spreads;
  • typed structural consumption and exports retaining every family.

All three compilers must receive the exact same source and tsconfig, accept it successfully, and produce no output under --noEmit before timing. Compiler versions remain JavaScript tsc 6.0.3 and native TypeScript 7.0.2.

Acceptance criteria

  • Commit the workload before any optimization and do not change it in response to results.
  • Keep the shape deterministic, auditable, dependency-free, and distinct from existing workloads.
  • Record source size/hash and an interleaved red/green baseline.
  • If Home loses, profile the unchanged workload and make only general compiler improvements.
  • Run focused tests plus the complete affected package suites.
  • Publish a 30-run result in the detailed performance document and README summary.
  • Update #416 with commits and results.

Refs #416

8 days ago

Pre-freeze validity gate is complete.

  • The generated workload is deterministic: 5,898 lines, 188,399 bytes, SHA-256 78105a6b28024f5ac1dd32cc81d3aa065a3a99a1972927e1880f0a28eec5324f.
  • JavaScript TypeScript 6.0.3 (tsc), native TypeScript 7.0.2 (tsgo), and Home now all accept the identical strict/noLib project silently.
  • The initial gate correctly exposed missing generic variadic-spread materialization, trailing conditional tuple inference, and recursive infer-binder preservation. Project-level validation also exposed an unsafe object-member borrow during union reduction and a recursive generic-inference cycle; both were fixed generally rather than hidden by the workload.
  • Checker regression coverage now includes concat materialization, conditional head/tail slices, and generic rest capture. The full checker suite passes 4,191/4,191 tests.

Correctness fix: a05b922

No timing result has been accepted yet. The next commit freezes this validated source before measurement.

8 days ago

Frozen 5-run baseline (20260827T072451Z, 1 warmup):

compilermean ± sd
tsc 6.0.3344.8 ± 10.3 ms
native tsgo 7.0.2100.7 ± 3.3 ms
Home209.4 ± 4.8 ms

Home is currently 2.08× slower than the fastest competitor on the accepted variadic-tuple workload. This is the optimization target; no timing result is being published as a win yet.

8 days ago

Variadic-tuple benchmark round complete.

The generated project remained unchanged: 5,898 lines, 188,399 bytes, SHA-256 78105a6b28024f5ac1dd32cc81d3aa065a3a99a1972927e1880f0a28eec5324f. JavaScript TypeScript 6.0.3, native TypeScript 7.0.2, and Home all validate the identical strict/noLib project successfully and silently.

Measurementtsc 6.0.3native TS 7.0.2HomeHome vs fastest
frozen 5-run baseline 20260827T072451Z344.8 ± 10.3 ms100.7 ± 3.3 ms209.4 ± 4.8 ms2.08× slower
final 30-run result 20260827T085121Z634.1 ± 72.1 ms160.8 ± 14.0 ms153.5 ± 13.9 ms1.05× faster

The final snapshot retains all 30 unfiltered samples and discloses elevated background workstation variance. Compiler order rotated within every round. Home improved 1.36× on the unchanged workload and records lower mean time than both competitors; the accepted suite is now 13/13.

Validation: all 4,191 checker tests pass, the ReleaseFast compiler builds, Pickier reports no errors in the touched docs/code, and every benchmark validity gate is silent.

Refs #416

Sign in to comment on this issue.