ReviewOS

home-lang/home

bench(ts): add overload resolution workload

#439
Closed chris opened this 24 days ago · 3 comments
24 days ago

Context

The accepted TypeScript frontend suite covers startup, file scaling, deep types, import/re-export graphs, TSX, generic calls, and control-flow narrowing, but it does not isolate overload candidate selection. This leaves a distinct checker cost unmeasured.

Proposed fair workload

Add one deterministic, dependency-free TypeScript module containing:

  • a fixed set of overload signatures distinguished by literal discriminants and structurally different payloads;
  • generic literal-ID inference through every overload;
  • contextual typing for object, tuple, nested-object, and callback arguments;
  • typed result assignments that require each chosen overload to preserve its discriminant, ID, and payload type; and
  • repeated call groups large enough to measure candidate selection rather than process startup.

All three compilers must receive the exact same source and tsconfig, accept it successfully, and produce no output under --noEmit before timing. Compiler versions remain tsc 6.0.3 and native TypeScript 7.0.2.

Acceptance criteria

  • Commit the workload before any optimization and do not change it in response to results.
  • Keep the shape logical, auditable, and meaningfully distinct from generic_calls.
  • Record an interleaved red/green baseline.
  • If Home loses, profile the unchanged workload and make only general compiler improvements.
  • Run focused tests plus the complete affected package suites.
  • Publish a 30-run result in the detailed performance document and README summary.
  • Update #416 with commits and results.

Refs #416

24 days ago

Committed the unchanged workload in be6e9da86.

Cross-validation:

  • tsc 6.0.3: success, silent
  • native TypeScript 7.0.2: success, silent
  • Home 0.1.0: success, silent
  • deterministic regeneration: SHA-256 stable
  • shape: one 1,679-line, 147,302-byte module; 128 groups × 8 calls = 1,024 overload calls

Five-run red baseline after two warmups (20260826T221712Z):

CompilerMeanStddev
tsc 6.0.3222.5 ms2.1 ms
native TypeScript 7.0.270.3 ms1.1 ms
Home126.9 ms0.9 ms

Home is 1.81x slower than the fastest competitor. The previous eight workloads remain Home wins. The committed workload and compiler configuration are now frozen while profiling proceeds.

Refs #416

24 days ago

Winning candidate validation:

  • Frozen workload unchanged; tsc 6.0.3, native TypeScript 7.0.2, and Home all accept it silently.
  • Five-run interleaved gate (20260826T231118Z): Home 61.6 ± 0.6 ms, TypeScript 7.0.2 67.8 ± 2.0 ms, tsc 6.0.3 212.9 ± 4.9 ms. Home is 1.10x faster than the fastest competitor.
  • All nine benchmark workloads are green in the same run.
  • Full checker validation: 4,190/4,190 tests pass.

The implementation conservatively rejects exact-literal overload mismatches before generic inference, indexes first simple lexical bindings per virtual source section, reuses cached annotated types only for ordinary reads, and retains the full declared-type path for assignment/flow semantics. Commit and 30-run publication evidence to follow.

24 days ago

Completed without changing the frozen workload.

Commits:

  • be6e9da86 — workload landed before optimization
  • 9c6a93b3f — general overload/lexical-resolution optimization
  • a8d16bc09 — README and detailed benchmark publication

Final authoritative result (20260826T231722Z, Apple M3 Pro, 30 interleaved runs after three warmups):

CompilerMean ± sample SD
tsc 6.0.3207.2 ± 10.5 ms
native TypeScript 7.0.266.4 ± 3.7 ms
Home 0.1.058.1 ± 2.2 ms

Home is 1.14× faster than the fastest competitor. The red baseline was Home 126.9 ± 0.9 ms versus native TypeScript 7.0.2 at 70.3 ± 1.1 ms, so the unchanged workload improved 2.18×. All 4,190 checker tests pass, and the full accepted suite is 9/9 Home wins.

Sign in to comment on this issue.