ReviewOS

home-lang/home

bench(ts): add structural object compatibility workload

#443
Closed chris opened this 24 days ago · 4 comments
24 days ago

Context

The accepted TypeScript frontend suite now leads across ten workloads, including type-level transforms, calls, control flow, overloads, and class inheritance. It does not isolate TypeScript's central structural compatibility path for plain object types: nested properties, optional and readonly members, generic function properties, tuples, intersections, and repeated source-to-target assignment.

Proposed fair workload

Add one deterministic, dependency-free TypeScript module containing repeated independent structural object families with:

  • explicit source object types and distinct target interfaces/type aliases;
  • nested required, optional, and readonly properties;
  • tuple and intersection members;
  • generic function properties whose parameters and returns are consumed;
  • valid assignments and typed function arguments that force source-to-target compatibility; and
  • typed exports that retain every family.

All three compilers must receive the exact same source and tsconfig, accept it successfully, and produce no output under --noEmit before timing. Compiler versions remain JavaScript tsc 6.0.3 and native TypeScript 7.0.2.

Acceptance criteria

  • Commit the workload before any optimization and do not change it in response to results.
  • Keep the shape deterministic, auditable, dependency-free, and distinct from existing workloads.
  • Record source size/hash and an interleaved red/green baseline.
  • If Home loses, profile the unchanged workload and make only general compiler improvements.
  • Run focused tests plus the complete affected package suites.
  • Publish a 30-run result in the detailed performance document and README summary.
  • Update #416 with commits and results.

Refs #416

24 days ago

Pre-optimization workload shape is now fixed:

  • one deterministic module
  • 128 independent structural source/target families
  • 5,632 lines, 219,950 bytes
  • SHA-256 a5f25c7533e69df663e3d84e0d785374255c01b340d0aef5516bc1727833e996
  • nested required-to-optional compatibility, readonly targets, tuples, intersections, generic function properties, excess source members, direct assignments, and typed argument passing
  • tsc 6.0.3, native TS 7.0.2, and Home 0.1.0 all accept the exact generated project silently
  • hash is stable across regeneration

The workload will be committed before the first timed baseline and will remain unchanged afterward.

24 days ago

Frozen pre-optimization baseline recorded from 3946ef904.

Result ID: 20260826T234904Z (5 interleaved runs after 1 warmup)

  • tsc 6.0.3: 198.4 ± 2.7 ms
  • native TS 7.0.2: 62.7 ± 2.7 ms
  • Home 0.1.0: 160.9 ± 4.5 ms
  • outcome: Home is 2.57× slower than the fastest competitor
  • the previously accepted ten workloads remain Home wins

The committed source and configuration are now frozen. Next step is profiling and general checker optimization against this unchanged workload.

24 days ago

First optimized gate on the unchanged workload (20260826T235512Z):

  • Home: 47.5 ± 0.3 ms
  • native TS 7.0.2: 56.2 ± 1.2 ms
  • tsc 6.0.3: 179.6 ± 0.8 ms
  • outcome: Home 1.19× faster than the fastest competitor
  • Home improvement from frozen baseline: 3.39×
  • complete accepted suite: 11/11 Home wins in the gate

Profile finding: repeated sourceLibDirectiveExcludesDomElement calls rescanned the full source for reference lib directives while lowering many annotations. The general fix memoizes the unchanged directive result in the existing per-source SourceFacts lifecycle.

Validation:

  • focused reference-lib tests passed
  • focused DOM-lib tests passed
  • full TypeScript checker suite: 4,190/4,190 passed
  • Pickier and git diff --check passed
24 days ago

Completed on main.

  • frozen workload: 3946ef904
  • optimized implementation: 617d29ead
  • published docs: 61e38efa3
  • full checker suite: 4,190/4,190 passed
  • authoritative 30-run / 3-warmup result: 20260827T000322Z
  • structural objects: tsc 6.0.3 180.6 ± 2.5 ms, native TS 7.0.2 57.0 ± 1.1 ms, Home 47.5 ± 0.8 ms — Home 1.20× faster
  • complete accepted suite: 11/11 Home wins

The source and compiler options were unchanged from the recorded red baseline. The general fix memoizes DOM-library availability in the existing per-source facts rather than rescanning the entire source while lowering annotations.

Sign in to comment on this issue.