ReviewOS

home-lang/home

bench(ts): add checked JavaScript JSDoc workload

#448
Closed chris opened this 8 days ago ·
8 days ago

Context

The accepted TypeScript frontend suite has thirteen workloads and Home records the lowest mean on all of them, but every workload is TypeScript or TSX. The suite does not isolate checked-JavaScript parsing, JSDoc type-expression lowering, JSDoc generic inference, callback typing, or checkJs value-flow work.

Proposed fair workload

Add one deterministic, dependency-free JavaScript module containing repeated independent families with:

  • @typedef object shapes and nested @property declarations;
  • constrained @template generic functions;
  • @callback, @param, and @returns annotations;
  • JSDoc-typed classes and instance method calls;
  • object construction, callback invocation, property reads, and explicitly typed result consumption that keep every family live.

The shared project enables allowJs, checkJs, strict, noLib, skipLibCheck, pretty: false, and noEmit. JavaScript TypeScript 6.0.3, native TypeScript 7.0.2, and Home must receive the exact same source and tsconfig, exit successfully, and produce no output before any timing is accepted.

Acceptance criteria

  • Open and link this issue before implementing the workload.
  • Commit the validated workload before measuring or optimizing it; do not alter it in response to results.
  • Keep the generator deterministic, auditable, dependency-free, and distinct from existing TS/TSX workloads.
  • Record generated source size and SHA-256 before the first accepted measurement.
  • Record an interleaved pre-optimization baseline on the frozen source.
  • If Home loses, profile the unchanged workload and make only general compiler improvements—no corpus detection, benchmark-only paths, relaxed checking, or compiler-specific options.
  • Add focused correctness coverage for any semantic fix and run the complete affected package suites.
  • Publish a 30-run result in the detailed performance document and root README table.
  • Update and link #416 with workload, optimization, validation, and documentation commits.

Refs #416

8 days ago

Pre-freeze validity gate passed.

  • Shape: one dependency-free checked-JavaScript module with 128 independent JSDoc families.
  • Generated source: 8,321 lines, 168,405 bytes.
  • SHA-256: 898573fc0f2567653fef3edb3675d768f825abd80b83e708e7d57d29049c03ec.
  • Shared options: allowJs, checkJs, strict, noLib, skipLibCheck, pretty: false, and noEmit.
  • JavaScript TypeScript 6.0.3, native TypeScript 7.0.2, and Home all exit 0 with empty stdout and stderr on the identical project.
  • Generator compilation and Pickier checks pass.

No benchmark result has been measured or accepted yet. The next commit freezes this exact source and configuration before timing.

Refs #416

8 days ago

The validated workload is frozen on main before timing:

  • commit: dd0968992
  • source: 8,321 lines, 168,405 bytes
  • SHA-256: 898573fc0f2567653fef3edb3675d768f825abd80b83e708e7d57d29049c03ec
  • exact compiler pins: JavaScript TypeScript 6.0.3 and native TypeScript 7.0.2
  • validity: all three compilers exit 0 with empty stdout/stderr on the same source and tsconfig

The generator, configuration, and generated source will remain unchanged while the frozen benchmark is measured and, if necessary, optimized.

Refs #416

8 days ago

The first 5-run timing (20260827T090859Z) is rejected and must not be treated as a Home win.

It reported Home at 5.6 ± 1.1 ms versus native TypeScript 7.0.2 at 105.0 ± 4.5 ms, which was implausibly close to startup. A negative-control copy changed the first JSDoc-declared numeric id to a string:

compilerexitdiagnostic
tsc 6.0.32TS2322 at the mutated property
native TS 7.0.21TS2322 at the mutated property
Home0none

Therefore silent success did not prove semantic participation: Home is currently skipping or failing to check the .js project path. The frozen source remains unchanged, no timing is accepted, and this issue now tracks the general checked-JavaScript project/CLI correctness gap before performance can be measured fairly.

Refs #416

8 days ago

Correctness gate restored and pushed in 5a89c14c7 (fix(tsc): check JavaScript project inputs).

Root cause: config glob discovery admitted only TypeScript-shaped inputs, so the initial Home timing checked lib.d.ts but silently omitted the frozen 8,321-line .js source. The fix now:

  • admits .js, .jsx, .mjs, and .cjs during every project discovery path only when allowJs is enabled;
  • propagates allowJs, checkJs, and noEmit from tsconfig into compilation;
  • preserves effective noEmit through normal and watch emission paths.

Validation:

  • zig build test -Dfilter=ts_driver passes;
  • zig build test -Dfilter=ts_cli passes;
  • Pickier and Zig formatting pass;
  • unchanged frozen corpus is clean;
  • one-line negative control (Model0.id: string where JSDoc requires number) is rejected by all three with TS2322: tsc 6.0.3 exit 2, native TypeScript 7.0.2 exit 1, Home exit 1.

The earlier 5.6 ms Home timing remains rejected. The real Home checked-JavaScript run now takes minutes, exposing the performance gap this issue is intended to measure. The frozen workload remains unchanged while I profile and optimize the general checker path.

8 days ago

First general checker optimization pushed in c1818b32a (perf(ts): cache JSDoc function ownership).

A macOS sample of the valid reduced workload showed a cubic pattern: repeated JSDoc typedef lookups rescanned every comment, and each comment repeatedly walked the entire HIR to rediscover its enclosing function. The checker now caches those immutable source-position containment results for the source lifetime.

Measured on the same generated profiling copy (not a benchmark corpus change):

JSDoc familiesBeforeAfter
160.36 s0.16 s
322.07 s0.66 s
64~14 s2.54 s

The growth drops from roughly cubic to quadratic, but Home is still far behind native TypeScript here. Validation: JSDoc-focused checker tests, Pickier/Zig formatting, and the full checker suite all pass. The frozen benchmark is unchanged; next I am removing the residual repeated whole-source typedef scans with a source-level index.

8 days ago

Implemented and pushed to main in fb72502, with the published benchmark snapshot in 2fafb98. The frozen checkjs_jsdoc project is accepted by Home, tsc 6.0.3, and native TypeScript 7.0.2; a preserved invalid variant is rejected by all three with TS2322. The authoritative interleaved 30-run result (20260827T132029Z, 3 warmups) is Home 68.4 ± 4.7 ms, native TS 7.0.2 76.0 ± 5.5 ms, and tsc 6.0.3 322.2 ± 40.9 ms: Home is 1.11× faster than the fastest competitor. The full checker suite passes 4,191/4,191 and the focused JSDoc suite passes 183/183. The full fair suite is now 14/14 Home wins. Detailed methodology and results are linked from the root README.

Sign in to comment on this issue.