ReviewOS

home-lang/home

test(ts): admit checked-JS benchmarks with direct JSDoc negative controls

#852
Closed chris opened this 8 days ago ·
8 days ago

Scope and evidence

Follow-up to #416/#838: the current20-workload matrix records140 admissions, but checkedJS/JSDoc is currently positive-only. The workload exercises Model/Box/Project typedefs, generic preservation, constructor annotations and callback reads. Timing a positive program alone cannot prove these types are checked rather than erased to any.

Work

Add logical first/middle/last negative mutations to the unchanged workload: annotated model property assignment, missing model member, callback return assignment, constructor-method callback argument, and nested generic value mismatch. Probe the byte-identical accepted Home and pinned TS6/nativeTS7/Rust/Bun under identical checking options, retain all raw diagnostics/exits and exact inputs. Do not weaken controls to accommodate an implementation. A failing checker cannot retain a timing row in a new fair matrix.

Acceptance

  • Existing positive inputs/options remain byte-identical; negative projects are separate copies and never timed.
  • Every compared compiler passes exact required diagnostic codes and normal error exit classes, with success-output policies preserved.
  • Harness/schema/reporting/evidence tests ensure this new gate cannot be skipped or mistaken for an older admission protocol.
  • Any Home failure is tracked as a compiler correctness gap, not hidden or converted to success.
  • README/docs explicitly qualify historical positive-only checked-JS rows; publish new evidence and update the current matrix only when full revised admission passes.
8 days ago

Protocol4 landed on main in3491cdb45a78a9b21db5602e5726957b06e7260c, pushed as Chris without coauthors. The direct combined15-error JSDoc control covers model/missing-member/callback-return/instance-argument/nested-generic cases at first/middle/last families; all5 pinned compilers pass all5 isolated controls and the combined gate (35 retained untimed records). Positive programs/options are byte-identical, negative copies never timed, fingerprints stable. Reporting now marks older JSDoc rows provisional rather than relabeling them; protocol4 requires retained admissions for base3 and extra-compiler modes, normal exits, exact15-error contract, complete rotating-order coverage, and cannot masquerade as a legacy sequential schedule. Full harness136/136 passes; failed-preflight tests now assert retained failure artifacts and zero timings. A fresh all20,30-round,3-warmup,5-compiler matrix is running on unchanged native c16c8ca with protocol4. Expected145 admissions (the additional JSDoc negative per compiler),600 rounds/3000 samples, unchanged3840 guard/disk floors. No new timing/ranking claim yet.

8 days ago

Completed the scoped admission work on origin/main, as Chris without coauthors: 3491cdb for protocol4/gates/tests and 6004a26 for the fresh complete matrix/profile evidence and README/docs. All five pinned compilers pass five isolated and one combined JSDoc control (35 process records), then all145 admissions before the fresh20-workload/600-round/3000-sample matrix20261008T213919Z. New protocol4 requires all15 exact diagnostics and normal error exits, retained admission for base3 and added compilers, complete rotating order, and cannot claim a legacy sequential protocol. Older positive-only JSDoc rows are explicitly provisional, never relabeled. Full136 harness tests and targeted Pickier/diff checks pass.

Current result: Home1/20 first medians, ahead of nativeTS7 on12/20; Bun18 leads, Rust imports. No causal cross-run or universal-win claim. Matrix archive d1f7b629929bdc7214bbdaeee3fcc5c79571f21d4808a38a6f3377160e6c2ca0,603 verified checksums. Admission/profile context db598c898ebbd9ef69ad3b3e5a043049780086ac51e237c150cb9c812aa9d5c0,50 verified checksums; complete exact fixture bytes and raw output retained. Matrix peak380 MB/profile308 MB under the unchanged3840 guard/disk floors. Compiler runtime remains byte-pinned c16c8ca; this is stronger admission, not a compiler speedup.

Report: https://github.com/home-lang/home/blob/main/docs/docs/TS_PERFORMANCE.md#jsdoc-rejection-gated-matrix-2026-10-08-utc

The accepted compiler diagnostic-only8192-family sample passes58 image admissions, identical original/renamed instruction bytes and stable fingerprints. It points to declaration-space and late checkSourceFile work, not a guessed cause; #853 owns the next profile-driven optimization. #416 full leadership, #838 canonical C++/platform scope and #548 production Zod correctness remain open.

Sign in to comment on this issue.