ReviewOS

home-lang/home

perf(ts): extend the fair compiler benchmark matrix

#838
Open glennmichael123 opened this 8 days ago ·
8 days ago

Published checkpoint (2026-10-08 UTC)

Current protocol-4 five-compiler matrix: 20261008T213919Z, source/harness 3491cdb45, compiler inputs db28b2e10, frozen native c16c8ca. Published in 6004a2634 as Chris without coauthors.

All 145 admissions, 600 original rounds / 3,000 retained samples, unchanged before/after fingerprints, and 603 matrix checksums verify. The direct JSDoc gate now requires15 exact errors across five semantic features at first/middle/last families before timing. Base3 modes also retain process evidence. Older positive-only JSDoc rows are explicitly provisional, never retroactively upgraded.

Home leads 1/20 (startup) and beats nativeTS7 on 12/20, using unrounded medians. Bun leads18; Rust imports. Current measured gaps: checkedJS2.14x, CommonJS2.12x, nullsafe2.01x, interfaces1.93x and TSX1.84x slower than fastest. These are descriptive comparisons, not causal patch claims or a universal lead. README10/full-doc20 rows exactly match raw observations.

Pins remain TS6.0.3, nativeTS7.0.2, Rust tsc-rs0.1.0/TS7.1-dev at72b339e4 and Bun1.4.3-canary.1+bd599f5. Both cached extra payloads verify original executable/inventory hashes; no moving-canary substitution. Matrix peak380 MB, host low-water30%, unchanged3840 MB guard/disk floors. Harness136/136, targeted Pickier/diff checks pass.

The additional admission/profile archive retains35 untimed isolated/combined control records, exact fixtures, diagnostic image58 admissions and an unchanged8192-family sample with equal original/renamed instruction bytes. Profile counts are exploratory, not latency or whole-run percentages; #853 owns source-level investigation. No compiler speed change is claimed by #852.

Matrix SHA-256: d1f7b629929bdc7214bbdaeee3fcc5c79571f21d4808a38a6f3377160e6c2ca0 (603 checksums). Admission/profile context SHA-256: db598c898ebbd9ef69ad3b3e5a043049780086ac51e237c150cb9c812aa9d5c0 (50 checksums).

C++ canonical selection remains outstanding. The source-pinned OnlySpecs candidate still fails all8 negative controls and has zero timings; other platforms and full production admission remain unverified. Zod still has195 TS+3HM9002, not production admission. #838/#416/#548 remain open.

Context

The current bench/vs_tsgo suite compares Home with pinned tsc and native TypeScript (tsgo) on identical admitted strict, no-emit projects. Result 20261007T183321Z is the current three-compiler baseline: 20 workloads, 30 rotating-order rounds after three warmups, 1,800 retained samples, and unchanged executable provenance before and after measurement.

Chris requested that the same benchmark also cover the emerging Rust and C++ TypeScript ports and Bun's check command. Numbers from screenshots, different machines, different corpora, or incompatible checking modes are context only and must not be merged into Home's table.

Related: #836, #837, #818.

Scope

  • Identify the canonical upstream repository, exact revision, license, build procedure, and supported CLI for:
    • Theo's Rust tsc-rs / ts-rust frontend;
    • the active C++ TypeScript port;
    • Bun's check command.
  • Extend the benchmark harness generically so an admitted competitor can be pinned and measured without weakening the existing tsc, tsgo, or Home controls.
  • Run candidates against the identical generated projects and equivalent check-only semantics. If a tool cannot perform the same work, report it separately rather than presenting an incomparable timing.
  • Preserve all diagnostics, exit codes, executable hashes, tool versions, warmups, samples, and raw measurement output.
  • Keep process order rotating on one machine, publish medians from every retained sample, and package deterministic checksum-covered evidence.
  • Document both successful admissions and exclusions, including the exact semantic or tooling reason for each exclusion.

Fairness requirements

  • Canonical upstreams and revisions are verified from primary sources.
  • Every included compiler accepts the same positive projects and matches the required negative-control diagnostic contract.
  • Compiler modes perform equivalent parse, bind, and type-check work with emission disabled.
  • Versions and executable/payload hashes are captured before admission and rechecked after measurement.
  • All included compilers run on the same host, in the same run, with the same warmups and repeat count.
  • No best-of selection, removed outliers, benchmark-specific shortcuts, test weakening, or copied third-party numbers.
  • Raw per-round artifacts and a deterministic evidence archive are retained.

Acceptance

  • The harness has regression tests for multi-competitor admission, provenance, reporting, and evidence packaging.
  • A fresh full-suite run publishes a traceable table for every fairly admitted compiler.
  • README and performance docs state versions, host, methodology, exclusions, result ID, and archive checksum.
  • Existing correctness gates and the complete Home checker suite remain green.
8 days ago

Primary-source competitor discovery and pinned payload checks are underway.

CandidateVerified upstream / pinCurrent status
Rust tsc-rspingdotgg/ts-rust v0.1.0, MITNative Darwin ARM64 archive and binary verified; executable reports Version 7.1.0-dev. Untimed admission queued.
Bun stablebun-v1.4.2The verified native binary returns exit 1, Script not found "check", for bun check --help. This release cannot be included as a type checker.
Bun canarybd599f5af912512b83bba0ef387d1ba3a7d5e550Binary reports 1.4.3-canary.1+bd599f5af and process.versions.typescript = 7.0.2; native check help is available. Untimed admission queued.
C++ portRepository selection outstandingTypeRunner and OnlySpecs are candidate search results, but neither is yet established as the active port meant here. Requested the intended GitHub URL while continuing the other checks.

Published archive SHA-256 values matched the downloaded bytes:

  • Rust Darwin archive: 96920e12262ebe390dfa4876c3cf8ed03aa594ce99222a00e0f1517e945ba205.
  • Bun 1.4.2 Darwin archive: 90987a3a16d7db556d886ac3d551e7b6d3edf0a1cf43acaed622e8676be1d12f.
  • Bun canary Darwin archive: 87f2d32f3189c67781e0cf81dac6ad91c014798317b078b2e0dbc3cba8c4bd09.

Native executable hashes:

  • Rust: 3a2736e002cdeaa7a7514adb9c5b2f367021fd8253d1fdfb4058f135ead04b49.
  • Bun canary: b0ac9bb272f8e04856d7723a04c3eeb8877a22c62f96f0f5392d94f2d4bf2e73.

The admission probe calls the existing run.validate on all 20 unchanged projects and their existing negative controls. It retains failures and fingerprints the executables, Rust lib payloads, validation code, manifest, and complete generated projects before and after. The process uses the shared machine lock and fixed 3,840 MB footprint ceiling; it is queued behind a confirmed live conformance run. No candidate timing or admission success is claimed yet.

Rust release assets, Bun check semantics. Third-party app timings in the Rust README remain context only and are not copied into Home's table.

8 days ago

The schema-3 reporting foundation for #838 is ready as a separate small chunk.

It renders every declared compiler, checks the complete rotating order and one successful sample per compiler per round, requires verified executable fingerprints for additions, and computes Home's ratio against the fastest median across all competitors. It retains the required tsc, tsgo, and Home entries. Schema-1/2 three-compiler archives remain readable with their original gates; additional compilers cannot enter through an unverified legacy schedule.

Validation:

  • Full benchmark harness: 110/110 tests pass.
  • A five-compiler test includes a Rust median below Home's and proves the displayed verdict changes to 2.00× slower, rather than silently comparing only against TypeScript 6/7.
  • Missing coverage, duplicate/reordered compilers, failed samples, missing executable provenance, malformed hashes, and legacy schedules with extra competitors are rejected.
  • The complete current archive 20261007T183321Z reports the same 20 rows and ratios, including startup 12.62× and large predicates 1.36×.
  • git diff --check passes. Repository-wide Pickier remains at 20,516 findings (10,181 errors, 10,335 warnings); no edits to its existing debt.

This step does not claim new competitor timings. Candidate registration, admission output adapters, fresh multi-compiler measurement, and the C++ repository selection remain open.

The first discovery probe found that the new worktree lacked 19 ignored generated projects. Those usage failures were retained, and the entire corpus has now been regenerated using the unchanged manifest and generator. Bun canary emits a success summary even with --silent; its source confirms this is normal CLI output. A correct validation output adapter must preserve its diagnostics and negative controls before the tool can be timed.

8 days ago

Pushed f0782c5b4 directly to origin/main as Chris Breuer, without coauthors.

Schema-3 reports now validate and display every additional measured compiler, retain the required TypeScript 6/7 and Home entries, and compare Home against the fastest median across the full set. Full harness validation passes 110/110 tests. The current three-compiler archive renders the same 20 rows and ratios.

This is the reporting foundation, not completion of #838. Generic candidate registration, native Bun success-output adaptation with preserved diagnostics, and a fresh admitted expanded measurement are still required. The unchanged complete corpus has been regenerated before the next candidate admission run; the earlier missing-project results remain retained separately.

8 days ago

The complete regenerated-corpus admission probe has finished. All compiler, Rust library, validation-source, manifest, and generated-project fingerprints match before and after.

Exact candidateExisting 20-workload admissionInterpretation
Rust tsc-rs v0.1.0, native binary 3a2736e0… / TypeScript 7.1.0-dev20/20 passEvery positive is silent and successful; all existing workload-specific negative-control code multisets pass. Ready for registration in the expanded timing harness.
Bun 1.4.3-canary.1+bd599f5af, native binary b0ac9bb2… / TypeScript 7.0.20/20 under the current silent-output transport contractAll 20 commands exit 0 with the normal No type errors summary. The existing harness stops at that output check, so this does not prove negative-control parity. An explicit status-output adapter is required before admission and timing.

The Bun issue is the CLI transport format, not an observed TypeScript error. Its --silent flag does not remove the checker summary. The exact upstream formatter emits that line on success: format.rs at bd599f5af. No compiler diagnostics were suppressed and no workload or admission control was changed.

The earlier missing-project probe is retained as an infrastructure failure, separately from this complete-corpus result. No new speed claim, cross-run ratio, or timing row is admitted. The next change will register pinned additional compiler commands and payloads generically, retain all CLI output, and explicitly parse the success status while continuing to enforce the same negative diagnostics. The schema-3 reporting foundation is already on main in f0782c5 with 110/110 harness tests.

8 days ago

The pinned registry, lossless output transport, and retained admission proof are ready for #838.

Exact compiler / modePositive projectsNegative-control projects
TypeScript 6.0.3 on Node20/208/8
TypeScript 7.0.2, direct native payload20/208/8
Current ReleaseFast Home20/208/8
Rust tsc-rs v0.1.0 / TypeScript 7.1.0-dev20/208/8
Bun 1.4.3-canary.1+bd599f5 / TypeScript 7.0.220/208/8

All 140 process records retain commands, exits, stdout, stderr, actual diagnostic codes, expected codes, and admission decisions. Compiler executables, full Rust/Bun payload inventories, manifest, complete generated projects, validation code, and project context have identical fingerprints before and after. The guarded admission peaked at 293 MB; the current stripped Home build peaked at 3,027 MB. Its SHA-256 is e3dc03289ef1e34b2273b9b26951f43bad8940614d032f3e40aceaf461600ab3.

The generic JSON registry cannot replace the required three compilers. It requires exact source revisions, executable and complete payload pins, and an exact version probe from the measured executable. checked-files accepts only one complete success line with a nonzero file count on one stream. Unknown output and diagnostics are not discarded; negative controls retain their original exact code multisets and normal-exit rules.

Reports and evidence packaging re-check retained admission coverage and raw diagnostics against the measured commands. A changed, missing, or failed admission cannot become a successful timing report. Known private path prefixes are normalized when publishing; all diagnostics and samples remain present. A valid project with a top-level throw also confirmed that the canonical Bun check command checks without executing source.

The current harness used TypeScript 7's npm Node launcher. New runs measure its pinned native payload directly, alongside native Home, Rust, and Bun. Existing archives keep their actual invocation and are not numerically adjusted or merged with new results.

Full harness tests pass 128/128. Repository-wide Pickier reports 22,073 findings (11,460 errors and 10,613 warnings) in the existing corpus; git diff --check passes. The next step is a fresh full 30-round rotating-order five-compiler measurement after this source chunk is pushed. No expanded speed claim is made from admission or telemetry times. The C++ repository selection remains outstanding.

8 days ago

Pushed 76d1016f4 directly to origin/main as Chris Breuer, without coauthors.

The registry, exact version/payload pins, retained raw admission proof, status-output policy, report revalidation, and deterministic evidence packaging are now available. The checked-in Darwin ARM64 profile is verified against the installed native payloads. The required TypeScript 7 command now invokes the actual native binary directly; historical wrapped measurements retain their original scope.

Validation: 128/128 full harness tests, 140/140 real process admissions across all five compilers (20 positives and 8 negatives each), unchanged before/after provenance, and git diff --check. The current Home build passed under the unchanged guard at 3,027 MB; real admission peaked at 293 MB. Pickier remains red on 22,073 existing findings.

The complete five-compiler cold run is now live: 20 unchanged workloads, 30 fresh-process rotating-order rounds after three warmups, 600 raw round files and 3,000 samples if every command succeeds. Source for the measured Home compiler is 9d27218fc; harness source is 76d1016f4. The latter changes only benchmark infrastructure. No expanded timing result has been claimed yet. The C++ repository selection is still outstanding.

8 days ago

The full admitted five-compiler result is verified: 20261007T214816Z.

  • Host: Apple M3 Pro / Mac15,6, 11 logical cores, Darwin 27.0.0 arm64.
  • Tools: Node 26.10.0, Hyperfine 1.20.0, Python 3.9.6.
  • Home source 9d27218fc, harness 76d1016f4, stripped ReleaseFast build.
  • All five compilers pass 20 positive and 8 negative-control projects each: 140/140 retained process records.
  • 30 fresh-process rotating-order rounds after three warmups on every workload: 600 raw round files, 3,000 successful samples, every sample retained.
  • All compiler/payload/input/harness provenance remains unchanged.
  • The published archive preserves every raw round byte and all admission diagnostics, with known private path prefixes normalized. All 603 payload checksums were independently verified.

Actual outcome

Home has the lowest median on 1/20 workloads and beats direct native TypeScript 7 on 11/20. It leads startup by 1.96×. Bun is fastest on 18 rows, Rust on import_graph, and Home on startup. The largest Home loss is the frozen reexport_graph: Home 19.3 ms versus Bun 8.6 ms, 2.24× slower.

This is a same-run matrix. Older hosts, competitor sets, and TypeScript 7 launcher modes are not mixed into it and their numerical differences are not regression evidence. The C++ repository is still awaiting selection, so #838 remains open.

The complete table, methodology, versions, input modes, evidence, and a brief README table are being published. Archive SHA-256: 90d1b8094cc95fe8ab98195f96a8e6d93fe4777003146f28c8124cc49aa4aefc. All 20 detailed documentation rows have been checked against the verified report. Harness tests pass 128/128, git diff --check passes, and Pickier reports the existing 22,073 findings.

The next compiler work is driven by #839: profile the unchanged re-export graph, establish a fixed same-toolchain A/B, and improve only general compiler paths while preserving every semantic control. No benchmark-specific shortcut is proposed.

8 days ago

Published ef0174c05 directly to origin/main as Chris Breuer, without coauthors, after rebasing on the latest main.

All 20 detailed table rows match the generated verified report exactly. The archive has 600 byte-identical raw round files, 3,000 retained samples, 140 successful admission records, and 603 verified payload checksums. Home's measured result is 1/20 lowest medians across all five admitted compilers, and 11/20 faster than direct native TypeScript 7. No older host, launcher, or competitor result is merged into this table.

The next source-level optimization is tracked in #839, starting with the largest relative gap, reexport_graph (2.24× slower than Bun). The full goal remains open: #548 production admission, additional platforms, C++ repository selection under #838, and all remaining five-compiler losses still require work.

8 days ago

Published two small conventional commits directly to origin/main, as Chris Breuer and without coauthors:

  • Source: b18f19203, perf(ts): reuse bound export owners across name queries (#839).
  • Evidence/docs: 3318885fb, docs(ts): publish export-owner reuse measurements (#839).

Detailed current snapshot and same-parent acceptance and brief README table are updated. All 20 detailed rows and all 10 README rows match the verified generated report exactly.

The independent unchanged re-export confirmation improves 20.178 → 17.430 ms, 1.158×, 27/30 paired wins; paired mean saving 95% t interval 1.565–3.403 ms. The diagnostic same-generator 4096-leaf confirmation improves 1.701 → 1.538 s, 10/10 pairs, interval 131.280–226.124 ms. All samples and reversed command order are retained. Full Program 228, checker 4473, CLI 74, and harness 128 tests pass on the measured candidate. Pinned unmodified 106-file Zod has identical complete 196 TS + 3 HM9002 diagnostic identity multisets, with no added or removed diagnostics; this is stability, not production admission.

The complete five-compiler checkpoint retains 600 raw rounds / 3000 samples / 140 successful admissions / 603 verified checksums. Home leads 1/20, beats direct native TS 7 on 12/20, and still trails Bun on re-exports by 1.94×. The frozen measured source is precisely ef0174c05 plus the retained source patch; intervening compiler changes present after rebase are not assigned these timings.

  • Matrix archive, SHA-256 20d645283831bd360212267cbc27be7c54590c83fda2e9a02b6af6316d1e6672.
  • Paired archive, 4902 verified checksums, SHA-256 2c59c9f4781daae166031d43af2ece2d17e9180083e4b6a15eace89b9d37bd9a; all 266 pair rounds, full scale inputs, exact patch/scripts and semantic logs retained.

Documentation-only Pickier and git diff --check pass; the frozen candidate's repository-wide Pickier run has pre-existing 22074 findings, not a clean pass. A fresh published-tip repository-wide run is in progress.

#839 remains open. Code inspection has identified the next repeated-work site: each export-facts request constructs fresh origin queries, and recursive facts requests reopen and bind owners despite the adapter's retained query. The next candidate will test reuse of immutable owners and complete origin-query work without changing the fact traversal or introducing special cases. Any further source change requires a new exact-parent A/B and full semantic gates. #416, #838 and #548 remain open for competitor gaps, platform/C++ coverage and full production admission.

8 days ago

Published source 69c3d28ee and evidence/docs eaaddf213 directly to origin/main, as Chris Breuer, without coauthors. Both use conventional, #839-linked commit subjects; the worktree is clean.

Current detailed snapshot and the brief README table are updated. All 20 detailed rows and all 10 README rows match the verified generated report exactly. Documentation-only Pickier and git diff --check pass; full repository Pickier reports its 22074 findings, not a clean pass.

Performance and semantic acceptance

  • Exact parent 3318885fb; candidate source exactly 69c3d28ee, with no intervening upstream source in the rebase.
  • Official independent confirmation: 32.665 -> 24.885 ms, 1.313x, 25/30 paired wins; paired mean saving 95% t interval +2.516 to +13.013 ms.
  • Same-generator 4096-leaf confirmation: 2.503 -> 1.950 s, 1.283x, 9/10 wins; interval +255.781 to +710.265 ms. Official 64-leaf workload is unchanged.
  • Every screen/confirmation observation is retained. All 20 families were screened; three >3%-slower medians triggered independent 30-pair rechecks. Those median slowdowns did not reproduce, but startup remains 0.7% slower in its recheck (9/30 wins, saving CI -0.323 to +0.007 ms). Overlap is not equivalence or proof of no regression.
  • Full Program 230/230, checker 4483/4483, core CLI 74/74, entrypoint 33/33, harness 128/128 pass. No named-test filters/source exclusions; native test symbols are stripped under the unchanged guard.
  • Full unmodified pinned Zod: identical complete 196 TS + 3 HM9002 diagnostic identity multisets, zero additions/removals. Observed footprint is 887 MiB parent / 991 MiB candidate, an explicit memory tradeoff. This is regression stability, not production admission.

Fresh five-compiler result

20261008T031844Z: 140 successful admissions / 600 raw rounds / 3000 retained samples / 603 verified payload checksums, with unchanged provenance. Home leads 1/20 and beats direct native TS 7 on 12/20; Bun leads 18, Rust leads imports. Re-exports remain 1.99x slower than Bun. The largest relative loss in this run is CommonJS, 2.35x slower. No earlier host/run/competitor timing is merged into this table or attributed as a patch A/B.

  • Full matrix archive, SHA-256 50ea6212fdf1de7b6884a83d298c3a654a023ad1712e877a53163e70107f90ce.
  • Paired archive: 356 raw pair rounds / 5002 verified checksums, exact patch/scripts, complete admissions, all scale inputs and semantic logs; SHA-256 6f7abd669e0c0ad3692e7f96117bf78c18f0883c05586bd4d2cf8f861d1ade3e.
  • Candidate executable SHA-256 4b4af24fdb57dcfdba69de49a460014a1800cacdda7164a4fc8614e727e9ec2d.

#839 and the overall #416 goal remain open; #838 still needs canonical C++ selection/coverage and #548 full production admission. Code inspection identifies remaining general repeated work in recursive export-facts preparation, which still opens/binds nested owners independently of the retained query. That is a next source-level investigation, not an implemented or measured speed claim.

Execution headroom fluctuated again after publication (about 481 MiB free, below the unchanged admission/continuation disk floors). The successful completed runs are not relabeled as failed, and no new heavy work will be started below the guard. #732 remains the execution-capacity tracker. No resource ceiling/floor was relaxed or unrelated file/job altered.

8 days ago

Published source 30d3824da and evidence/docs a444cf021 directly to origin/main, as Chris Breuer, without coauthors. Both are conventional, issue-linked commits. The worktree is clean.

Current detailed snapshot and brief README table are updated. All 20 detailed rows and 10 README rows match the verified report exactly. Documentation-only Pickier/diff checks pass; repository Pickier retains 22082 findings, not a clean pass.

  • Same-parent independent official confirmation: 25.546 -> 20.730 ms, 1.232x, 25/30 wins; saving 95% t interval +2.446 to +9.779 ms. Unchanged-generator scale: 2.888 -> 2.351 s, 1.228x, 10/10, interval +375.313 to +791.154 ms.
  • The adverse initial overload screen (34.391 -> 42.957 ms, 2/10, interval -9.375 to -0.516 ms) is retained. All ten >3%-slower-median families receive independent 30-pair rechecks. Overloads are flat in that recheck; no recheck has an interval wholly below zero. Overlap is not equivalence. CommonJS remains 0.7% slower in its recheck and is explicitly documented.
  • Full Program 231/231, checker 4483/4483, core CLI 74/74, entrypoint 33/33, harness 128/128 pass. Unchanged checker artifact executes from cache; no named filters or source exclusions.
  • Pinned unmodified 106-file Zod has equal complete 196 TS + 3 HM9002 identity multisets, zero additions/removals. Observed footprints 1087/936 MiB (parent/candidate) are not a universal resource claim or production admission.
  • Fresh 20261008T042639Z: 140 admissions / 600 raw rounds / 3000 retained samples / 603 verified checksums, unchanged provenance. Home leads 2/20 (startup, overload_resolution) and beats direct native TS 7 on 15/20. Bun leads 17; Rust leads imports. Counts/margins are descriptive, not significance or cross-run causal claims. Re-exports still trail Bun by 1.27x, so #839 remains open.
  • Matrix archive: SHA-256 ed253feb8650bcb68995ba7c21ad83c07c2ed6652fc4747a879c00f35c60c8fd.
  • Paired archive: 566 byte-identical pair rounds / 5209 verified checksums, all admissions/observations, full scale inputs, exact patch/scripts and semantic logs; SHA-256 37199143215b6a03705bf3cba9975de232c43c8f4e0e9693ea0b8699a80e53bb.

No guard ceiling/floor is changed and no unrelated file/job is altered. Parent eaaddf2 has exactly the same compiler inputs as 69c3d28 (only documentation/evidence differ); frozen parent reuse is recorded, not mislabeled as a fresh compile. Candidate SHA-256: 92338a9715ef01c3bf7e7238f980e9df72e856340f6684daabf98924ceea4960.

The inherited eight-level fact cutoff is unchanged and explicitly outstanding in #843; storage reuse does not certify unbounded correctness. #416's full goal, #838 C++ selection/platform coverage, and #548 production admission also remain open. The next correctness investigation is a cycle-safe complete fact traversal under #843, alongside remaining performance gaps. No overall completion claim.

8 days ago

#848 produced a new accepted Home-only exact-parent A/B, but not a replacement expanded matrix.

The checked-in Darwin ARM64 profile still pins tsc-rs revision 72b339e412f2560549033cea8db337b1bd44c312 and Bun revision bd599f5af912512b83bba0ef387d1ba3a7d5e550. During the fresh matrix attempt:

  • the tsc-rs 0.1.0 archive verified at SHA-256 96920e12262ebe390dfa4876c3cf8ed03aa594ce99222a00e0f1517e945ba205;
  • the current moving Bun-canary download was ba9474e75240afb8c486f29e5ec571da0237b5a3d08aa3701ac5f2f843df1828, not the required archive 87f2d32f3189c67781e0cf81dac6ad91c014798317b078b2e0dbc3cba8c4bd09;
  • the exact pinned competitor payload directory was therefore incomplete, so the harness correctly recorded zero matrix samples. No current-canary substitution, profile hash edit, or mixed-run result was accepted.

The failure/provenance record is retained in the #848 archive at matrix/attempt.json. The previous admitted five-compiler snapshot remains current.

Remaining #838 work is unchanged and now explicit for the next matrix:

  1. restore or deliberately replace/review the exact Bun pin as a new versioned profile;
  2. identify the canonical active C++ TypeScript port, revision, license, build and equivalent no-emit CLI from primary sources;
  3. add that C++ compiler through the same executable/payload hash, positive-output, negative-control and rotating-order schema used by tsc-rs and Bun;
  4. run Home, TypeScript 6, native TypeScript 7/tsgo, Bun check, tsc-rs and C++ together on one host with all raw rounds retained.

No screenshot timing or third-party number has been merged into Home's tables.

8 days ago

Checkpoint: #849 is closed by 74b181394. Its exact-parent grouped require-binding A/B is documented with complete raw evidence, and the integrated checker is green at 4,521/4,521.

This is not a new cross-compiler matrix result. No moving or unpinned tool was substituted and no ranking was updated. The next admitted matrix still needs the identical corpus, options, semantic admissions, machine, warmups, repetitions, and raw evidence for TypeScript 6, native TypeScript 7/tsgo, Bun check, tsc-rs/ts-rust, and the active C++ TypeScript port. The official and 1,024-family #849 observations remain inconclusive; only its independent 4,096-family same-Home-parent A/B has a wholly positive saving interval.

8 days ago

The original pinned Bun/Rust payloads survive in this task cache. I verified the exact executable hashes and full inventory hashes against profiles/darwin-arm64.json; no profile edit or current-canary substitution is needed on this host. C++ discovery is still not a canonical-upstream selection: TypeRunner at 21f9ce8 explicitly documents a narrower proof-of-concept subset and stalled development; OnlySpecs at 8fd72a2 claims TS6 compatibility and exposes a project/noEmit CLI. I am running an untimed source-pinned OnlySpecs build/admission audit without patching its source. Any build or semantic admission failure will be retained as an exclusion, not converted into a speed row. Current matrix ranking remains unchanged.

8 days ago

Pushed as Chris Breuer, without coauthors: 7b9e6f2 — docs(ts): retain C++ admission and current profile evidence (#838, #850). Normal fetch/rebase/push to origin/main succeeded.

The unmodified OnlySpecs C++ candidate at 8fd72a2 built successfully (AppleClang 21, Release/C++20, one worker), binary SHA-256 c2934fc0381d317f8d2861f8533a15ecfefbf43538808e2341280166e1d9631b. It accepts 20/20 positives but passes 0/8 negative controls: every invalid project exits 0 silently. Source inspection shows the CLI reads diagnostics without calling Program::compile/check. It is excluded with ZERO timing samples; no canonical C++ selection or new ranking is claimed. The first runner stopped at a SystemExit; the complete corrected probe and independent exact-input capture reproduce all 28 outcomes, retaining every failure and all project bytes.

The old Bun/Rust executable and complete payload inventories still match the checked-in pins in this task cache; no moving-canary substitution or profile change.

README admission table and detailed report: https://github.com/home-lang/home/blob/main/docs/docs/TS_PERFORMANCE.md#cpp-admission-and-current-source-profile-2026-10-08-utc

Raw evidence: bench/vs_tsgo/evidence/20261008-current-profile-cpp-admission-838-850.tar.gz — SHA-256 7f538b457302bd0bc307065ca0b1d7895a05e3d16a1382100bae1d5a16b2bbb6; 9,794 payload checksums verify. Full benchmark harness 128/128 and targeted Pickier/diff checks pass. #838 remains open.

8 days ago

Starting the fresh complete pinned five-compiler matrix on landed source 211d28c (compiler tree equivalent to 00a9dcf), native SHA-256 c16c8caaf52d35180a7fcd53b87cac0dd71ff7b86526ef285e09bb4e088335f4. Both original Rust and Bun cached payloads verify every executable/inventory pin, so no moving-canary substitution or profile change is needed. All 20 unchanged workloads, all 140 semantic admission records, three warmups and 30 rotating-order rounds per workload will run under the shared 3840 MB guard. C++ remains excluded/unselected: the unmodified OnlySpecs candidate fails all eight negative-control projects with silent exit0, as documented, and is not granted a timing row. The current ranking remains unchanged until the whole run and its provenance/evidence verify; no new result claimed yet.

8 days ago

Current full matrix is now20261009T004232Z, source962957813/compilerac3488350, nativefb77a18e. Published in4bbbd191d83df1e8c6f44c8b0bd96a9490f57e0c, with all145 schema4 admissions,600 raw rotating rounds,3000 samples, unchanged five pinned payloads and603 checksums. Home leads1/20 and beats direct nativeTS7 11/20; Bun18, Rustimports. The initial verifier was invoked before explicit packaging; that workflow sequencing outcome is preserved, then existing raw data packaged/verified with no timing rerun or removed samples. Fresh README/docs values verified against all raw rows. Canonical admitted C++ and other platforms remain unresolved; no failed-admission compiler gets a timing row, and this does not prove universal leadership. Report: https://github.com/home-lang/home/blob/main/docs/docs/TS_PERFORMANCE.md#landed-class-owner-matrix-2026-10-09-utc .

8 days ago

Current whole matrix is now20261009T033439Z, sourceb9e6d2591/compileraf5fb320a/native69836df6, published in5ea87f0f81ce20d18782abcad2d0412797b9d3a0. All145 schema4 admissions /600 original rounds /3000 samples retained, pinned five payloads and before/after context unchanged,603 archive checksums verified. Home1/20 and direct nativeTS7 11/20, Bun18 leaders/Rustimports. README/full docs compared against every unrounded sample median/ratio; earlier dated matrices kept. Canonical admitted C++ and other platforms still unresolved; no failed-admission compiler gets a timing row and no universal leadership claim. Report: https://github.com/home-lang/home/blob/main/docs/docs/TS_PERFORMANCE.md#landed-nearest-prior-matrix-2026-10-09-utc .

8 days ago

Landed source 30330c1 / documentation 71fa7db adds complete virtual-section source facts with unchanged full diagnostics and scalar OOM fallback. Same-parent A/B: checked-JS scales 512/8192 improve medians 4.0%/15.1%, all 30 large pairs faster; 502 rounds and 660 admissions retained. Official interval spans zero; recursive-generics adverse warning is unresolved, not equivalence or a universal speedup. Five-compiler ranking is not inferred from this A/B: published matrix remains Home 1/20 leaders, TS7 wins 11/20. Fresh landed-source matrix is next, maintaining protocol 4 strict negatives and original competitor pins. C++ and Linux/Windows coverage remain unresolved. https://github.com/home-lang/home/blob/main/docs/docs/TS_PERFORMANCE.md#virtual-section-prefix-facts-2026-10-09-utc

8 days ago

Updated fair matrix published in 484ad69: result 20261009T045701Z, all 145 protocol-4 admissions, 600 rotating rounds / 3,000 original samples and unchanged before/after fingerprints. Original TS6/nativeTS7/Rust/Bun payload pins retained; no compiler/family/sample exclusions. Home still leads 1/20, beats native TS7 11/20. Rounded generic_calls TS7/Home both27.3ms uses unrounded values: Home is slower, not counted as a win. Raw matrix/context archives have 603/17 verified checksums. No causal source regression or equivalence claim is inferred from cross-run changes. C++ and Linux/Windows coverage remain unresolved. https://github.com/home-lang/home/blob/main/docs/docs/TS_PERFORMANCE.md#landed-virtual-section-matrix-2026-10-09-utc

8 days ago

Read-only C++ upstream refresh while Home validation waits on the shared lock: latest OnlySpecs head is still8fd72a2a34f615d71e12f450d3e571683e16404b (2026-03-12), and TypeRunner head is still21f9ce87a72fbfe011be9bfcf404f6e76bd74fe6 (2025-03-06), exactly the previously audited revisions. Neither provides a newer revision to re-admit. This refresh executes no compiler, claims no new admissions/timings, changes no pins and does not prove no other C++ checker exists. Prior OnlySpecs negative-admission failure and TypeRunner subset/CLI limitations remain the relevant unchanged evidence; canonical admitted C++ coverage is still unresolved. API commands/raw responses/checksums retained in the current #854 evidence cache. Primary repositories: https://github.com/lidangzzz/TypeScript-C-Implementation-by-OnlySpecs and https://github.com/marcj/TypeRunner . No semantic checking shortcut or patched competitor is substituted.

8 days ago

Additional primary-source discovery: ASDAlexander77/TypeScriptCompiler is a maintained C++/LLVM candidate, current source95bfa6990b6ba8300cb58b78a0670b3ba7f78049. Release v0.0-pre-alpha92 (source b617c71) offers Linux/Windows x64 assets, not a local Darwin-arm64 payload. Primary CLI defaults to JIT execution; its listed emission actions include AST/IR/native-output modes, and no check-only/noEmit option was found in that audited file. AST dump returns before later compilation setup, so it is not treated as equivalent checking. No compiler installed/executed/registered, no admission or timing claimed. Source/release metadata, asset digests and scoped CLI audit retained. Canonical frontend-only admission remains to be established, not rejected based solely on discovery. Separately attempted the existing Linux benchmark workflow on main: GitHub HTTP422 says Actions is disabled for home-lang/home, and run inventory is empty. No job created or repo setting changed. Asked user to enable Actions or provide Linux/Windows runner access while local #854 experiment continues. Sources: https://github.com/ASDAlexander77/TypeScriptCompiler and https://github.com/ASDAlexander77/TypeScriptCompiler/blob/95bfa6990b6ba8300cb58b78a0670b3ba7f78049/tslang/tslang/tslang.cpp

8 days ago

Fairness/platform audit and complete rejected #854 experiment now published in68ea38d7a, README/doc table and checksum evidence included. All successful admissions/rounds, failed/refused/interrupted attempts and scoped recovery are retained. No compiler/payload pin changed and no unadmitted C++/Linux result was promoted. Additional C++/LLVM source/asset discovery is scoped to95bfa6990 and releasepre-alpha92; no runtime admission or canonical check-only equivalence established. Existing Linux workflow dispatch gotHTTP422 because repository Actions is disabled, no job created/settings changed. User was asked for Actions or runner access while local performance work continues. Current main keeps newer #846/runtime work; rejection data remains exact-parentf60a1fa and is not a new cross-platform or competitor matrix. https://github.com/home-lang/home/blob/main/docs/docs/TS_PERFORMANCE.md#shared-require-binding-presence-rejected-2026-10-09-utc

8 days ago

Fair complete presence-only rejection evidence published in624916ec4:532 original paired rounds/660 complete admissions, all5 large batches, full20-family screen, selected and carried risks retained with source/native/input/harness fingerprints verified. No pin or admission policy changed, no variable observation removed, no competitor/platform ranking inferred. Target gains do not override independent interface-composition regression; production restored. Existing Linux workflow access remains externally constrained by disabled repository Actions, and recorded C++ discovery is not admission. README/report and checksum archives are linked: https://github.com/home-lang/home/blob/main/docs/docs/TS_PERFORMANCE.md#presence-specific-require-facts-rejected-2026-10-09-utc

8 days ago

Updated fair matrix published inabe538a86: result20261009T194221Z, all145 protocol4 admissions,600 equally rotating rounds/3000 retained original samples, unchanged source/native/input/harness/project-context fingerprints and original TS/Rust/Bun pins. Home1/20 leaders and13/20 ahead of nativeTS7 are descriptive unrounded medians, not significance/equivalence or causal cross-run statements. Matrix/context/profile/publication archives verify603/17/33/7 payload checksums. No excluded C++ candidate or unsupported platform row is promoted; recorded Linux dispatch remains blocked by disabled repository Actions pending user access. README and full results: https://github.com/home-lang/home/blob/main/docs/docs/TS_PERFORMANCE.md#restored-current-main-matrix-2026-10-09-utc

8 days ago

Current-source five-compiler refresh is now live after accepted #854 JSDoc ancestor-prefix source11055ba4746b224735c8ec2cba2f5e449bfff980 and docs0d2b0097abf6dbb37630505b69e8a261c7d53c9a were pushed. A clean matching current-main stripped ReleaseFast build succeeds and reproduces native SHA25668c6ed517f9125ae1b570ff2e585c5b368745389af828258f6621fe5e6e0311c. Same unchanged Darwin-arm64 competitor manifest, canonical complete-work check-only modes, protocol4, full145 admissions,20 workloads,3 warmups and30 equally rotating fresh processes per compiler/workload. Target600 original rounds/3000 samples; no result or new rank claimed until terminal success and full raw verification. Earlier1/20 published rank predates this change; scoped Home-only large confirmation is9.1% lower but does not establish competitor leadership. C++ and other-platform admission remain unresolved, no runner/Actions settings changed.

8 days ago

Fresh complete five-compiler matrix is now published in main b6608b1. Result20261009T205353Z, native68c6ed517f9125ae1b570ff2e585c5b368745389af828258f6621fe5e6e0311c, original pins/canonical check-only modes/protocol4/145 complete admissions/600 rotating rounds/3000 samples, stable provenance. Home1/20 leads,12/20 vsnative TS7; Bun18 andRust imports. Canonical Bun checked-file status is losslessly admitted; exact negative code multisets independently verified. Supplemental verifier initially asserted silence incorrectly, then needed the separate canonical evidence-export step; both failure records retained, actual benchmark exit0 and original timings fully verified without remeasurement. Full report/readme updated: https://github.com/home-lang/home/blob/main/docs/docs/TS_PERFORMANCE.md#landed-jsdoc-ancestor-prefix-matrix-2026-10-09-utc . Raw603-checksum archive SHA2563d65ea6df449742f89053f94b2ae444fb5815ecdfc658b495f4a184fccc98915; context28 SHA2562ffe1ac71fa8281953be14098fb0a22ed3f95a5a23fff11d35c6fca965e89eb7. C++ and Linux/Windows admission remain unresolved; no Actions/runner policy change or unsupported platform claim. This issue stays open.

8 days ago

Fresh platform readiness revalidation on2026-10-10UTC: actions/permissions reportsenabled:true,allowed_actions:all,sha_pinning_required:false and TypeScript Frontend Benchmark workflow345052841 isactive. However ACTUAL authorized existing-workflow dispatch(refmain at observed2fc2a14bff1f3fd230ee00a3682ab79cac4b9e7f) againreturnsHTTP422: Actions has been disabled for this repository. Subsequent workflow_dispatch runs total0. No Linux job/admission/timing/evidence occurred; permissions/workflow listing are NOT execution proof. Exact discrepancy/command/response retained with current#861 context. No repo/org/account settings modified and no repeat-to-pass dispatch loop. Existing workflow measures3pinned compilers only,notfive/allplatform; canonical C++/broader Linux/Windows coverage stillincomplete. Continue local#861 work meanwhile; executable runner eligibility requires trusted external-state change or user action.

8 days ago

Current C++-port admission check from the source Chris shared: John / @jrysana post 2106775342467661921 publicly claims a TS6 C++ port and real-project parity, but the post exposes no repository, source revision, release asset, license, build procedure, or equivalent check-only CLI. GitHub API currently reports jrysana with zero public repositories. A later public performance claim therefore remains context, not an admissible compiler input.

The matrix requirement is unchanged: when a canonical artifact becomes available it must run the same 20 projects and negative diagnostics, on the same machine and rotating schedule, with complete hashes and raw evidence. Until then there will be no placeholder C++ row and no copied 25–750x claim. The admitted current matrix remains Home/Zig, TS6, native TS7, tsc-rs, and Bun check.

Sign in to comment on this issue.