ReviewOS

home-lang/home

bench(ts): add shared async and await type-checking coverage

#473
Open chris opened this 9 days ago ·
9 days ago

Follow-up to #416 after the 18-workload suite reaches lower Home means, including a separately confirmed large-predicate lead. Async/await and Promise result propagation are not represented by a dedicated current workload.

Add a deterministic, representative async workload using exactly the same source, shared declaration inputs, strict/noEmit project flags, and pinned TS 6.x/native TS 7.x commands for every compiler. Cover awaited generic results, Promise-like values, typed async returns, and control-flow consumption; use only declarations faithful to the chosen TypeScript semantics, not weakened stubs that erase types. Inspect actual compiler support before accepting the workload. Add negative controls that prove inferred awaited values/return types are checked, and fix any discovered Home semantic gaps rather than timing rejected or unchecked inputs.

Preserve all existing 18 workloads and frozen predicate hashes. Publish the complete expanded suite, including losses and variance, in README and detailed docs. Continue round-robin process-cold/cache-warm measurement, exact installed-version preflight, and all-sample retention. This broadens feature coverage; it does not substitute for future pinned real-world projects or cross-platform results. Use small conventional commits linked to this issue and #416.

Admission status after 9e45e10

  • Independently audit the same strict ES2022/noLib candidate and unchanged shared Promise declarations against verified TS 6.0.3 and native TS 7.0.2.
  • Fix general primitive-return checking and block-callback return erasure (#474; code 9e45e10, documentation 75e02f3).
  • Preserve inferred Promise chain and Promise.all tuple results (#475).
  • Infer the generic async projection without false TS2322 (#476).
  • Correct awaited-target diagnostic wording (#478).
  • Admit the full positive/negative async workload only after these controls pass, then publish the complete expanded timed suite.

The current accepted suite remains 18 workloads, not 19. All 4,226 checker tests, 99 program tests, 173 driver tests, 16 harness/report tests, existing positive projects and negative controls pass. Full result 20260827T202656Z retains 18/18 lower Home means after stronger return validation; the separately predeclared large-predicate confirmation 20260827T203044Z also has a 1.08× mean lead. All 1,710 timed samples are retained. These are local synthetic measurements, not universal leadership or full semantic parity. Results and correctness audit.

9 days ago

Correctness audit against verified TS 6.0.3 and native TS 7.0.2 found four independently isolated gaps using the same strict ES2022/noLib project and unchanged Promise/PromiseLike/Awaited/PromiseConstructor signatures extracted from TS 6.0.3: (1) a generic async map accepting a U | PromiseLike callback return is rejected by Home (TS2322), but accepted by both baselines; (2) an awaited .then(value => value.label) result incorrectly accepts assignment to number in Home; (3) an awaited Promise.all tuple element incorrectly accepts assignment to the wrong primitive type; (4) an async function annotated Promise<string> incorrectly accepts returning a number. Baselines reject controls 2–4 with TS2322. No async performance measurements accepted or published. Starting with underlying primitive-return enforcement in #474; remaining propagation gaps stay required before admitting the workload.

9 days ago

Async admission remains blocked on semantic parity, not timing. The declaration audit isolated #479 (generic receiver/callback instantiation; six focused regression groups now pass, full validation in progress) and #480 (global interface types erased by the program name fallback). #480 is reproduced with ordinary/custom-named interfaces, so the fix must not special-case Promise. Keep #475/#476 and #478 open, and keep async excluded from the accepted benchmark count until the shared positive/negative cases pass.

9 days ago

#479 is now fixed and closed by 0612a90, with final validation documented in 262db42. All 72 shared receiver/callback CLI checks pass, alongside 4,232 checker tests and validation of the existing 18-workload corpus. Fresh post-fix probes confirm that #475 (Promise.then/all negative results), #476 (valid generic async projection), #480 (global declaration erasure), and #478 (async error text) remain open. No async workload is admitted or timed yet. https://github.com/home-lang/home/blob/main/docs/docs/TS_PERFORMANCE.md#generic-receiver-callback-validation-untimed

9 days ago

Latest main progress (small conventional commits, no coauthors):

  • 9817696: actual/forward global declarations preced the name-only fallback.
  • 8a737cd: config-relative explicit roots, fixes #483.
  • 6cba2f2: separate value narrowing from type lookup.
  • 16f84e7: imported-graph rejection gates and withdrawal of unsupported historical speed claims (#487).
  • 50abd82: reproducible global admission audit; 2e146e7: final results/documentation.

The fairness correction is important: Home silently accepts invalid imported-property checks in both graph workloads. The full suite now stops before any timing/results creation, and the README/report/docs mark those graph comparisons ineligible while retaining the observations. Other rows remain provisional pending broader negative controls. There is no all-benchmarks victory claim.

Final validation: 4,232 checker, 101 program, 173 driver, 69 CLI, and 25 harness/report tests pass. All 72 same-file declaration CLI checks pass; the default 132-check global audit retains 14 Home cross-file failures (#480). All 72 prior callback and 18 config-path controls pass. Eighteen positive workloads validate, but only sixteen Home workload gates pass; both graph gates correctly fail. Frozen predicate hashes are unchanged.

Async remains untimed: Promise.then/Promise.all wrong-result controls are still accepted (#475), the valid generic async projection still fails (#476), and diagnostic text remains tracked in #478 under #473. Next priority is real cross-file type ownership (#480/#487), broader admission checks, then fair retiming on a suitable machine. No new timings were collected under concurrent workstation load.

Details: https://github.com/home-lang/home/blob/2e146e71e/docs/docs/TS_PERFORMANCE.md#global-declaration-and-graph-admission-audit-untimed .

9 days ago

Pushed three small conventional commits to main under the configured user identity, without coauthors:

  • 134a910: assertion signatures return void, fixing #489.
  • 091f151: core checked metadata is retained with Compilation; the retained relation engine no longer points into a destroyed Checker.
  • 97ab7d4: results, README link, and explicit remaining ownership work.

The ownership work is a prerequisite, not completed cross-file linkage. #488 remains open after auditing interpretation tables still owned by Checker (including NoInfer/this markers, readonly/pattern indexes, and tuple/array origins). Equal numeric TypeIds from separate compilations are not interchangeable. #480/#487 still require actual owner-aware type transfer and resolution.

Validation: 4,236 checker, 175 driver, 101 program, 69 CLI, and 25 harness/report tests pass. All 36 shared assertion controls and 72 earlier callback controls match TS 6.0.3 and native TS 7.0.2. The global audit remains at 14 Home failures out of 132 checks. Sixteen workload gates pass, both graph gates fail, and no benchmark is readmitted. No new timing run or speed claim; published observations and frozen sources are unchanged.

Details: https://github.com/home-lang/home/blob/97ab7d4e4/docs/docs/TS_PERFORMANCE.md#checked-type-ownership-and-assertion-returns-untimed .

Sign in to comment on this issue.