ReviewOS

home-lang/home

test(ts): remove fabricated conformance passes and rebuild trustworthy parity evidence

#840
Open chris opened this 8 days ago ·
8 days ago

The TypeScript conformance harness at e2f49901ca3524bff48156b037c3c50db828ee5e can report exact passing fixtures without matching compiler diagnostics. This invalidates the published 3,220/3,220 exact prefix, earlier prefixes, and coarse-corpus aggregates until fresh validation. Related: #66 and #832.

Confirmed paths in packages/ts_conformance/src/ts_conformance.zig:

  • 2,499 fixture-name entries select the expected diagnostics and expected count as the actual result.
  • 13 fixture names return precompiled exact passes without compiling.
  • Missing no-position library errors are copied from the expected baseline into actual output.
  • Named/source-pattern mismatch rescues turn real exact and coarse failures into passes.
  • Some actual codes/messages and diagnostic filters depend on expected baseline content.
  • Option-validation errors are discarded from both expected and actual streams.
  • Virtual-file routing depends on expected errors and selected fixture names.

The active fix removes these paths and replaces the exception-ledger tests with real mismatch controls. Compiler work in the same batch fixes type-only import namespace meaning and erased computed property keys. Original upstream cases, baselines, deadlines, skips, TODOs and the 3,840 MB supervisor ceiling remain unchanged.

Acceptance:

  • Land removal of known result fabrication and baseline-dependent diagnostic transformations, with negative controls on both compiler routes.
  • Preserve and publish fresh actual outcomes; retract all affected green corpus claims.
  • Finish the input/options/library-mount audit (#841): strictness inferred from diagnostic contents, synthetic declarations, file inclusion and variant selection must match the pinned upstream harness.
  • Restore real corpus failures by implementing compiler/program features, without filters, substitutions or baseline changes.
  • Run the complete applicable Bun acceptance suite in Home; TypeScript controls earn no Bun original-case credit.

Published source checkpoint: d739fdfea, compiler batch 8e12ac031. Checker 4,482/4,482; program 228/228; negative honesty controls 4/4; compiler build succeeds. Fresh normal conformance is 1,400/1,423 with 23 retained failures, and the baseline-aware survey is 582/586 with four mismatches. These are intermediate-harness observations: input/options/library fidelity remains unverified. Raw logs and source/binary pins are committed in docs/docs/bun-port-evidence/2026-10-07-ts-harness-honesty. Full Bun parity remains open in #66.

8 days ago

Published 0234bc59818bd3ab20fd0cee6fadb2fb802bd1ad directly to origin/main as Chris Breuer without coauthors, following compiler 8e12ac031 and harness correction d739fdfea.

The harness no longer replaces actual diagnostics for 2,499 named fixtures, bypasses compilation for 13 names, copies expected library errors, rescues mismatches, or drops option-validation errors. Both compilation routes retain actual diagnostics; coarse and exact modes use the same route. Prior exact-prefix and full coarse-corpus 100% claims are withdrawn.

The compiler preserves namespace meaning through erased exports, accepts non-emitting computed keys, and applies the real any/unknown/undefined and definite-assignment rules to computed fields. Required fields still fail. The first original target run exposed an extra TS2564; that failed log is retained, and the compiler fix now gives 2/2 exact matches on the unchanged typeOnlyMerge2 and computedPropertyName cases.

Observed validation:

  • complete final checker: 4,483/4,483; program tests: 228/228;
  • harness honesty controls: 4/4; semantic field controls: 4/4 including module root;
  • real survey controls: 10/10, including absent corpora, empty selections and mismatches;
  • final stripped compiler builds; pinned 106-file Zod output remains normalized-identical at 196 TS errors and three HM9002 warnings, unchanged ceiling three;
  • fresh leading exact survey: 172/200, 28 actual failures, zero skips, exits 1;
  • normal conformance at d739fdfea: 1,400/1,423, 23 retained failures; baseline survey 582/586 with four mismatches. This older aggregate is not claimed as a rerun of the final field-initialization change.

Source/binary/fixture/baseline hashes and compressed successful and failing logs are committed here. All commands use the normal machine lock and 3,840 MB ceiling. Original upstream inputs, deadlines, skips, TODOs and ratchet ceiling remain unchanged. Twenty inactive local Zig object directories (about 1,445 MiB) were deleted after preserving evidence.

#841 tracks remaining baseline-derived strictness, real declaration/library mounting, file inclusion and upstream variant/configuration fidelity. These intermediate corpus observations do not establish faithful full-corpus parity. #840 and #832 remain open; full Bun acceptance remains incomplete in #66. TypeScript controls earn no original Bun case credit.

Sign in to comment on this issue.