ReviewOS

home-lang/home

Add diagnostic snapshot tests

#9
Closed glennmichael123 opened this 24 days ago · 0 comments
24 days ago

Context

docs/ERROR_MESSAGES.md describes a Rust-style rich diagnostic system: colored output, source context, primary/secondary labels, "did you mean?" suggestions, error recovery. This is a real differentiator vs. Zig (whose error messages are functional but plain).

Rich diagnostic output is also the first thing to silently regress when codegen / parser / type checker code gets refactored. A formatting tweak in the diagnostic renderer can degrade error UX in ways no unit test catches.

Proposal

  • Build a small snapshot-test harness that:
    • Reads representative bad .home programs from tests/diagnostics/
    • Runs home check on each
    • Compares stderr against a checked-in .expected file
    • Fails CI on any diff (with a --update flag for intentional updates)
  • Seed the corpus with ~20 representative diagnostics:
    • type mismatch
    • undeclared variable
    • missing return
    • "did you mean" typo (function name / variable name / field name)
    • exhaustive match missing arm
    • borrow / move violations
    • unresolved import
    • parse error with recovery showing multiple errors at once
    • generic monomorphization failure
  • Render in both colored and plain modes; snapshot the plain mode (terminal-portable)

Why this matters

  • Locks in the diagnostic UX as a feature, not an artifact
  • Catches accidental quality regressions during refactors
  • Gives contributors a concrete way to improve error messages: write a worse-than-it-could-be diagnostic into the corpus, then improve the snapshot

References

  • Rust uses this approach extensively (tests/ui/)
  • Elm's friendly compiler errors are tested similarly

Sign in to comment on this issue.