Context
docs/ERROR_MESSAGES.md describes a Rust-style rich diagnostic system: colored output, source context, primary/secondary labels, "did you mean?" suggestions, error recovery. This is a real differentiator vs. Zig (whose error messages are functional but plain).
Rich diagnostic output is also the first thing to silently regress when codegen / parser / type checker code gets refactored. A formatting tweak in the diagnostic renderer can degrade error UX in ways no unit test catches.
Proposal
- Build a small snapshot-test harness that:
- Reads representative bad
.homeprograms fromtests/diagnostics/ - Runs
home checkon each - Compares stderr against a checked-in
.expectedfile - Fails CI on any diff (with a
--updateflag for intentional updates)
- Reads representative bad
- Seed the corpus with ~20 representative diagnostics:
- type mismatch
- undeclared variable
- missing return
- "did you mean" typo (function name / variable name / field name)
- exhaustive match missing arm
- borrow / move violations
- unresolved import
- parse error with recovery showing multiple errors at once
- generic monomorphization failure
- Render in both colored and plain modes; snapshot the plain mode (terminal-portable)
Why this matters
- Locks in the diagnostic UX as a feature, not an artifact
- Catches accidental quality regressions during refactors
- Gives contributors a concrete way to improve error messages: write a worse-than-it-could-be diagnostic into the corpus, then improve the snapshot
References
- Rust uses this approach extensively (
tests/ui/) - Elm's friendly compiler errors are tested similarly