ReviewOS

also looking at this

stacks/buddy

feat(ci): check the docs against the source they describe

#1403
Merged glennmichael123 wants to merge ci/check-docs-against-source into main
3 files +311 -1
9 days ago

Replaces #1401, which GitHub auto-closed when its stacked base branch was deleted on merge of #1400. Same branch, same commit, retargeted to main.

#1399 and #1400 fixed ~25 false documentation claims. This one stops them coming back.

Why the drift was invisible

The Docs workflow is path-filtered on docs/:

on:
  push:
    paths:
      - 'docs/**'
      - 'bunpress.config.ts'

A change under src/ that renamed a flag, retired a command or dropped a config key never re-checked the page describing it. Combined with bin/cli.ts (3,401 lines, ~30 commands) being imported by zero tests, nothing in the repo could notice. That is why so much of the drift was flag-shaped.

What this adds

bun run check:docs derives the truth from source at run time, so it cannot go stale the way a hand-maintained list would:

  • CLI surface — every buddy <command> [--flags] in the docs must resolve to a command registered in bin/cli.ts, with flags that command actually declares.
  • Config keys — every key in a config example must exist in BuddyConfig (src/types.ts) or PackageRule (src/rules/engine.ts).

It lives in CI rather than the Docs workflow precisely because CI has no path filter, and publish-commit now depends on it so drift cannot reach the registry.

Keeping it honest rather than merely loud

Two refinements, both real false positives found during development:

  • Keys under a Record<string, …> are user data, not schema — a package name in pin is not reported.
  • A fence that declares types is describing a shape rather than using one.

That second rule is also the blind spot: an invented interface in an API reference page still passes. The script header says so rather than implying coverage it does not have.

Proof it works

Injecting three regressions into docs/usage.md:

docs/usage.md
  724: unknown command: buddy frobnicate
  725: buddy scan: unknown flag --totally-fake-flag
  732: unknown config key: imaginarySection

✗ 31 claim(s) in 3 file(s) do not match the source.

Exit 1. On the current tree:

✓ 75 pages agree with 30 commands and 221 config keys

Verification

  • 1885 pass / 0 fail, lint clean, typecheck clean
  • check:docs green on this branch

🤖 Generated with Claude Code

https://claude.ai/code/session_01UWouahBJT3msK7V2VDYY6T

3 changed files on the files tab, with 0 review threads.

Sign in to comment on this pull request.