ReviewOS

stacks/buddy

Pre-merge checks and post-merge actions (merge gates)

#1382
Closed chris opened this 8 days ago ·
8 days ago

Part of #1367 (Phase 2 — AI Review Parity).

Goal

CodeRabbit's merge-workflow integration (pre-merge checks, post-merge actions):

Pre-merge checks

Configurable checks reported as a commit status / check run (buddy-bot/gate) so branch protection can require it. Each check has a mode: off | warning | error (error fails the check run → blocks merge under branch protection):

  1. Title/description validation — conventional-commit title format, description completeness (template sections filled)
  2. Docstring coverage — changed public APIs have doc comments; threshold percent
  3. Linked-issue assessment (AI) — does the diff actually address the linked issue's requirements? Post gaps found
  4. Custom checks (AI) — config-defined natural-language assertions evaluated against the diff (e.g. "DB migrations must be backwards compatible", "no new dependencies without a justification in the description")
  5. Dependency gate (our twist, non-AI) — fail when the PR introduces a dependency with: known vulnerabilities (existing OSV integration src/services/security-advisories.ts), deprecation flag (existing deprecated-dependencies-checker.ts), or a license outside an allowlist (new: fetch license from packuments — small addition to src/registry/registry-client.ts)

Post-merge actions

Config-defined actions on merge of matching PRs (on: { branches, labels, paths }):

  • Built-ins: append to CHANGELOG.md section, comment on linked issues, dashboard refresh (buddy.ts:createOrUpdateDashboard)
  • Custom AI prompts (agent read-only tier + specific write target) — CodeRabbit's "post-merge actions" (changelog updates, ticket filing)
  • Notification hook events (emits through #1387 once it lands)

Design

  • src/gates/ — checks.ts (registry + evaluation), status.ts (check-run creation/update via POST /repos/{o}/{r}/check-runs; requires GitHub App token or Actions checks: write — document both), post-merge.ts
  • Triggers: pre-merge checks run on pull_request: [opened, synchronize, edited]; post-merge on pull_request: [closed] with merged == true — both via the unified workflow (src/setup.ts)
  • Non-AI checks must run without any API key; AI checks skip with neutral conclusion + explanatory summary when no key
  • Config:
gates?: {
  preMerge?: {
    titleFormat?: 'off' | 'warning' | 'error'
    description?: { mode: ..., requireSections?: string[] }
    docstringCoverage?: { mode: ..., threshold?: number }
    issueAssessment?: 'off' | 'warning' | 'error'
    dependencyGate?: { mode: ..., licenseAllowlist?: string[], blockVulnerable?: boolean, blockDeprecated?: boolean }
    custom?: { name: string, instructions: string, mode: ... }[]
  }
  postMerge?: { name: string, on?: {...}, action: 'changelog' | 'comment-issues' | 'dashboard-refresh' | { prompt: string } }[]
}

Tasks

  • Check-run plumbing in src/git/github-provider.ts (create/update, annotations for failures)
  • Non-AI checks: title/description/docstring-coverage/dependency-gate (+ license fetch in registry client)
  • AI checks: issue assessment + custom assertions via #1374 read-only mode with structured verdict output (#1373 json mode)
  • Post-merge dispatcher + built-ins + custom prompts
  • Config + validation + docs/gates.md (incl. branch-protection setup guide)
  • Tests: mode matrix (off/warning/error), neutral-on-no-key, check-run payloads, post-merge filters

Acceptance criteria

  • Branch protection can require buddy-bot/gate; an error-mode failure blocks merge and annotates why
  • Dependency gate fails a fixture PR adding a known-vulnerable package — with zero AI configured
  • Post-merge changelog action appends a correct entry on merge

Dependencies

  • Requires #1375 (AI checks reuse review context); complements #1369 (auto-merge waits on the gate). Non-AI parts independent.
8 days ago

Implemented in src/gates/checks.ts plus createCheckRun on the provider.

Title format, description completeness and a dependency policy gate, reported as a check run rather than a comment so branch protection can require it — a gate that cannot block is only advice.

Every check here runs with no API key and no model. The vulnerability, deprecation and license facts all come from data buddy-bot already collects, which makes this the gate a repository gets for free.

Each check has an off/warning/error mode and only error blocks; a warning failure reports without turning the check red, which is what makes a gate adoptable before a team has agreed to enforce it. A dependency whose license is not on the allowlist is reported rather than assumed acceptable — that is what an allowlist is for. Check-run creation failures are logged rather than thrown, since a default GITHUB_TOKEN may lack checks: write and a missing gate result should not fail the run that produced it.

Scope note: the non-AI checks, the mode machinery and the check-run plumbing are done. Not implemented: the AI checks (linked-issue assessment, custom natural-language assertions) and post-merge actions. Both are additive against the GateResult shape — an AI check is one more entry in runGates returning the same type — so they are follow-up work rather than a blocker.

Sign in to comment on this issue.