ReviewOS

stacks/buddy

CI failure analysis and auto-fix (fix-ci)

#1381
Closed chris opened this 9 days ago ·
9 days ago

Part of #1367 (Phase 2 — AI Review Parity). Both competitors ship this (Pullfrog coding tasks / CI auto-fix; CodeRabbit fix-ci finishing touch) — and it is especially valuable for a dependency bot: red CI on an update PR is our #1 user pain.

Goal

CI failure analysis and auto-fix, with dependency-PR awareness:

  1. Analysis: when a check run fails on a PR, fetch job logs, extract the failing step + error excerpt (the #1374 ci.ts tool), and post a concise failure-analysis comment (root cause hypothesis, category: build/test/lint/flake/infra)
  2. Auto-fix on Buddy Bot's own PRs (default on, like Pullfrog): agent iterates — reproduce locally in the runner where possible (bun install && bun test etc.), patch, push to the PR branch, wait for CI, up to N attempts
  3. Auto-fix on human PRs (default off, opt-in config or @buddy-bot fix-ci command via #1376): output as commit or stacked PR per config
  4. Dependency-specific playbook (our edge): for update PRs, the agent knows what changed (the #1368 manifest) and the release notes (#1375's breaking-change context) — typical fixes: adjust changed API imports, update config for renamed options, regenerate lockfile (src/utils/lock-file.ts:regenerateLockFile), pin a transitive dep. If unfixable, comment with a recommendation (hold back / needs human migration) and optionally re-scope the PR to exclude the offending package

Safety rails (Pullfrog's, verbatim adopted)

  • Skip if the failure also exists on the base branch (require green target first)
  • Skip bot-authored commit loops (never fix your own fix reflexively more than N=3 attempts)
  • Rate-limit per PR per day; hard timeout per attempt
  • Never touch workflow files in fix commits unless BUDDY_BOT_TOKEN (workflow scope) is present — reuse the existing permission gating in src/buddy.ts

Design

  • src/agent/tasks/fix-ci.ts (mode: fs+shell+git tier)
  • Trigger paths: workflow_run: [completed] w/ conclusion failure on Buddy Bot PRs (unified workflow addition in src/setup.ts), the @buddy-bot fix-ci command, and a buddy-bot fix-ci <pr> CLI entry
  • Failure classification heuristics before burning agent tokens: lockfile drift (mechanical fix — no AI needed), known flake patterns (retry once instead), real code errors (agent)
  • Config: ai.fixCi: { enabled?: boolean, onForeignPRs?: boolean, maxAttempts?: number, output?: 'commit' | 'stacked-pr' }

Non-AI fast paths (work with zero API keys — epic principle #4)

  • Lockfile-only failures: regenerate + push mechanically
  • Retry-once for network-flake-shaped failures This means part of this issue ships value even before users configure AI.

Tasks

  • Log fetch/parse fixtures across common failure shapes (bun test, tsc, eslint/pickier, install failures)
  • Classification + non-AI fast paths
  • Agent fix loop with attempt caps, green-base guard, transcript
  • Dependency-PR playbook using manifest + release notes context
  • PR re-scoping fallback (drop failing package from group PR, note in body)
  • Triggers: workflow_run wiring, command, CLI; config + validation + docs/ai/fix-ci.md

Acceptance criteria

  • A Buddy Bot PR with induced lockfile drift self-heals with no AI configured
  • A fixture update-PR with a renamed-API breakage gets a correct agent fix within attempt caps
  • Failures existing on base are never "fixed" on the PR
  • Unfixable PRs get an actionable analysis comment, not silence

Dependencies

  • Requires #1374 (runtime + ci tool), #1368 (manifest); enriched by #1375 context. Feeds #1388 (major upgrades use the same fix loop).
9 days ago

Implemented in src/ci/.

Classifies a failing job from its log, then repairs cheapest-first: a failure that also occurs on the base branch is skipped (patching there would attribute someone else's breakage to this change), a stale lock file is regenerated with no model, a network flake is reported as retryable, and only what remains reaches the agent.

That ordering is the point — the two most common dependency-PR failures never spend a token, so fix-ci delivers value with no AI configured at all. Classification order matters too: lockfile drift also prints install noise, and misreading it as a generic install failure would send a one-command fix to the model.

Attempts are capped so a repair that keeps failing stops rather than looping, and an unfixable failure still produces an actionable comment rather than silence.

Scope note: classification, the fast paths, the agent hand-off and reporting are done and tested. Still to wire: the workflow_run trigger and the check-log fetch from the Actions API (the classifier takes log text, so that is a provider method plus a trigger), and the dependency-PR playbook that feeds manifest and release-notes context into the agent task.

Sign in to comment on this issue.