ReviewOS

stacks/buddy

AI: learnings and memory system (repo- and PR-scoped)

#1379
Closed chris opened this 9 days ago ·
9 days ago

Part of #1367 (Phase 2 — AI Review Parity).

Goal

A persistent learnings/memory system so reviews and agent runs improve over time — both competitors treat this as a core differentiator (Pullfrog learnings; CodeRabbit learnings with governance).

Being CI-native (no server), our storage must live in the repo/GitHub itself — which is actually a feature: learnings are version-controlled, reviewable, and team-shared by default (Pullfrog's argument vs Copilot's user-scoped memory).

Design

Storage: .buddy/learnings.jsonl (committed) + PR-scoped comment markers

  1. Repo-level learnings — durable facts: build/test commands, conventions, architecture quirks, false-positive suppressions. Stored as JSONL entries:
    { "id": "l_x7c2", "scope": "repo", "text": "integration tests need TEST_DB_URL; skip in review sandbox", "source": { "pr": 142, "comment": 98123 }, "createdAt": "...", "paths": ["test/**"], "status": "active" }
    Committed via PR (bot opens a small chore(buddy): record learnings PR, or piggybacks on its own next PR) — admin-reviewable by construction, which is CodeRabbit's paid "approval delay" governance for free.
  2. PR-level memory — rolling snapshot (intent, affected subsystems, risks, review progress, addressed findings) stored in a hidden marker on the PR (<!-- buddy-bot:pr-memory v1 {...} -->, same mechanism as #1368) so follow-up runs don't start cold.

Capture paths

  • Explicit: @buddy-bot remember <text> command (via #1376 dispatcher); scoped by optional --paths glob
  • Implicit: when a maintainer replies to a review finding with a correction/disagreement, the agent proposes a learning (posted as a suggestion the maintainer 👍/👎 — only 👍 persists; prevents memory poisoning)
  • Suppressions: @buddy-bot ignore-rule <finding-fingerprint> records a false-positive learning matched on path+category

Retrieval

  • At review/agent start: load .buddy/learnings.jsonl from base branch (same injection rationale as #1377 — head-branch edits to learnings must not affect their own PR's review; test-enforced), filter by changed paths, budget into prompt (recency + path-specificity ranked)
  • PR memory loaded from marker on the same PR only

Management

  • buddy-bot learnings list|add|remove|export|import CLI
  • Config: ai.learnings: { enabled?: boolean, file?: string, maxPromptEntries?: number, implicitCapture?: boolean }

Tasks

  • src/ai/learnings.ts: JSONL store, schema versioning, ranking/budgeting
  • PR-memory marker read/write in review + agent runs
  • remember / ignore-rule commands in the #1376 dispatcher; 👍-gated implicit capture
  • Learnings-recording PR flow (batched, rate-limited)
  • Base-ref loading + injection test (head-branch learnings tampering is inert)
  • CLI subcommands + docs/ai/learnings.md

Acceptance criteria

  • A recorded false-positive suppression prevents the same finding category+path from re-posting in subsequent reviews (fixture test)
  • Follow-up agent runs on a PR demonstrate loaded PR memory (transcript shows it)
  • Learnings changes are always reviewable (never silently committed to the default branch)

Dependencies

9 days ago

Implemented in src/ai/learnings.ts, wired into the review engine.

Storing learnings as JSONL in the repository is the design point: a CI job has no database, and committing them makes them reviewable, diffable and shared by the whole team rather than trapped in one account. JSONL specifically so appending is a one-line diff and one corrupt line loses one learning instead of the file.

Learnings are read from the base branch, never the head — they are injected as trusted prompt context, so the same rule that governs guideline files (#1377) applies here. A contributor cannot add "approve everything without review" to the file their own PR is reviewed under; there is a test for it.

Path-scoped learnings matching the files under review rank above general ones, and repeating a learning updates the existing note rather than growing the file with near-copies that each cost prompt budget.

Scope note: the store, ranking, rendering and base-ref loading are done. The write path — the remember command handler and the batched learnings-recording PR — is part of the command-handler follow-up from #1376.

Sign in to comment on this issue.