ReviewOS

stacks/buddy

Reporting and metrics: scheduled dependency-health and activity reports

#1389
Closed chris opened this 9 days ago ·
9 days ago

Part of #1367 (Phase 3 — Dependency Engine v2).

Goal

Reporting & metrics parity with CodeRabbit's dashboards/scheduled reports (reports) — CI-native, no SaaS backend: reports are generated in Actions and delivered as issues/comments/notifications.

Spec

1. Metrics collection (non-AI)

src/reports/metrics.ts — computed from GitHub API + run transcripts, cached as a JSON artifact per run:

  • Dependency health: total deps by ecosystem, outdated count/percent, major-behind count, vulnerable (OSV), deprecated, EOL base images (#1386), average dependency freshness (days behind latest)
  • Bot activity: PRs opened/merged/closed per period, time-to-merge, auto-merge rate, rebase count, fix-ci success rate (#1381 transcripts)
  • Review activity (once #1375 lands): reviews run, findings by severity, acceptance rate (suggestions committed vs dismissed)
  • Token/cost spend per feature (from #1373 usage accounting)

2. Report generation

  • buddy-bot report [--period 7d|30d|90d] [--format markdown|json] [--prompt "<focus>"]
  • Deterministic template baseline (works with zero AI): summary tables + deltas vs previous period
  • AI narrative mode (when configured): natural-language prompt shapes the report (CodeRabbit's prompt-defined reports) — e.g. --prompt "focus on security posture for a leadership audience" — using #1373 with the metrics JSON as context (metrics are trusted data; safe to inline)
  • Delivery: dedicated pinned "Dependency Report" issue (reuse findExistingDashboard pattern, src/buddy.ts:2568), and/or #1387 notification sinks (Slack digest)

3. Scheduling

  • reports config + a schedule line in the unified workflow (src/setup.ts), e.g. weekly Monday 9:00
  • History: keep last N report payloads as JSON in a buddy-bot/reports branch or workflow artifacts (decide: artifacts expire; branch is durable and diffable — prefer branch, document repo-pollution tradeoff)
reports?: {
  enabled?: boolean
  schedule?: string           // cron
  period?: '7d' | '30d' | '90d'
  prompt?: string             // AI narrative focus
  delivery?: ('issue' | 'slack' | 'discord' | 'webhook')[]
}

4. Dashboard v2 tie-in

Add a compact "Health" header block to the existing dashboard issue (src/dashboard/dashboard-generator.ts): outdated %, vulnerable count, deprecated count, freshness score with trend arrows vs last report.

Tasks

  • Metrics collectors + JSON schema + period-delta computation (needs the history store)
  • History store on a buddy-bot/reports branch (orphan branch, JSON files, pruned to N entries)
  • Deterministic markdown report + AI narrative mode
  • Report issue delivery + notification-sink delivery; workflow schedule emission
  • Dashboard Health header
  • Tests: metric math with fixture API responses, delta correctness, no-AI baseline, history pruning
  • docs/reports.md

Acceptance criteria

  • buddy-bot report --period 30d produces a correct markdown report with deltas, zero AI required
  • Scheduled weekly report lands on the pinned issue and (when configured) Slack
  • Dashboard shows the Health block with accurate counts

Dependencies

9 days ago

Landed in 42b151d.

buddy-bot report [--period 7d|30d|90d] [--format markdown|json] [--prompt <focus>] [--publish].

Everything is arithmetic over data the scan already produced, so it works with no AI provider and no external service. A configured provider adds three sentences above the tables and is told never to invent or restate figures — a report whose numbers came from a language model is not a report. A failing narrative leaves the numbers intact.

Two details easy to get backwards. Delta direction is per metric: fewer outdated is an improvement, more vulnerable is not, and inferring it would produce a report congratulating a repository for regressing. And 'merged vs opened' is labelled rather than presented as a score, since a PR opened on the last day has had no chance to merge.

History is JSONL in the repo, read from the base branch — the history decides what the published deltas say, and a feature branch must not decide what the trend looks like. A corrupt line loses one snapshot, not the trend. Comparison is only ever against the same period.

Sign in to comment on this issue.