Plan limits, usage metering, and retention

Context

The plan matrix (#17) enforced: one limits module that every gate consults, plus usage metering users can see. Limits fail soft and sell the upgrade; they never 500 and never silently drop data without accounting for it.

Scope

• app/Billing/limits.ts: the tier matrix as typed config (single source of truth, imported by #17's pricing page so marketing numbers can never drift from enforcement); can(user|project, capability) + remaining(project, meter) helpers.
• Metering: monthly event counts per project (increment on ingest #6, stored via a model, month-keyed); report/project/member counts derived live; meter reads cached.
• Enforcement points:
  • Ingest (#6): over-quota projects get 429 with a machine-readable reason + grace behavior (accept N% overage once per month, then reject) - document the exact policy in docs/limits.md.
  • Report/project/member creation (#4, #9): friendly in-UI blocks with upgrade prompts.
  • Schedules + XLSX (#15), unbranded shares + embeds (#16): tier checks via can().
• Retention: a scheduled pruning job deleting events past the tier's retention window (and rollups per #10's rules), batched, with a dry-run mode and logged counts; downgrades prune on next run - warned in the UI at downgrade time.
• Usage UI: a usage panel on the billing page (#17) and a compact meter in the project view - events this month vs quota, retention window, with honest near-limit states (80%, 100%).
• Emails: near-quota (80%) and over-quota notifications via app/Mail/ - once per threshold per month, not a nag storm.
• Feature tests: every enforcement point at limit-1 / limit / limit+1, month rollover, downgrade retention prune, grace policy.

Acceptance

• A Free project at its event quota gets documented 429s, sees an accurate meter, receives one warning email, and upgrading to Hobby in Stripe test mode immediately unblocks ingest.

Depends on #6, #17. Gates #15, #16. Part of #1 (M6 Monetization). Ground rules: see #1.

Most of this is built and verified, across 1c3039d, 0d27e55, 56f7d2d and aa9cc0b. CI green, 396 app tests. Leaving open for the two items that are genuinely blocked or not started, listed at the end.

app/Billing/limits.ts is the only place a limit is defined. Every gate reads it and the pricing page will too. A marketing table maintained separately from enforcement is a promise somebody eventually breaks by editing one of them, and the person who finds out is a customer who paid for a number that turned out not to be true. Two tests guard the shape of the matrix itself: tiers must increase in price and generosity together, and a higher tier can never lose a capability a lower one had, because somebody upgrading must not find something they could do before is now gone.

Limits fail soft. Nothing throws, reading is never blocked, and the ingest path carries a project through a documented 10% grace band. A project that crosses its limit mid-month is usually having a good week, and cutting its data off at exactly 100% means the report that would have shown them that week is the one with a hole in it.

Nothing is dropped without being counted. The quota check sits after the body is parsed rather than before it. Refusing early is cheaper and wrong: it means refusing an unknown number of events, and 'we dropped some of your data' is a thing a customer is owed a number for. Refusals record the count, return 429 with error: quota_exceeded, the numbers, and a Retry-After measured in the project's own month. That is deliberately distinguishable from the rate limiter's 429, which means 'too fast' rather than 'too much this month' - conflating them would tell somebody to slow down when what they need is a larger plan.

Verified over HTTP through all three states: a normal write meters, the quota returns warning: over_quota while still accepting, past grace returns 429 with rejected recorded, and flipping the plan column unblocks the very next write with no other change. That last one is the acceptance criterion.

Metering. Events are counted, not measured: the ingest path cannot afford a COUNT(*), and the count must outlive retention, since a project that sent four million events in March still used four million events in March when the invoice arrives. The counter is an upsert on a unique (project_id, month) index, because a read-modify-write loses one of every pair of concurrent ingests, which is routine. Everything else is measured live from indexed rows rather than kept in four more counters that can drift. Months are the project's own, in its own timezone, and the owner counts toward the member allowance - a plan saying '1 member' that in fact allowed two people would be a limit nobody could reason about.

Refusals are actionable. 'You have reached the Free limit' is a dead end; 'Free covers 5 reports. Hobby covers 25.' is a next step. Gates run before the write, so nothing half-exists, and answer 402 rather than 403, since a client treating every 403 as 'sign in again' would send somebody round a loop that cannot help them. Template provisioning is exempt from the report limit: the engine delivering the reports it promised is not a person choosing to exceed their plan.

Retention prunes raw events in bounded batches with a dry-run mode, per project, one failure not stopping the rest. Rollups outlive the events they were built from, so an old range still totals correctly and only drilling into individual events goes away. A downgrade prunes on the next run, which is what the downgrade warning promises.

Usage UI: the event meter is on the project page with three states, because 'nearly out' and 'out' need different words.

docs/limits.md states the whole policy including the exact grace behaviour.

Not done.

• The near-quota and over-quota emails. Blocked on mail, which is not configured until #21. The plumbing is in place - notified_at_percent is on the counter and markNotified() will not lower it, so the nag storm is already prevented - but nothing sends yet.
• The usage panel on the billing page, which belongs with #17 since the page does not exist.
• Turning the gates on failed 25 existing tests, which was the gates working: fixtures building a dozen reports were doing it on a free plan. Those fixtures are now on Pro so each test measures what it is named after.

Done, except one item that belongs to #17. CI green, 427 tests.

The quota emails, which were the last thing blocked on mail, now work. Two thresholds, 80% and the quota itself, each sent at most once per project per month. The counter enforces that rather than the job: notified_at_percent only ever moves up, so an account hovering around 80% gets one email rather than one per run. A nag storm is how somebody learns to filter us into a folder they never open, which is exactly where the message about their data stopping should not be.

Sent hourly on a schedule rather than from the ingest path, because ingest is the one request that must stay cheap and an email is a network call: a customer write should never wait on our ability to warn them about it. A project jumping from 40% to 105% in a single batch gets the "you have used it" email rather than "you are approaching it" - sending both, or the wrong one, is worse than sending nothing. The threshold is marked after the send, so a failure retries rather than silently losing the one notice that matters.

Verified against the live mail server: the notice arrived in a real mailbox with the subject, "41,000 of 50,000" and the upgrade line naming Hobby, and a second run sent nothing.

Everything else from this issue landed earlier: the tier matrix as the single source of truth, the counter-based metering, the 10% grace band, quota enforcement on ingest returning 429 with the numbers and a Retry-After in the project own month, the creation gates answering 402 with the tier that would lift them, retention pruning that leaves rollups alone, the meter on the project page, and docs/limits.md.

Not done, and it belongs to #17: the usage panel on the billing page, since there is no billing page yet. The numbers it needs are already exposed by usageFor().

One thing worth carrying into #17: the plan lives as a column on users, and every gate reads it there rather than asking Stripe. That is why "upgrading in test mode immediately unblocks ingest" is already true and tested - a webhook writing that column is the whole of an upgrade.