Event ingestion API: keys, batching, rate limits, taxonomy

Context

Everything downstream (reports, auto-reports, billing meters) feeds on ingested events. Mirror loghq's ingestion discipline (routes/logs.ts, app/Logs/normalize.ts, app/Errors/{ingest,limits}.ts, docs/ingest.md) but for analytics events and metrics rather than log lines.

Scope

• Event model (append-only stream): project-scoped, name (dot taxonomy, e.g. commerce.order.created), occurred_at, arbitrary bounded JSON properties, optional value/currency, optional user_key (customer-supplied pseudonymous id for uniques/retention), session_key, indexes chosen for time-range + name queries. Schema from the model only (#5).
• POST /ingest with X-ReportsHQ-Key: rhq_... (write-only key from #4): batched { events: [...] }, 512 KB body cap, batch-size cap, per-event field bounds and normalization (drop-with-count, never 500 on bad rows), response 201 { ok, stored, dropped, skipped }.
• Rate limits: per-project and per-IP over a sliding window (port loghq's Redis-or-process limiter or use ts-rate-limiter from ~/Code/Libraries - whichever is stronger; if ts-rate-limiter is missing a capability, improve it upstream + bun run release:patch).
• Read API under bearer auth: GET /api/projects/{id}/events?name=&from=&to=&before= with keyset pagination (loghq pattern, no offset pagination).
• Standard event taxonomy doc docs/events.md: the reserved commerce.*, user.*, cms.* names with required/optional properties per event. This doc is the contract for the SDKs (#7, #8) and the auto-report templates (#14).
• Wire contract doc docs/ingest.md (request/response shapes, limits, error semantics - match the completeness of loghq's).
• Feature tests: happy path, oversize body, bad key, mixed-validity batch, rate-limit 429s, cross-project isolation.

Acceptance

• A curl batch from the docs works verbatim against a dev server; tests cover every documented limit.

Depends on #4. Blocks #7, #8, #10, #14, #18. Part of #1 (M2 Accounts and data). Ground rules: see #1.

Shipped in e1879e9. CI green, 96 tests.

POST /ingest authenticates with the project's ingest key and nothing else. That key ships inside the customer's application, so it is public by design and grants exactly one capability: append events to one project. It can never read; reads use a bearer token on a different route. A missing, malformed, unknown and revoked key all return the same 401, because anything more specific tells whoever is probing which guess was closer.

Two rules run through the whole path.

A partial batch is a success. Events arrive in batches from a running application, and rejecting fifty good ones because the fiftieth had an unusable name loses real data at the worst possible moment. Bad rows are dropped, counted, and returned with their index and reason, so a client can log errors and fix its own payload without opening a conversation with us.

Nothing a caller can send produces a 500. Names are folded (Commerce/Order Created becomes commerce.order.created, which matters because two spellings from one codebase would otherwise be two series on every chart). Timestamps are clamped to 30 days back and an hour forward rather than refused, since a device with a wrong clock is still a real event the customer paid for. Property bags are trimmed property by property so the result stays valid JSON rather than a truncated string. Over-long batches are truncated and the remainder reported as skipped.

Events are append-only and never deduplicated after the fact, because a report that changes retroactively is one nobody can act on. Corrections are new events. The table carries both occurred_at and received_at because they answer different questions: reports bucket on the first, and the second is what explains a gap. There is no updated_at, which on an append-only table could only ever be a lie.

Rate limits count requests, not events. Per project and per source address, sliding window rather than fixed (a fixed window lets a caller send a full allowance either side of the boundary, so the real ceiling is double at exactly the moment a flood starts). Combined with the batch cap a request ceiling is still an event ceiling, and it keeps the decision cheap enough to make before parsing a body. Event volume against a plan is a different question with a different consequence, it should bill rather than block, and is metered separately in #18. The limiter fails open: dropping a customer's data because our own counter is unavailable is the worse of the two outcomes.

Reads are keyset paginated, not offset. An append-only stream grows underneath a reader, so OFFSET 100 silently shifts as events arrive and pages either repeat or skip rows; a cursor also costs the same on page 1 and page 400.

Verified against a live server, not just in tests. The curl in docs/ingest.md, run verbatim:

{"ok":true,"stored":1,"dropped":0,"skipped":0,"errors":[]}
HTTP 201

and a 135-request burst returned exactly 120 × 201 then 15 × 429 with Retry-After: 10, which is the documented limit.

docs/ingest.md and docs/events.md are the contracts the SDKs (#7, #8) and the auto-report templates (#14) get written against. The taxonomy documents the reserved commerce.*, user.* and cms.* names with their required and recommended properties, and calls out the two traps worth knowing: sending both order.created and order.paid without thinking double-counts revenue, and a user_key is what makes active users and retention mean anything.

Three upstream fixes were needed to build this honestly:

• ts-rate-limiter could hang any process that used it. MemoryStorage with auto-cleanup starts a setInterval and never unrefs it, so a three-line script that consumed three tokens sat there until it was killed: no output, no error, nothing pointing at a rate limiter. Fixed and released as 0.4.5, with a test that runs a subprocess and requires it to exit.
• ResponseStatus in bun-router omitted 413, so an endpoint refusing an oversized body could not say so without an as any, which turns off checking for the whole call. It was also missing 402, which #17 and #18 will need. Widened to the codes handlers actually return and released as 0.0.27.
• CSRF blocked the endpoint entirely. The framework enforces a double-submit token on every unsafe method, which is right for anything a browser session reaches and wrong for a machine-to-machine endpoint holding no cookie. .skipCsrf() is the framework's own mechanism for it and is applied to this one route, with the reasoning recorded where it is applied rather than in a commit nobody will find.

One small thing left deliberately: error: "invalid_json" is unreachable over HTTP, because the router parses the body before a handler runs and refuses a malformed one upstream with its own envelope. The status is still 400, and docs/ingest.md says to match on the status rather than the code for that one case, since it is the response an integrator is most likely to hit.