ReviewOS

stacks/analyticshq

[feature] Import GA4 history straight from the Data API, no CSV

#42
Closed glennmichael123 opened this 9 days ago ·
9 days ago

/features/import promises history arrives "in 1 click ... without a spreadsheet or a migration project". Until now the only importer was scripts/analytics/import-ga.ts, a CLI that reads a CSV a human first exported by hand from the GA4 UI. The page described a product we did not ship.

What was built

  • POST /api/sites/{siteId}/import/ga4 — owner-only, and a dashboard panel that walks the date range a calendar year at a time with progress.
  • scripts/analytics/import-ga4.ts — the same import as a CLI.
  • app/Analytics/ga4.ts — service-account auth, pagination, (other) handling.
  • app/Analytics/ga-import.ts — the normalization and row synthesis, now shared by the CSV and API importers so they cannot drift.

Service account, not OAuth

OAuth would mean registering an application with Google, a consent screen, verification for a sensitive scope, and holding refresh tokens for customers' Google accounts — a standing Google dependency for a product that sells not having one, and the open question in #25.

With a service account the customer creates the credential in their own Cloud project and grants it Viewer on their property. We register nothing with Google and hold no long-lived credential: the key is used for one token exchange and dropped. It is never stored and never logged — redactKey also scrubs Google's error bodies, which can echo parts of a malformed assertion back at us.

The cost is that creating the service account is not itself one click. The import is; the setup is three steps, documented on the feature page.

Rows are synthetic, and always were

GA4's API does not expose individual pageviews any more than its CSV exports do. Both importers synthesize rows that reproduce the daily aggregates. The synthetic id hash deliberately keeps the defaulted source ('Direct') rather than the raw one, preserving the key the CSV importer used before the synthesis moved into the shared module — changing that string would silently double the history of anyone who re-imports without --replace.

Nothing is silently dropped

GA4 buckets high-cardinality dimensions into a literal (other) row: real traffic under a fake label, which imported naively puts a page called (other) in the pages report. Those rows are counted, dropped, and reported. Page-cap truncation and undated rows are reported the same way. A partial import that claims success is worse than a failure, because the missing history reads as a traffic drop that never happened.

Synchronous, because there is no worker

Jobs here are cron-driven via app/Scheduler.ts; the queue worker in config/cloud.ts is commented out. A background import would be enqueued and never picked up — and would look like it worked. The request does bounded work instead, and the dashboard chunks the range.

Verification

  • 28 checks against a fake Google that verifies our RS256 assertion with the matching public key, so the JWT signing is proven rather than assumed — plus pagination, (other) handling, and synthesis reproducing GA's view/session/visitor totals.
  • The CSV importer was rewritten onto the shared module, so both were run over the same fixture into the same site id and diffed: byte-identical, 145 page_views / 59 sessions either way.
  • 17 checks against dashboard.stx rendered as a real authenticated owner, confirming the panel is absent for a share viewer.
  • 18 mutations, each turning a test red.

Filed retroactively for the record. Shipped in 6f4e379.

9 days ago

Shipped in 6f4e379. Filed after the fact for the record — the work was authorized directly, not from an issue.

Sign in to comment on this issue.