ReviewOS

stacks/wildloop

Decide whether trail ingestion still needs an HTTP trigger

#976
Closed glennmichael123 opened this 9 days ago ·
9 days ago

The original problem no longer describes the system

This was filed when ingestion was one CLI command and app/Actions/Scraper/ was an empty directory waiting for an HTTP action. Neither is true now:

  • app/Actions/Scraper/ does not exist.
  • Ingestion is trails:ingest (app/Commands.ts:35, with an ingest:trails alias) backed by app/TrailIngestWorker.ts.
  • It runs continuously in production under a systemd unit of its own — the ingest site in config/cloud.ts — because building the US catalog is a multi-day job (~1,400 Overpass tiles at two requests a minute, plus 466 Forest Service and Park Service shards) that has to survive deploys and re-sync afterwards.
  • That unit already answers / on loopback with live shard counts and per-source trail totals, which is how you check on it.

So the gap this issue names — "no HTTP/admin trigger for scheduled or remote scraping" — is partly filled by something better than a trigger: a worker that does not need triggering, and a status endpoint for watching it.

What is actually still missing

Only a write surface: nothing outside the box can ask for a region to be re-ingested. Whether that is worth building is a real question rather than an assumption, which is why this issue is now a decision rather than a task.

  • If yes: a guarded POST taking a region and a limit, admin-only, that enqueues shards for the existing worker rather than scraping inline — an HTTP request must not hold a multi-day job open.
  • If no: close this, and note in the worker that re-ingestion is deliberately operator-driven.

Acceptance, if it is built

  • Admin-guarded endpoint that enqueues work for TrailIngestWorker and returns immediately.
  • Re-entrant: asking twice for the same region does not double the shards.
  • Rate-limited, and it cannot be used to aim our Overpass budget at somebody else's region on a whim.
8 days ago

Built, and live on wildloop.org. POST /api/maintenance/reingest-region with { "region": "CO", "limit": 20 } → 202 with { region, source, tiles, queued, alreadyQueued, recentlyDone, deferred }.

Against the acceptance list:

  • Admin-guarded, enqueues, returns immediately. role:admin on the route, and the action re-checks the role itself (as RouteEffortReviewAction does): signed out → 401, an athlete → 403. The request only writes the region's rows in trail_ingest_shards back to pending (inserting any the worker never seeded); TrailIngestWorker claims them like any other pending shard, within one idle sleep (≤ 5 min) when it is caught up. Nothing is fetched in the request — a unit test fails if fetch is called.
  • Re-entrant. A tile is one row keyed on shard_key; tiles already pending or running come back as alreadyQueued and cost nothing. Colorado twice → queued: 28, then queued: 0, alreadyQueued: 28, in the unit suite and over HTTP on the QA stack.
  • Rate-limited, and cannot aim Overpass anywhere on a whim. The region must be a code from app/Ingest/regions.ts (US states + DC, the German and Austrian Länder, the Swiss cantons), else 422. Its tiles are drawn only from shards the worker already sweeps, counted when a real share of the tile is in the region (Colorado is its own 28 tiles, not 48 with neighbours' slivers). At most 50 tiles per request (the rest reported as deferred, and asking again continues), tiles finished in the last 24h are skipped (recentlyDone), and the route has its own throttle:10,1h.

Scope notes:

  • OSM only. Park Service and Forest Service shards are paged by park unit and forest, not by area, so there is no region to ask for them by; buddy trails:requeue already redoes them wholesale.
  • No priority. Requeued tiles join the pending queue in id order, so during a long initial build they wait behind it.

app/Ingest/reingest.ts, app/Actions/Maintenance/ReingestRegionAction.ts; tests in tests/unit/trail-reingest.test.ts and tests/browser/trail-reingest.test.ts (the QA stack now seeds an admin so the route is tested from both sides).

Commits: 33c643f, 52d05e2.

Sign in to comment on this issue.