ReviewOS

stacks/wildloop

Backfill elevation gain for the whole trail catalog (0 of 400 sampled have it)

#1003
Closed glennmichael123 opened this 9 days ago ·
9 days ago

Part of #1002.

Problem

Elevation gain is missing on essentially the entire catalog. A 400-trail sample across production (offsets 0, 60k, 200k, 400k of 596,556) found 0 with elevation gain; the first 200 in the default ordering had 8. So a trail page reads "Not recorded" where the number a hiker checks second, after distance, belongs — and every derived thing is wrong with it (see the difficulty issue below).

Why this is tractable

The machinery is already here:

  • climbAlong() in app/Support/routing.ts turns a polyline into gain and loss in feet, via Valhalla's elevation service, resampling the path first. GET /api/geo/climb already exposes it for drawn routes.
  • Every trail already stores geometry, which is the input that needs.
  • RepairTrailDistances and RequeueTrailShards in app/Commands/ are the precedent for a sharded repair pass over the catalog.

So this is a batch job over existing data with an existing function, not new capability.

Acceptance

  • A repair command that walks the catalog in shards and fills elevation from geometry, resumable and safe to re-run.
  • Trails whose geometry is missing or too short to sample are left null rather than written as 0 — "Not recorded" is honest, "0 ft" is not.
  • Rate-limited against our Valhalla instance, with the public fallback path unchanged.
  • elevation_high filled in the same pass where the sampled heights give it.
  • Coverage reported at the end, so we can see what it actually achieved rather than assume.
  • A test over a known polyline pinning gain against a hand-checked value.
9 days ago

Shipped in 7a97b7e as buddy trails:repair-elevation.

Verified by running, against real production geometry in a throwaway database — four trails measured through Valhalla:

trailmilesgainft/mi
Gwt Francis Peak To Morgan Co.13.91,637 ft118
Gwt Francis Peak To Morgan Co.1.4259 ft192
Canjilon Mountain6.7856 ft127
Canjilon Mountain0.566 ft147

Plausible for the Wasatch and northern New Mexico. The seeded row with no geometry was left alone rather than written as 0, and an immediate re-run reported "Nothing to repair", so resumability holds.

Two acceptance items did not land as written:

  • elevation_high is not filled. Nothing in the app reads that column — no view, no action, no API field. Filling one nothing displays means guessing what it should hold, so it is deferred until something shows it.
  • No test pinning gain against a hand-checked value. The arithmetic belongs to ts-maps' climb(), and measuring needs a routing server, so a unit test would either mock the answer or assert someone else's code. What is ours — which line to measure, when not to ask, what not to store — is unit-tested in tests/unit/trail-elevation.test.ts (15 cases), and the measurement itself was checked by the run above.

Still to do: run it against production. It is rate-limited by --concurrency (4 by default) and supports --country and --limit, so it can go out a slice at a time.

9 days ago

Follow-up before the production run: the backfill as shipped would not have finished.

Sized against real production geometry, read-only through the catalog API — 150 US trails with geometry and no gain recorded:

outcomeshare
gain written49%
measured at exactly zero33% (all under 0.5 mi)
line too coarse to measure18%
failed0

The 33% is the problem. elevationOutcome never writes a zero — correctly, since elevation reads 0 as "not recorded" — but selection was keyed on elevation <= 0, so those rows were read, sent to Valhalla, discarded, and left selectable for the next run to do again. At the measured 1.2 trails/s (concurrency 4, against the public OSM server) that is roughly 200,000 requests per run for answers already known, and the catalog never gets closer to done.

Fixed in 5cecf5a and 012a5f1:

  • new elevation_checked_at column, backfilled from updated_at where a gain already exists
  • selection keys on it, so every answered row drops out
  • every row that reaches an answer is stamped — a gain, a flat trail, a line too coarse, a reading too large to be true. Only a routing server that could not be reached is left open, since that is the one outcome a re-run can change
  • what to store moved to elevationRecord() in app/Support/trailElevation.ts, tested without a routing server

Verified on 25 real production trails in a throwaway database: first run answered all 25 (17 with a gain, 6 flat, 2 too coarse), second run reported nothing to repair, and the geometry-less row was left alone rather than written as measured.

Also worth recording for whoever runs it: production has its own routing server (VALHALLA_URL: http://91.99.124.84:8002 in config/cloud.ts), so a run on the box does not touch the public OSM instance. The 1.2 trails/s above is the public server over the internet and should not be read as the rate on the box.

Still outstanding: the run itself. Nothing has yet written a single elevation figure to production — a fresh 400-trail sample today is still 0% gain, 0% rating, 0% reviews, 0% real photos.

Sign in to comment on this issue.