ReviewOS

stacks/bench-review

search: migrate from in-memory filter to Meilisearch

#41
Closed glennmichael123 opened this 10 days ago ·
10 days ago

Current search is in-memory substring filter (JudgeSearchAction.ts does Judge.all().filter(includes(needle))). Won't scale, no fuzzy match, hard 20-row cap. Meilisearch is running locally on 127.0.0.1:7700, the framework already has a driver, the model useSearch traits are declared. The wiring is largely there.

Detailed plan was drafted earlier in a separate analysis pass. Filing as an issue for tracking.

Scope summary

  1. Phase 0 — Fix config drift: config/search-engine.ts:11 is driver: 'opensearch', flip to 'meilisearch'.
  2. Phase 1 — Model useSearch blocks + per-model toSearchableObject() overrides for relational fields (judge ↔ court denormalization, review ↔ judge denormalization).
  3. Phase 2 — Run ./buddy search-engine:update --flush --settings for the initial backfill.
  4. Phase 3 — Replace JudgeSearchAction with a Meilisearch-backed call. Add CourtHouseSearchAction.
  5. Phase 4 — Add /api/search global endpoint that fans out across judges / court_houses / judge_reviews indexes. Filter reviews on status = published at query time.
  6. Phase 5 — Migrate SearchView.stx to use the server-side global search instead of client-side filter.
  • stacksjs/stacks#1917 — toSearchableObject returns null for useSearch: true (blocks Phase 1 if we ever use the boolean form).
  • stacksjs/stacks#1918 — toSearchableObject ignores belongsTo relations (we work around with per-model toSearchableObject overrides until this lands).

Acceptance

  • Typeahead in the header uses Meilisearch — typo tolerance works ("Robets" finds Roberts).
  • /search page is server-side, full-text searches reviews by title + content + judge name.
  • Pending/rejected reviews never appear in public search results (filter status = published).
9 days ago

Phase 0 + Phase 1 shipped — config + model traits. Phases 2-5 (backfill / action rewrite / global search endpoint / SearchView migration) are explicit follow-ups.

Phase 0 — config drift:

  • `config/search-engine.ts:11` flipped from `'opensearch'` → `'meilisearch'`. The dispatcher at `storage/framework/core/search-engine/src/index.ts` reads this directly with no env-var fallback, so this is the single source of truth.

Phase 1 — model traits:

  • Judge (`useSearch`): same-table `name` stays searchable; `practice_area` + `court_house_id` added as filterable so the directory filter pills (#38) can move to Meili facets when SearchView migrates.
  • CourtHouse: `city` + `state` + `address` added as searchable — this is where Meili shines (typo-tolerant city lookup, partial address match). `state` added as filterable.
  • JudgeReview: `status` + `type` + `judge_id` + `user_id` + `anonymized` are all filterable. `status` is load-bearing — every public read path queries `status = 'published'`; that's now expressible as a Meili filter expression.
  • User: `useSearch` commented out entirely. Indexing emails to a shared service is a PII leak surface; admin user-search stays on the DB `LIKE` query in `/admin/users`.

Cross-table searchable fields deliberately NOT added yet — Judge searching by court name, JudgeReview searching by judge name. The natural fix is denormalising those fields into the indexed document, but that depends on either upstream stacksjs/stacks#1918 landing (toSearchableObject auto-walking belongsTo relations) or us hand-rolling per-model overrides. Either way, follow-up.

To activate (user runs locally): ```bash ./buddy search-engine:update --flush --settings ``` Drops existing indexes, applies the new settings from the model traits, walks every `useSearch`-enabled model, runs `Model.all()` + `toSearchableObject()` per row, pushes to Meili. ~30 seconds at bench-review's current scale.

Verify after: ```bash curl -s 'http://127.0.0.1:7700/indexes/judges/settings/searchable-attributes' | jq curl -s 'http://127.0.0.1:7700/indexes/court_houses/settings/filterable-attributes' | jq curl -s 'http://127.0.0.1:7700/indexes/judge_reviews/documents?limit=2' | jq ```

Phase 2-5 — deferred to follow-ups

  • Phase 2 — backfill is the one-liner above; user-run, not a code change.
  • Phase 3 — replace `JudgeSearchAction` with `useSearchEngine().search('judges', q, { limit: 20 })`. Tiny change but breaks the existing court-name + city-state fallback in the typeahead. Best done together with the cross-table denormalisation so we don't ship a regression.
  • Phase 4 — `/api/search` global endpoint fanning out across judges + court_houses + judge_reviews. Filter reviews on `status = 'published'` + `anonymized = 0` (the #36 piece) inside the Meili filter expression. ~50 lines once the indexes are populated and verified.
  • Phase 5 — `SearchView.stx` migration from client-side `.includes()` to the new `/api/search` endpoint.

Why not all 5 phases here

Phases 0+1 are pure config — they don't break anything that's working. Phases 3-5 actively replace working code paths (typeahead, search page) and need a working backfill + smoke test against the live indexes to avoid shipping a worse experience than the current in-memory filter. That smoke test wants the dev server live + the backfill command run + Meili behaviour validated end-to-end — a focused session not a tail-end commit.

Will land 3-5 as a dedicated bench-review session once you've verified the indexes look right after the backfill.

Sign in to comment on this issue.