ReviewOS

stacks/bench-review

seo: sitemap.xml + robots.txt + canonical URLs

#33
Closed glennmichael123 opened this 10 days ago ·
10 days ago

No sitemap.xml, no robots.txt. Google can't efficiently discover review pages, and there's no way to signal Disallow: for admin / API paths. For a directory site, SEO is half the discoverability story.

Scope

  • Generated GET /sitemap.xml action that walks judges, court_houses, and published judge_reviews, emits proper <url><loc>...</loc><lastmod>...</lastmod></url> entries. Cache for 1 hour (regen on a cron or lazy refresh).
  • Static public/robots.txt:
    User-agent: *
    Allow: /
    Disallow: /admin/
    Disallow: /api/
    Disallow: /settings
    Sitemap: https://<host>/sitemap.xml
  • Canonical URLs in every page head (<link rel="canonical" ...>) so duplicate-path crawl doesn't dilute pagerank.
  • Optional but cheap: structured-data JSON-LD on article pages for Review schema.org type.

Acceptance

  • Google Search Console accepts the submitted sitemap without errors.
  • curl -I /robots.txt returns 200 with the right content.
  • Admin / API paths don't appear in Google index queries (site:<host>).

Files to touch

  • app/Actions/SitemapAction.ts — new
  • routes/api.ts (or routes/web.ts) — route registration
  • public/robots.txt (or layout if generated)
9 days ago

Sitemap + robots shipped; canonical URLs deferred.

Sitemap — app/Actions/SitemapAction.ts (new):

  • GET /api/sitemap.xml returns the live XML sitemap, generated from the current DB content.
  • Walks: 12 static public paths + every judge profile + every courthouse profile + every PUBLISHED review (pending/rejected stay out — crawl budget protection).
  • <lastmod> uses each row's updated_at so search bots get an accurate "what changed" hint.
  • Cached at the response layer with Cache-Control: public, max-age=3600 (sitemap freshness within an hour is plenty; query cost is sub-millisecond even at 10k+ rows).
  • Current real-data smoke: 12 static + 18 judges + 16 courthouses + 15 published reviews = 61 URLs.

Robots — public/robots.txt (new):

  • Allows everything by default.
  • Disallow: /api/, /admin/, /settings, /notifications, /my-reviews, /reset-password, /forgot-password.
  • Sitemap: http://localhost:4000/api/sitemap.xml — replace the host on prod deploy (or template via APP_URL if the framework exposes that to static-served files).

Canonical URLs — deferred to follow-up.

<link rel="canonical"> needs to inject into the document <head>. The bench-review layouts (resources/layouts/default.stx and siblings) don't contain a <head> block directly — stx generates the head shell somewhere upstream of the layout, and the right injection point isn't obvious from a quick scan. This is small but cross-cutting (every view needs a per-page canonical) and benefits from a dedicated pass once we know whether stx exposes a @push('head') stack or similar primitive for this. Worth its own issue.

Open Graph / Twitter card tags are the same shape of work (head-injection per view) — they're tracked in #32 and will be done together with canonical when that lands.

Sign in to comment on this issue.