ReviewOS

also looking at this

stacks/commshq

fix(seo): ship a robots.txt so the production smoke test passes

#10
Merged glennmichael123 wants to merge fix/robots-txt into main
2 files +46 -0
public/robots.txtadded+24-0
Changes to public/robots.txt
@@ -0,0 +1,24 @@
1# https://commshq.org — crawl policy.
2#
3# The auth-walled routes below also send `robots: noindex, nofollow` from their
4# own useSeoMeta call, which is what keeps them out of an index once fetched.
5# This file stops well-behaved crawlers spending the request at all, and is the
6# only place that can speak for routes that never render HTML.
7#
8# No sitemap is published yet (sitemap.xml is a 404). Add a Sitemap: line here
9# when one exists rather than pointing crawlers at a missing file.
10
11User-agent: *
12
13Disallow: /dashboard
14Disallow: /login
15Disallow: /register
16
17# Landing pages reached from email links; nothing worth indexing.
18Disallow: /preferences-saved
19Disallow: /subscription-confirmed
20
21# The JSON API: no HTML, nothing to index.
22Disallow: /api/
23
24Allow: /