also looking at this
fix(seo): ship a robots.txt so the production smoke test passes
#10
2 files
+46
-0
| @@ -0,0 +1,24 @@ | ||
| 1 | # https://commshq.org — crawl policy. | |
| 2 | # | |
| 3 | # The auth-walled routes below also send `robots: noindex, nofollow` from their | |
| 4 | # own useSeoMeta call, which is what keeps them out of an index once fetched. | |
| 5 | # This file stops well-behaved crawlers spending the request at all, and is the | |
| 6 | # only place that can speak for routes that never render HTML. | |
| 7 | # | |
| 8 | # No sitemap is published yet (sitemap.xml is a 404). Add a Sitemap: line here | |
| 9 | # when one exists rather than pointing crawlers at a missing file. | |
| 10 | ||
| 11 | User-agent: * | |
| 12 | ||
| 13 | Disallow: /dashboard | |
| 14 | Disallow: /login | |
| 15 | Disallow: /register | |
| 16 | ||
| 17 | # Landing pages reached from email links; nothing worth indexing. | |
| 18 | Disallow: /preferences-saved | |
| 19 | Disallow: /subscription-confirmed | |
| 20 | ||
| 21 | # The JSON API: no HTML, nothing to index. | |
| 22 | Disallow: /api/ | |
| 23 | ||
| 24 | Allow: / | |