fix(site): a missing page says 404, and the crawler files are checked on deploy

public/README.mdmodified+10-0

Changes to public/README.md

		⋯11 linesShow the 11 lines above this hunk@@ -12,6 +12,16 @@ Let's say you want to overwrite the default `robots.txt` file. You can do this b
12	12
13	13	If you want to overwrite the default `favicon.ico` file, you can do this by putting a file named `favicon.ico` in this folder.
14	14
	15	+## Uplink's crawler-facing files
	16	+
	17	+`robots.txt` and `sitemap.xml` are served from here rather than generated by a
	18	+route, so they are the intended home and there is nothing to keep them in sync
	19	+with. **Adding a public page means adding its URL to `sitemap.xml` by hand.**
	20	+
	21	+The deploy workflow checks all three of `/404`, `/sitemap.xml` and
	22	+`/robots.txt` against the live site after every deploy, because a CDN in front
	23	+of the app can serve its own managed `robots.txt` and quietly replace this one.
	24	+
15	25	## Security
16	26
17	27	If you want to make sure that a file is not publicly accessible, you can put it in the `storage/private` folder. This folder is not accessible by the web server.