stacks/uplink
fix(site): a missing page says 404, and the crawler files are checked on deploy
#60
Merged
glennmichael123
wants to merge
fix/crawler-surface
into
main
4 files
+40
-5
.github/workflows/deploy.ymlmodified+14-1Changes to .github/workflows/deploy.yml | @@ -241,4 +241,17 @@ jobs: |
| 241 | 241 | *"not one Uplink issued"*) ;; |
| 242 | 242 | *) echo "::error::/api/license/check did not answer as the license API: ${said:0:200}"; exit 1 ;; |
| 243 | 243 | esac |
| 244 | | - echo "Home page, pricing, checkout, license API, 404s and assets check out." |
| 244 | + |
| 245 | + |
| 246 | + |
| 247 | + [ "$(status no-such-page-here)" = 404 ] || { echo "::error::A missing page answers $(status no-such-page-here), not 404."; exit 1; } |
| 248 | + |
| 249 | + |
| 250 | + |
| 251 | + [ "$(status sitemap.xml)" = 200 ] || { echo "::error::/sitemap.xml answers $(status sitemap.xml)."; exit 1; } |
| 252 | + robots=$(curl -s --max-time 20 "${url}robots.txt") |
| 253 | + case "$robots" in |
| 254 | + *"Sitemap: https://uplink.stacksjs.com/sitemap.xml"*) ;; |
| 255 | + *) echo "::error::/robots.txt is not the one in public/, so something is serving its own: ${robots:0:200}"; exit 1 ;; |
| 256 | + esac |
| 257 | + echo "Home page, pricing, checkout, license API, 404s, robots, sitemap and assets check out." |
Changes to public/README.md | @@ -12,6 +12,16 @@ Let's say you want to overwrite the default `robots.txt` file. You can do this b |
| 12 | 12 | |
| 13 | 13 | If you want to overwrite the default `favicon.ico` file, you can do this by putting a file named `favicon.ico` in this folder. |
| 14 | 14 | |
| 15 | +## Uplink's crawler-facing files |
| 16 | + |
| 17 | +`robots.txt` and `sitemap.xml` are served from here rather than generated by a |
| 18 | +route, so they are the intended home and there is nothing to keep them in sync |
| 19 | +with. **Adding a public page means adding its URL to `sitemap.xml` by hand.** |
| 20 | + |
| 21 | +The deploy workflow checks all three of `/404`, `/sitemap.xml` and |
| 22 | +`/robots.txt` against the live site after every deploy, because a CDN in front |
| 23 | +of the app can serve its own managed `robots.txt` and quietly replace this one. |
| 24 | + |
| 15 | 25 | ## Security |
| 16 | 26 | |
| 17 | 27 | If you want to make sure that a file is not publicly accessible, you can put it in the `storage/private` folder. This folder is not accessible by the web server. |
Changes to public/sitemap.xml | @@ -1,7 +1,15 @@ |
| 1 | 1 | <?xml version="1.0" encoding="UTF-8"?> |
| 2 | +<!-- |
| 3 | + Four pages, matching the `const path` of the four public views. Adding a |
| 4 | + public page means adding it here: nothing generates this file. |
| 5 | + |
| 6 | + No <lastmod>: it was a hardcoded date on every entry, already wrong for `/` |
| 7 | + the day after it was written, and a date that lies is worse for a crawler |
| 8 | + than no date. No <priority> either, which Google has ignored since 2023. |
| 9 | +--> |
| 2 | 10 | <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"> |
| 3 | | - <url><loc>https://uplink.stacksjs.com/</loc><lastmod>2026-09-29</lastmod><priority>1.0</priority></url> |
| 4 | | - <url><loc>https://uplink.stacksjs.com/pricing</loc><lastmod>2026-09-29</lastmod><priority>0.8</priority></url> |
| 5 | | - <url><loc>https://uplink.stacksjs.com/privacy</loc><lastmod>2026-09-29</lastmod><priority>0.3</priority></url> |
| 6 | | - <url><loc>https://uplink.stacksjs.com/terms</loc><lastmod>2026-09-29</lastmod><priority>0.3</priority></url> |
| 11 | + <url><loc>https://uplink.stacksjs.com/</loc></url> |
| 12 | + <url><loc>https://uplink.stacksjs.com/pricing</loc></url> |
| 13 | + <url><loc>https://uplink.stacksjs.com/privacy</loc></url> |
| 14 | + <url><loc>https://uplink.stacksjs.com/terms</loc></url> |
| 7 | 15 | </urlset> |
Changes to resources/views/404.stx | @@ -2,6 +2,10 @@ |
| 2 | 2 | // The not-found page. With `ui.defaultViews: false` the framework's own |
| 3 | 3 | // errors/404 view is not served, so the app answers with this instead. |
| 4 | 4 | const title = 'Not found - Uplink' |
| 5 | +// The body said Not found and the status line said 200, so a crawler indexed |
| 6 | +// every wrong URL as a real page and a client treated the miss as a hit. The |
| 7 | +// dashboard already does this; this page did not. |
| 8 | +notFound() |
| 5 | 9 | </script> |
| 6 | 10 | |
| 7 | 11 | @nolayout |