fix(ssg): sitemap hardcodes `http://localhost`, has no route exclusion, and the build emits no robots.txt

Summary

The SSG's sitemap generator has three gaps that compound: the domain defaults to http://localhost and is never read from the app's site config; SSGConfig has no exclusion option of any kind, so every generated route — auth-walled pages, partials, anything — is advertised; and no robots.txt is produced on this build path even though stx contains two robots generators.

Repro

A two-page fixture (index.stx owning a document, login.stx a fragment), built with the shipped CLI:

$ stx build --pages resources/views --out dist
Generated sitemap.xml with 2 URLs

$ cat dist/sitemap.xml
<url><loc>http://localhost/</loc>…</url>
<url><loc>http://localhost/login</loc>…</url>

$ ls dist
404.html  index.html  login.html  sitemap.xml     # no robots.txt

Impact (real app)

analyticshq declares url: 'https://analyticshq.org' in site.config.ts:7. Every one of the 34 entries in its dist/sitemap.xml is http://localhost/…, and the list includes /account, /dashboard, /login, /register, /sites/new, plus /components/site-nav and /components/site-footer — partials enumerated as public pages.

The app's config/ui.ts:51-53 documents the exclusion half:

Route discovery and the SSG have no exclusion for layout directories, so a layout under resources/views/ is a public URL that gets built to HTML and listed in the sitemap.

Its response was structural: delete resources/views/layouts/ so layouts cannot be routed. For robots it fell back to hand-authoring, and said why in the file (public/robots.txt:8-9):

stx ships a robots generator (site-builder/robots.js) but it only runs from the site-builder build path, which this app does not execute — so this file is authored.

That is 26 hand-written lines with explicit Disallow for /dashboard /account /login /register /sites/ /api/ /payments/, plus robots: 'noindex, nofollow' added to each auth-walled page's useSeoMeta (account.stx:6, sites/new.stx:6, 404.stx:22) as a second line of defence.

Evidence in stx

packages/stx/src/ssg.ts

• :874 const stxConfig = await loadStxConfig() — the config is loaded…
• :881 domain: options.domain || 'http://localhost' — …but no site URL is read from it, and nothing falls back to site.config.
• :1258 const sitemap = generateSitemap(sitemapEntries, cfg.domain)
• grep -n 'exclude' packages/stx/src/ssg.ts → no matches. SSGConfig (:60-120) has sitemap, rss, generate404, trailingSlash … and no way to omit a route.
• Robots generators exist but not here: packages/stx/src/site-builder/robots.ts:4 (generateRobots, site-builder path only) and packages/stx/src/seo.ts:847 (generateRobotsTxt, exported, called by nothing in ssg.ts).

The caller compounds it: Stacks builds views with bunx --bun @stacksjs/stx build --pages resources/views --out dist (storage/framework/core/actions/src/build/views.ts:21) and passes no --domain, even though the CLI accepts one (packages/stx/bin/cli.ts:982).

Prior art

• @nuxtjs/sitemap takes siteUrl and exclude patterns; robots.txt comes from @nuxtjs/robots on the same build.
• Next.js app/sitemap.ts and app/robots.ts are both first-class conventions fed by one config, generated in the same pass.
• spatie/laravel-sitemap requires an explicit base URL — there is no localhost default to fall into.

Ask

1. Default cfg.domain from the loaded site config (site.url / app.url) before falling back to http://localhost, and warn loudly when the fallback is used in a production build.
2. Add sitemapExclude?: (string | RegExp)[] (and honour definePageMeta({ sitemap: false }) / useSeoMeta({ robots: 'noindex' })) so a page can keep itself out.
3. Exclude non-page directories from route discovery by default — components/, partials/, layouts/ under pagesDir should never become public URLs.
4. Generate robots.txt on the SSG path from the same config that drives the sitemap, reusing generateRobotsTxt (seo.ts:847), and write the sitemap URL into it.

Related: #1821

Fixed in 04ca349 — all four asks.

Route discovery (ask 3). This was the load-bearing one, and it wasn't a missing feature — it was two lists that disagreed. codegen.ts:39 had excludeDirs = ['components', 'layouts', 'partials'], but that filter only ever ran over the generated manifest. Router.routes — what serve routes from and what the SSG builds from — had no filter at all. So .stx/routes.ts said one thing and the live route table said another, and the extra entries got served, built to HTML, and published in the sitemap.

The rule now lives in packages/router/src/page-routes.ts, and the scanner skips those directories, so they never enter the route table. It also matches at any depth — the codegen version checked only a leading segment, so views/admin/components/x.stx slipped through even the manifest. A page merely named components.stx is still a page.

Site URL (ask 1). resolveSiteUrl() (packages/stx/src/site-url.ts) checks --domain → stx.config.ts site.url (new field) → site.config.ts → STX_SITE_URL/SITE_URL/URL/VERCEL_URL/CF_PAGES_URL → localhost. Worth noting site.config.ts already had url and the site-builder's sitemap already read it — the two generators were simply never wired together. It returns the source it used, and the build now warns loudly on the fallback, naming the fix.

Exclusion (ask 2). sitemapExclude?: (string | RegExp)[] — exact match, /prefix/ for a subtree, or RegExp. A page can also exclude itself via definePageMeta({ sitemap: false }) or a noindex robots meta it already declares; that last one matters because apps were adding that meta because the sitemap ignored them.

robots.txt (ask 4). Generated via generateRobotsTxt (seo.ts) from the same resolved URL, with a Disallow for every sitemapExclude entry and a Sitemap: line — so the two files can't drift. robots: false opts out; robots: { userAgent, allow, disallow, sitemap } customises.

Verified against a fixture matching the reported shape (resources/views/ with components/, layouts/, /login, /account, and url in site.config.ts):

BEFORE  Found 6 routes   sitemap: http://localhost/{,login,account,components/site-nav,
                                  components/site-footer,layouts/app}   no robots.txt
AFTER   Found 3 routes   sitemap: https://analyticshq.org/    (login+account excluded)
                         robots.txt: Disallow /login, /account + Sitemap: line
                         no dist/components/, no dist/layouts/

On the workaround in config/ui.ts and public/robots.txt — deleting resources/views/layouts/ so layouts can't be routed, and hand-authoring 26 lines of robots.txt, should both be unnecessary now. The hand-written Disallow list can move to sitemapExclude and drive both files from one place. Worth re-running the build before removing anything, though.

Covered by packages/router/test/page-routes.test.ts and packages/stx/test/ssg/sitemap-robots.test.ts (22 tests).