ReviewOS

stacks/stx

dev: detect + auto-rebuild stale pantry runtime; simplify pantry/node_modules/cache resolution story

#1745
Closed glennmichael123 opened this 23 days ago · 3 comments
23 days ago

Problem

An adopter who edits a pantry/@stacksjs/stx/src/*.ts file has to manually track FIVE layers of staleness before the change reaches the browser:

  1. Pantry src — the file you edit
  2. Pantry dist — must rebuild with bun build.ts after src edit
  3. node_modules/@stacksjs/stx/dist/ — Bun.resolveSync resolves here, NOT pantry. May or may not be symlinked.
  4. bun-plugin-stx bundled runtime — its serve.ts bundles the runtime at boot, caches in memory
  5. _cachedSignalsRuntime module-level cache — inside caching.ts, persists across requests until process restart

I personally hit each of these in the last week. The maintainer hit it too in their #1737 close comment ("the bug was a #1735 stale-bundled-runtime failure mode"). It's a real, repeated developer experience problem.

Symptoms

  • Edit pantry/@stacksjs/stx/src/signals.ts, restart dev server, change doesn't appear → forgot to rebuild dist
  • Rebuild dist, restart dev server, change doesn't appear → node_modules/@stacksjs/stx isn't symlinked to pantry
  • Symlink + restart, change doesn't appear → bun-plugin-stx cached the runtime in memory pre-restart
  • Restart bun-plugin-stx, change doesn't appear → _cachedSignalsRuntime was hit by a request between restart and your verify-curl
  • Hard-reload page, change doesn't appear → browser cached the old runtime IIFE blob

That's a 5-level fan of "did I clear all the caches?"

Proposed

A. Auto-rebuild pantry dist on src edit

Watch pantry/@stacksjs/stx/src/** in dev mode. On change:

  • Run bun build.ts automatically
  • Invalidate _cachedSignalsRuntime
  • Send HMR signal to all connected browsers

Same pattern as Vite's dependency pre-bundling watch.

If pantry/@stacksjs/stx exists, bun install should symlink the corresponding node_modules/@stacksjs/stx entry to it. Adopter doesn't have to remember to do this manually.

Per project memory project_stx_pantry_override.md: "pantry is canonical for framework packages." The framework should enforce that resolution path.

C. Invalidate _cachedSignalsRuntime on every dev-mode dist change

Today's caching.ts:113:

if (debug) {
  // In debug mode, always regenerate to avoid stale runtime issues
  const { generateSignalsRuntimeDev } = await import('./signals')
  return generateSignalsRuntimeDev()
}
if (_cachedSignalsRuntime === null) { ... }
return _cachedSignalsRuntime

In dev mode (non-debug), the cache is held forever. Should be:

  • File-mtime-based invalidation (check dist/signals.js mtime on each request, regenerate if changed)
  • OR file-watcher-driven invalidation

D. Make the resolution clear in ./buddy doctor

$ ./buddy doctor

stx runtime resolution:
  ✓ pantry/@stacksjs/stx/src/signals.ts (mtime: 5 min ago)
  ✓ pantry/@stacksjs/stx/dist/signals.js (mtime: 4 min ago, rebuilt after src ✓)
  ✓ node_modules/@stacksjs/stx → pantry (symlinked correctly)
  ✓ bun-plugin-stx loaded from: pantry/bun-plugin-stx/dist/serve.js
  ✓ Served runtime length: 100956 chars (matches expected)

If any step is misaligned, error clearly:

$ ./buddy doctor

stx runtime resolution:
  ✓ pantry/@stacksjs/stx/src/signals.ts (mtime: 5 min ago)
  ✗ pantry/@stacksjs/stx/dist/signals.js (mtime: 2 hours ago, OLDER than src by 1h 55m)
     → Run: cd pantry/@stacksjs/stx && bun build.ts
  ⚠ node_modules/@stacksjs/stx is a real dir, NOT symlinked to pantry
     → Run: rm -rf node_modules/@stacksjs/stx && ln -s ../../pantry/@stacksjs/stx node_modules/@stacksjs/stx

Prior art

  • Vite — file watcher rebuilds dependencies + invalidates cache + HMRs the browser. Zero manual steps.
  • Next.js — same model.
  • Vue CLI / Vue dev server — same.
  • SvelteKit — same.

The "edit source, see change in browser" loop is table stakes. Stacks today has the longest "edit-to-see" lag of any modern framework I've worked in.

Impact

  • Per-edit time: cuts dev-loop iteration time from ~30 sec (manual rebuild + restart cycle) to ~3 sec (auto).
  • Cognitive load: adopters stop tracking "did I clear cache N?" — there is one cache and it's managed.
  • Onboarding: new adopter setup goes from "follow the symlink + rebuild checklist" to "just bun install and run."

Phasing

  • Phase 1: ./buddy doctor reports staleness clearly (Section D)
  • Phase 2: File-mtime invalidation of _cachedSignalsRuntime (Section C)
  • Phase 3: Auto-symlink node_modules → pantry on bun install (Section B)
  • Phase 4: Watch + auto-rebuild pantry dist in dev mode (Section A)

Happy to PR Phase 1.

22 days ago

Item C landed in 73abece.

The dev file-watcher cleared the app-render caches (htmlCache, crosswindCssCache, routes, sourceFiles) on every .stx edit but never the stx package's framework-level caches (fileContentCache, signals runtime, router script) — those survived until a full process restart. So a stale @included partial whose transitive dependency slipped past the mtime tracker (the #1926 failure class) could outlive an edit. The watcher now also calls the resolved module's clearDevCaches().

Race-free + regression-proof: a sync handle to the resolved stx module is captured once at startup (before any edit), and the call is optional-chained — a missing module/method is a no-op, never a failure. A unit test pins the cache-hit vs regenerate-after-invalidate semantics via a counting-generator mock.

Honest scope note — items A & B remain open: item C closes the cache-coherence gap, but the headline "edit pantry/@stacksjs/stx/src/*.ts and see it live without restart" still needs a process restart. A running JS process can't hot-swap the generateSignalsRuntime function code regardless of cache state — clearing _cachedSignalsRuntime re-runs the same (already-loaded) generator. The real fix for that is item A (watch pantry src → bun build.ts → restart the dev process) and item B (auto-symlink node_modules/@stacksjs/stx → pantry). Leaving this issue open for A + B.

22 days ago

Phase 1 / Section D — stx doctor: shipped in 17879bb35f (on main).

It makes the staleness chain visible — inspecting each layer a framework-source edit must cross and reporting which one is stale, with the exact fix:

stx runtime resolution:
  ✓ @stacksjs/stx resolves to: …/node_modules/@stacksjs/stx/dist/index.js
  ✗ pantry dist freshness: dist (2h ago) is OLDER than src/signals.ts (5m ago) by … — your edit isn't built
     → cd pantry/@stacksjs/stx && bun build.ts
  ⚠ node_modules → pantry: is a real directory, NOT symlinked to pantry — edits to pantry src/dist won't be resolved
     → rm -rf node_modules/@stacksjs/stx && ln -s ../../pantry/@stacksjs/stx node_modules/@stacksjs/stx
  · dev runtime cache: cleared automatically on source edits by the dev watcher (clearDevCaches, #1745 item C)
  ✓ signals runtime: generated 102,685 chars

It exits non-zero when a layer is stale (so it can gate CI or a pre-dev check) and supports --json. Logic is in a pure, injectable src/doctor.ts (runDoctor/formatDoctorReport/findPantryPackage), unit-tested against simulated pantry layouts (test/doctor/doctor.test.ts).

Status of the other sections:

  • Section C (Phase 2) — already done in 73abece576: the dev watcher invalidates the stx framework runtime cache (clearDevCaches) on every source edit, so layer #5 (_cachedSignalsRuntime surviving forever) is gone.
  • Section A (Phase 4) auto-rebuild pantry dist on watch, and Section B (Phase 3) auto-symlink node_modules → pantry on bun install — these are Stacks/buddy/pantry tooling, not stx-framework concerns (the stx repo has no pantry; bun install hooks + the symlink policy live in buddy). They'd belong in buddy/the Stacks meta-repo. stx doctor already surfaces both as actionable warnings with the one-liner fix in the meantime.

So the in-stx-framework scope of this issue (C + D) is complete. I'd suggest the remaining A/B move to a buddy issue — happy to close here, or leave open as the tracking issue for them.

22 days ago

Closing — the two parts of this that live in the stx framework are both done:

  • Section C / Phase 2 (invalidate the framework runtime cache on source edits) — shipped in 73abece576 (the dev watcher calls clearDevCaches()).
  • Section D / Phase 1 (stx doctor — make the staleness chain visible: pantry dist freshness, node_modules→pantry symlink, resolution, runtime length) — shipped in 17879bb35f.

Sections A and B (auto-rebuild pantry dist on watch; auto-symlink node_modules→pantry on bun install) are Stacks/buddy/pantry tooling — the stx repo has no pantry, and the bun install hooks + symlink policy live in buddy. stx doctor already surfaces both as actionable warnings with the one-liner fix in the meantime. They'd belong in a buddy issue; happy to reopen/refile there if wanted.

Sign in to comment on this issue.