ReviewOS
Reload

also looking at this

stacks/ts-cloud

fix(deploy): keep SQLite across deploys, and preflight an attached host's services

#174
Merged glennmichael123 wants to merge fix/sqlite-shared-path-and-attach-preflight into main
16 files +916 -12

Review threads live on the whole diff, not on one commit, so none are shown here - a thread's line means something in the branch's final form, and painting it into an intermediate step would put it on code it is not about.

docs/config.mdmodified+48-0
@@ -344,6 +344,54 @@ clean. If a site targets a server (`deploy: 'server'`, or `start` set) but no
344344actionable error instead of failing silently at runtime — set `deploy: 'bucket'`
345345or add a server.
346346
347### State that must survive a deploy
348
349Each deploy unpacks into a NEW `releases/<id>` directory and flips `current` at
350it; old releases are pruned. Anything the app writes and must keep therefore has
351to live in the site's `shared/` directory and be symlinked in, which is what
352`site.sharedPaths` declares. `.env` is always shared.
353
354```typescript
355sites: {
356 app: {
357 start: 'bun run server.ts',
358 sharedPaths: ['storage', 'public/uploads'],
359 },
360}
361```
362
363A **SQLite database is shared automatically**. The deploy already knows the
364connection and the file path from the environment it writes to the box, so when
365`DB_CONNECTION` is `sqlite` and `DB_DATABASE` names a path inside the release,
366that file is added to `sharedPaths` for you and the deploy log says so. Without
367it the database sits inside a release directory and the next deploy starts the
368app on an empty one — silently, with the data still in a release that is about
369to be pruned.
370
371Two cases it does not cover:
372
373- **`DB_DATABASE` unset.** An app can default its own path internally, which the
374 deploy never sees. Guessing the filename would report the data as safe while
375 sharing a path the app may not use, so the deploy warns instead — set
376 `DB_DATABASE`, or list the file in `sharedPaths` yourself.
377- **An absolute path.** A database outside the release tree already survives; a
378 deploy replaces the release, not the filesystem around it.
379
380Turning existing on-box state into shared state does not throw it away: the
381first deploy to share a path copies the live release's copy into `shared/`
382(SQLite's `-wal`/`-shm` sidecars included), and a site's first deploy seeds a
383still-empty shared file from the copy the artifact shipped.
384
385Several sites of one project can share ONE file — an app and its API on one
386SQLite database — with the object form, which names an absolute `target`:
387
388```typescript
389sharedPaths: [{ path: 'database/app.sqlite', target: '/var/www/acme-app/shared/database/app.sqlite', seed: false }]
390```
391
392Each site installs under its own base, so a plain string would give each of them
393a database of its own. `seed: false` marks the sites that do not own the file.
394
347395### CDN / caching
348396
349397The `cache` hint applies to either origin:
packages/core/src/types.tsmodified+5-0
@@ -1243,6 +1243,11 @@ export interface SiteConfig {
12431243 * A release is a fresh directory, so anything the app WRITES and must keep
12441244 * has to be listed here or the next deploy silently starts it from empty.
12451245 *
1246 * A SQLite database is the one exception, added for you: when the site's
1247 * resolved env says `DB_CONNECTION=sqlite` and `DB_DATABASE` names a path
1248 * inside the release, the deploy shares that file without being asked. An
1249 * env that says SQLite but not WHERE is warned about rather than guessed at.
1250 *
12461251 * An entry may instead be a {@link SharedPathSpec} naming an absolute
12471252 * `target`, which is how SEVERAL sites of one project point at ONE file —
12481253 * an app and its API sharing a single SQLite database, say. Each site
packages/ts-cloud/src/drivers/shared/compute-deploy.tsmodified+9-0
@@ -17,6 +17,7 @@ import { resolveNotifications, sendNotifications } from './notifications'
1717import { buildPhpFpmPoolScript, phpFpmPoolListen } from './php-fpm-pool'
1818import { buildDeployHistoryHeader, buildSiteOwnerGuard } from './releases'
1919import { buildRpxConfig, buildRpxFragmentRefreshScript, buildRpxLbConfig, buildRpxProvisionScript, certDomainsForConfig, rpxCertRenewServiceName, usesRpxProxy } from './rpx-gateway'
20import { inferSqliteSharedPath } from './sqlite-shared-path'
2021
2122export interface ComputeDeployLogger {
2223 info(message: string): void
@@ -95,6 +96,14 @@ export async function deploySiteRelease(
9596 ...productionEnv,
9697 }
9798
99 // A SQLite database inside the release directory is thrown away by the next
100 // deploy. The script builders share it automatically when the env says where
101 // it lives; when the env says SQLite but not where, only the operator can
102 // resolve it — so say so here rather than deploying quietly over it.
103 const sqlite = inferSqliteSharedPath(envWithServices, site.sharedPaths ?? [])
104 if (sqlite.warning) logger.warn(`Site '${siteName}': ${sqlite.warning}`)
105 else if (sqlite.path) logger.info(`Site '${siteName}': sharing '${sqlite.path}' so the SQLite database survives deploys.`)
106
98107 // PHP/Laravel sites deploy via git clone + atomic releases on the box (no
99108 // tarball upload). The box clones the repo, runs the deploy script inside the
100109 // new release, flips `current`, then nginx is (re)pointed at it.
packages/ts-cloud/src/drivers/shared/deploy-script.tsmodified+10-1
@@ -13,6 +13,7 @@
1313import type { SharedPathEntry } from '@ts-cloud/core'
1414import { formatEnvFile } from './env-file'
1515import { buildActivateRelease, buildDeployLock, buildEnsureReleaseLayout, buildLinkSharedPaths, buildPromoteStagedRelease, buildPruneReleases, buildResetReleaseDir, buildStrandedReleaseTrap, dedupeSharedPaths, DEFAULT_KEEP_RELEASES, releasePaths } from './releases'
16import { sqliteSharedPaths } from './sqlite-shared-path'
1617
1718/**
1819 * Translate a `start` command (e.g. "bun run server.ts") into an absolute
@@ -184,7 +185,15 @@ export function buildSiteDeployScript(options: BuildSiteDeployScriptOptions): st
184185 const serviceName = `${unitBase}.service`
185186 const tarball = releaseTarballTmpPath(slug, siteName, releaseId)
186187 // `.env` is always shared; a site adds anything else it writes and must keep.
187 const sharedPaths = dedupeSharedPaths(['.env', ...(options.sharedPaths ?? [])])
188 // A SQLite database is added for it: the deploy already knows the connection
189 // and the file path from the env it is about to write, and an undeclared
190 // SQLite file inside the release is discarded by the NEXT deploy.
191 const declaredSharedPaths = options.sharedPaths ?? []
192 const sharedPaths = dedupeSharedPaths([
193 '.env',
194 ...declaredSharedPaths,
195 ...sqliteSharedPaths(envEntries, declaredSharedPaths),
196 ])
188197
189198 const envFile = formatEnvFile(envEntries)
190199
packages/ts-cloud/src/drivers/shared/laravel-deploy.tsmodified+6-1
@@ -19,6 +19,7 @@ import { formatEnvFile } from './env-file'
1919import { buildGitCheckoutScript } from './git-deploy'
2020import { PANTRY_PROJECT_DIR, pantryEnvActivation } from './package-manager'
2121import { buildActivateRelease, buildDeployHistoryHeader, buildEnsureReleaseLayout, buildLinkSharedPaths, buildPruneReleases, DEFAULT_KEEP_RELEASES, DEFAULT_SHARED_PATHS, releasePaths } from './releases'
22import { sqliteSharedPaths } from './sqlite-shared-path'
2223
2324export const MACRO_CREATE_RELEASE = '$CREATE_RELEASE'
2425export const MACRO_ACTIVATE_RELEASE = '$ACTIVATE_RELEASE'
@@ -142,7 +143,11 @@ export function buildLaravelDeployScript(options: LaravelDeployOptions): string[
142143 // pinned at install time (php.net@<version>), not in the deploy command.
143144 const phpBin = 'php'
144145 const paths = releasePaths(base, releaseId)
145 const sharedPaths = site.sharedPaths ?? DEFAULT_SHARED_PATHS
146 const declaredSharedPaths = site.sharedPaths ?? DEFAULT_SHARED_PATHS
147 // Share the app's SQLite database automatically when the env names one inside
148 // the release — see `./sqlite-shared-path`. A `database/*.sqlite` left out of
149 // `sharedPaths` is thrown away by the next deploy.
150 const sharedPaths = [...declaredSharedPaths, ...sqliteSharedPaths(site.env, declaredSharedPaths)]
146151 const keepReleases = site.keepReleases ?? DEFAULT_KEEP_RELEASES
147152 const template = site.deployScript?.length ? site.deployScript : defaultDeployScriptFor(site.type ?? 'laravel')
148153
packages/ts-cloud/src/drivers/shared/releases.tsmodified+28-0
@@ -339,6 +339,31 @@ export function buildPromoteStagedRelease(paths: ReleasePaths): string[] {
339339 ]
340340}
341341
342/**
343 * Seed an as-yet-empty shared FILE from the copy the incoming release shipped.
344 *
345 * {@link buildAdoptSharedPathFn} rescues state from the release that is
346 * currently live, which covers a path that becomes shared on an existing site.
347 * It cannot cover a site's FIRST deploy: there is no live release to adopt
348 * from, so the layout step leaves a zero-byte placeholder and the link below
349 * would replace the artifact's real file with it — an app shipping a seeded
350 * SQLite database would come up empty on the very deploy that created it.
351 *
352 * Narrow on purpose: only a regular, non-empty file in the release, and only
353 * when the shared target is still zero bytes (what a placeholder looks like,
354 * and what no real SQLite database ever is — a database with any schema in it
355 * is at least one page). It can therefore only ever put content where there
356 * was none.
357 */
358function buildSeedSharedFromRelease(link: string, target: string): string[] {
359 return [
360 `if [ -f ${link} ] && [ ! -L ${link} ] && [ -s ${link} ] && [ ! -s ${target} ]; then`,
361 ` cp -a ${link} ${target}`,
362 ` echo "[ts-cloud] seeded shared/ from the release's own copy of the file"`,
363 'fi',
364 ]
365}
366
342367/**
343368 * Symlink every shared path from `shared/` into the freshly checked-out release,
344369 * replacing whatever the checkout shipped (e.g. the repo's empty `storage`).
@@ -355,6 +380,9 @@ export function buildLinkSharedPaths(
355380
356381 // A site that owns the target always links: the layout step just created it.
357382 if (seed) {
383 // `.env` is excluded: the deploy writes the shared one itself, and the
384 // release's own env files are deleted before this runs.
385 if (p !== '.env' && isFileSharedPath(p)) lines.push(...buildSeedSharedFromRelease(link, target))
358386 lines.push(...relink)
359387 continue
360388 }
packages/ts-cloud/src/drivers/shared/sqlite-shared-path.test.tsadded+96-0
@@ -0,0 +1,96 @@
1import { describe, expect, it } from 'bun:test'
2import { buildSiteDeployScript } from './deploy-script'
3import { inferSqliteSharedPath, sqliteSharedPaths, usesSqlite } from './sqlite-shared-path'
4
5describe('inferSqliteSharedPath', () => {
6 it('shares a release-relative sqlite file', () => {
7 expect(inferSqliteSharedPath({ DB_CONNECTION: 'sqlite', DB_DATABASE: 'database/stacks.sqlite' }))
8 .toEqual({ path: 'database/stacks.sqlite' })
9 })
10
11 it('accepts the sqlite3 spelling and a leading ./', () => {
12 expect(inferSqliteSharedPath({ DB_CONNECTION: 'SQLite3', DB_DATABASE: './database/app.sqlite' }))
13 .toEqual({ path: 'database/app.sqlite' })
14 })
15
16 it('leaves a non-sqlite app alone', () => {
17 expect(inferSqliteSharedPath({ DB_CONNECTION: 'pgsql', DB_DATABASE: 'loghq' })).toEqual({})
18 expect(usesSqlite({ DB_CONNECTION: 'mysql' })).toBe(false)
19 })
20
21 /**
22 * A path outside the release is not replaced by a deploy — it needs no
23 * shared/ entry, and inventing one would symlink a path the app does not use.
24 */
25 it('leaves an absolute path alone', () => {
26 expect(inferSqliteSharedPath({ DB_CONNECTION: 'sqlite', DB_DATABASE: '/var/data/app.sqlite' })).toEqual({})
27 })
28
29 it('refuses a path escaping the release', () => {
30 expect(inferSqliteSharedPath({ DB_CONNECTION: 'sqlite', DB_DATABASE: '../shared/app.sqlite' })).toEqual({})
31 })
32
33 it('leaves an in-memory database alone', () => {
34 expect(inferSqliteSharedPath({ DB_CONNECTION: 'sqlite', DB_DATABASE: ':memory:' })).toEqual({})
35 })
36
37 /**
38 * Guessing the filename would report the data as safe while sharing a path
39 * the app may never write. Say what is unknown instead.
40 */
41 it('warns when the app is on sqlite but names no file', () => {
42 const { path, warning } = inferSqliteSharedPath({ DB_CONNECTION: 'sqlite' })
43 expect(path).toBeUndefined()
44 expect(warning).toContain('DB_DATABASE')
45 })
46
47 it('does not duplicate a path the site already declares', () => {
48 const env = { DB_CONNECTION: 'sqlite', DB_DATABASE: 'database/app.sqlite' }
49 expect(inferSqliteSharedPath(env, ['database/app.sqlite'])).toEqual({})
50 // Including the spec form, which is how sibling sites share one database.
51 expect(inferSqliteSharedPath(env, [{ path: 'database/app.sqlite', target: '/var/www/api/shared/db.sqlite' }]))
52 .toEqual({})
53 expect(sqliteSharedPaths(env, ['database/app.sqlite'])).toEqual([])
54 })
55})
56
57describe('server-app deploy script', () => {
58 const script = (env: Record<string, string>): string =>
59 buildSiteDeployScript({
60 siteName: 'app',
61 slug: 'acme',
62 artifactFetch: [],
63 releaseId: 'rel1',
64 execStart: '/usr/local/bin/bun run server.ts',
65 envEntries: env,
66 port: 3000,
67 }).join('\n')
68
69 /**
70 * The failure this guards against: a SQLite file written inside the release
71 * directory, which the NEXT deploy leaves behind in a pruned release.
72 */
73 it('shares the sqlite database without being told to', () => {
74 const out = script({ DB_CONNECTION: 'sqlite', DB_DATABASE: 'database/stacks.sqlite' })
75 expect(out).toContain('/var/www/app/shared/database/stacks.sqlite')
76 expect(out).toContain('ln -sfn /var/www/app/shared/database/stacks.sqlite /var/www/app/releases/rel1/database/stacks.sqlite')
77 })
78
79 it('records it in the shared-paths manifest, so a rollback relinks it', () => {
80 const out = script({ DB_CONNECTION: 'sqlite', DB_DATABASE: 'database/stacks.sqlite' })
81 expect(out).toContain('database/stacks.sqlite\t/var/www/app/shared/database/stacks.sqlite')
82 })
83
84 it('adopts the live copy before placeholding it, so existing data is not lost', () => {
85 const out = script({ DB_CONNECTION: 'sqlite', DB_DATABASE: 'database/stacks.sqlite' })
86 const adopt = out.indexOf("ts_cloud_adopt_shared 'database/stacks.sqlite'")
87 const touch = out.indexOf('touch /var/www/app/shared/database/stacks.sqlite')
88 expect(adopt).toBeGreaterThan(-1)
89 expect(touch).toBeGreaterThan(adopt)
90 })
91
92 it('adds nothing for a postgres app', () => {
93 const out = script({ DB_CONNECTION: 'pgsql', DB_DATABASE: 'loghq' })
94 expect(out).not.toContain('shared/loghq')
95 })
96})
packages/ts-cloud/src/drivers/shared/sqlite-shared-path.tsadded+123-0
@@ -0,0 +1,123 @@
1/**
2 * Infer the shared path for an app whose database is SQLite.
3 *
4 * A release directory is disposable: the next deploy is a NEW directory and the
5 * old one is pruned. Anything the app writes and must keep therefore has to live
6 * in `shared/` and be symlinked in (see {@link import('./releases')}). `.env` is
7 * shared implicitly; everything else has to be declared in `site.sharedPaths`.
8 *
9 * A SQLite database is the one case ts-cloud can work out on its own, because
10 * the connection and the file path are already in the environment it writes to
11 * the box. Left undeclared, the file lands inside the release, and the deploy
12 * after it starts the app on an empty database with no warning — the whole
13 * dataset is one deploy from being orphaned inside a pruned release.
14 *
15 * So: read `DB_CONNECTION`/`DB_DATABASE` out of the site's resolved env, and
16 * when they describe a release-relative SQLite file, share it automatically.
17 * When they say SQLite but do not say where, say so loudly instead of guessing
18 * a filename — a wrong guess would share a path the app never writes and leave
19 * the real database exactly as exposed, while reporting that it is safe.
20 */
21import type { SharedPathEntry } from '@ts-cloud/core'
22import { sharedPathOf } from './releases'
23
24/**
25 * `DB_CONNECTION` values that mean "a SQLite file on this box". Stacks and
26 * Laravel both spell it `sqlite`; `sqlite3` shows up in hand-written configs
27 * and PDO DSNs.
28 */
29const SQLITE_CONNECTIONS = new Set(['sqlite', 'sqlite3'])
30
31/** In-memory databases have no file to keep — nothing to share, nothing to warn about. */
32const IN_MEMORY = new Set([':memory:', 'memory'])
33
34/** Is this site's resolved environment pointing at a SQLite database? */
35export function usesSqlite(env: Record<string, string | undefined> | undefined): boolean {
36 const connection = env?.DB_CONNECTION?.trim().toLowerCase()
37 return connection != null && SQLITE_CONNECTIONS.has(connection)
38}
39
40/**
41 * Normalize the configured database path to a release-relative one.
42 *
43 * Returns `undefined` when the path is not release-relative — absolute (it
44 * already lives outside the release and survives on its own) or escaping the
45 * release via `..` (which `shared/` cannot express, and which a symlink would
46 * point somewhere surprising).
47 */
48function releaseRelativePath(value: string): string | undefined {
49 const trimmed = value.trim().replace(/^\.\//, '')
50 if (trimmed === '' || trimmed.startsWith('/') || trimmed.startsWith('~')) return undefined
51 if (trimmed.split('/').includes('..')) return undefined
52 return trimmed.replace(/\/+$/, '')
53}
54
55export interface SqliteSharedPathInference {
56 /**
57 * Release-relative path to add to the site's shared paths. Absent when the
58 * app is not on SQLite, the file already survives deploys, or the path could
59 * not be determined.
60 */
61 path?: string
62 /**
63 * Why nothing could be inferred even though the app IS on SQLite — an
64 * operator-facing sentence naming what to declare. Absent when there is
65 * nothing to worry about.
66 */
67 warning?: string
68}
69
70/**
71 * Work out whether a site's SQLite database needs to be added to its shared
72 * paths, given the environment the deploy writes to the box and whatever the
73 * site already declares.
74 */
75export function inferSqliteSharedPath(
76 env: Record<string, string | undefined> | undefined,
77 declared: readonly SharedPathEntry[] = [],
78): SqliteSharedPathInference {
79 if (!usesSqlite(env)) return {}
80
81 const configured = env?.DB_DATABASE?.trim()
82 // An in-memory database has no file to keep — nothing to share, and nothing
83 // an operator could do about it if there were.
84 if (configured && IN_MEMORY.has(configured.toLowerCase())) return {}
85
86 if (!configured) {
87 // An app can default its own database path internally (Stacks writes
88 // `database/stacks.sqlite`), which the deploy never sees. Guessing that
89 // filename would share a path the app may not use and report the data as
90 // safe when it is not, so name the problem instead.
91 return {
92 warning:
93 'DB_CONNECTION is sqlite but DB_DATABASE names no file, so ts-cloud cannot tell where the database lives. '
94 + 'If it is written inside the release directory, the next deploy discards it. '
95 + "Set DB_DATABASE, or list the file in the site's `sharedPaths`.",
96 }
97 }
98
99 const relative = releaseRelativePath(configured)
100 // Absolute (or `..`-escaping) paths are outside the release tree already —
101 // a deploy replaces the release, not the filesystem around it.
102 if (!relative) return {}
103
104 // Already declared — the operator's own entry wins, including a
105 // `SharedPathSpec` pointing at a database shared with a sibling site.
106 if (declared.some(entry => sharedPathOf(entry) === relative)) return {}
107
108 return { path: relative }
109}
110
111/**
112 * The shared-path entries to append for a site's SQLite database — `[]` when
113 * there is nothing to add. Kept separate from {@link inferSqliteSharedPath} so
114 * script builders can splice it in without having to care about the warning,
115 * which only the deploy driver can surface.
116 */
117export function sqliteSharedPaths(
118 env: Record<string, string | undefined> | undefined,
119 declared: readonly SharedPathEntry[] = [],
120): SharedPathEntry[] {
121 const { path } = inferSqliteSharedPath(env, declared)
122 return path ? [path] : []
123}
packages/ts-cloud/test/drivers/shared-paths.test.tsmodified+43-0
@@ -154,3 +154,46 @@ describe('shared paths pointed at a project-level target', () => {
154154 expect(buildLinkSharedPaths(paths, [spec]).join('\n')).not.toContain('if [ -e ')
155155 })
156156})
157
158/**
159 * Adoption rescues state from the LIVE release, which covers a path that
160 * becomes shared on an existing site. A site's FIRST deploy has no live release
161 * to adopt from, so without this the empty placeholder would replace whatever
162 * the artifact shipped — an app shipping a seeded SQLite database would come up
163 * empty on the very deploy that created it.
164 */
165describe('seeding a shared file from the incoming release', () => {
166 const link = (shared: SharedPathEntry[]): string => buildLinkSharedPaths(paths, shared).join('\n')
167
168 it('copies the release copy in when the shared file is still a placeholder', () => {
169 const out = link(['database/app.sqlite'])
170 expect(out).toContain('if [ -f /var/www/site/releases/rel1/database/app.sqlite ]')
171 expect(out).toContain('[ ! -s /var/www/site/shared/database/app.sqlite ]')
172 expect(out).toContain('cp -a /var/www/site/releases/rel1/database/app.sqlite /var/www/site/shared/database/app.sqlite')
173 })
174
175 it('still links the shared file afterwards', () => {
176 const out = link(['database/app.sqlite'])
177 const seed = out.indexOf('cp -a /var/www/site/releases/rel1/database/app.sqlite')
178 const ln = out.indexOf('ln -sfn /var/www/site/shared/database/app.sqlite')
179 expect(seed).toBeGreaterThan(-1)
180 expect(ln).toBeGreaterThan(seed)
181 })
182
183 it('leaves .env alone — the deploy writes the shared one itself', () => {
184 expect(link(['.env'])).not.toContain('cp -a')
185 })
186
187 it('leaves directories alone', () => {
188 expect(link(['storage'])).not.toContain('cp -a')
189 })
190
191 /**
192 * A site that does not own the target must not seed it: the owner's own
193 * deploy adopts or writes it, and a copy from here would make that a no-op.
194 */
195 it('leaves a target this site does not own alone', () => {
196 expect(link([{ path: 'database/app.sqlite', target: '/var/www/api/shared/app.sqlite', seed: false }]))
197 .not.toContain('cp -a')
198 })
199})