test: retain execution lifetime failure observations

.github/workflows/ci.ymlmodified+14-1

Changes to .github/workflows/ci.yml

		⋯57 linesShow the 57 lines above this hunk@@ -58,6 +58,19 @@ jobs:
58	58	env:
59	59	RUNTIME_HOST_FILE_LIMIT_EVIDENCE: ${{ runner.temp }}/runtime-host-file-limit-full-evidence
60	60	RUNTIME_MAINTENANCE_EVIDENCE: ${{ runner.temp }}/runtime-maintenance-full-evidence
	61	+ EXECUTION_LIFETIME_EVIDENCE: ${{ runner.temp }}/execution-lifetime-failures
	62	+ - uses: actions/upload-artifact@v4
	63	+ if: always()
	64	+ with:
	65	+ name: execution-lifetime-failure-evidence
	66	+ if-no-files-found: ignore
	67	+ retention-days: 7
	68	+ path: |
	69	+ ${{ runner.temp }}/execution-lifetime-failures/*/fixture.stdout
	70	+ ${{ runner.temp }}/execution-lifetime-failures/*/fixture.stderr
	71	+ ${{ runner.temp }}/execution-lifetime-failures/*/lifecycle.json
	72	+ ${{ runner.temp }}/execution-lifetime-failures/*/access.sqlite*
	73	+ ${{ runner.temp }}/execution-lifetime-failures/*/bunfig.toml
61	74	- uses: actions/upload-artifact@v4
62	75	if: always()
63	76	with:
		⋯401 linesShow the 401 lines above this hunk@@ -465,7 +478,7 @@ jobs:
465	478	- run: bun install --frozen-lockfile
466	479	- run: cp .env.example .env
467	480	- run: ./buddy key:generate
468		- - name: Qualify actual ENOSPC and fresh runtime recovery in a private filesystem
	481	+ - name: Qualify actual zero-byte and partial ENOSPC writes with fresh runtime recovery
469	482	shell: bash
470	483	run: |
471	484	set -euo pipefail

docs/adr/0031-full-host-filesystem-fixture.mdmodified+13-4

Changes to docs/adr/0031-full-host-filesystem-fixture.md

		⋯12 linesShow the 12 lines above this hunk@@ -13,17 +13,25 @@ Keep signed repositories and the owned build outside the bounded filesystem.
13	13	Place the actual protocol store, candidates and slots inside it. Confirm a real
14	14	release-7 host session, prepare release 9, then publish new signed root/role
15	15	metadata. At the public slot-selection progress boundary, fill an owned sibling
16		-file through real writes until ENOSPC and release only 1 MiB of its allocation.
17		-The larger signed image must make actual partial progress before ENOSPC. Leave
	16	+file through real writes until ENOSPC. Run two independent campaigns: release
	17	+only 1 MiB of its allocation for a partial image write, or leave zero free bytes
	18	+for failure before any image bytes are written. Pin the selected budget in the
	19	+private fixture marker and worker input, and read it back from the filesystem
	20	+before the copy. Require exactly the selected number of retained image bytes,
	21	+including an existing empty image file and its SHA-256 in the zero-byte case. Leave
18	22	the filesystem full until the trial process has exited, including its attempts
19	23	to record failure. Never replace filesystem or SQLite methods with faults.
	24	+The existing public inspector reports an empty image as `unavailable` because
	25	+valid protected images must be nonempty. A nonempty partial image is `mismatch`.
	26	+Neither status grants health or installation authority; the campaign separately
	27	+reads the actual retained file to distinguish empty, partial and missing bytes.
20	28
21	29	Inspect retained bytes and public status after removing only the owned filler.
22	30	Require the previous known-good image and accepted metadata/security floors to
23	31	survive without a new health confirmation. A pending attempt and selected slot
24	32	must undergo explicit fresh-process metadata recovery and exact-slot abandonment
25	33	before an unrestricted fresh trial can confirm a different slot through actual
26		-new boot/session evidence. Preserve failed partial bytes and the older image.
	34	+new boot/session evidence. Preserve failed partial or empty files and the older image.
27	35
28	36	Copy and hash the full volume before freeing the filler or opening it for
29	37	inspection, then archive later retained states before unmounting the volatile
		⋯5 linesShow the 5 lines above this hunk@@ -35,6 +43,7 @@ no positive filesystem-exhaustion evidence.
35	43	Linux documents tmpfs capacity limits and loss of contents on unmount in
36	44	[tmpfs(5)](https://man7.org/linux/man-pages/man5/tmpfs.5.html). This is real logical
37	45	filesystem exhaustion on a memory-backed filesystem. It does not qualify storage
38		-device exhaustion, persistent-media durability, EDQUOT, EIO, sync failures or
	46	+device exhaustion, a write syscall returning zero without an error,
	47	+persistent-media durability, EDQUOT, EIO, sync failures or
39	48	electrical power loss. Those remain separate acceptance requirements. The fixture
40	49	adds no production storage layer, privileged application command or dependency.

docs/public-safety-execution.mdmodified+22-0

Changes to docs/public-safety-execution.md

		⋯115 linesShow the 115 lines above this hunk@@ -116,3 +116,25 @@ EXECUTION_BROWSER_QA=true BROWSE_BROWSER=/path/to/google-chrome bun test tests/u
116	116	Native controller and HTTP checks separately cover authority replacement,
117	117	elapsed deadlines, stream capacity, cleanup and durable COMMIT with a withheld
118	118	response. The existing workspace browser scenarios remain required regressions.
	119	+
	120	+The native lifetime fixture emits bounded phase observations around setup,
	121	+authority reads, retention waits and cleanup. Parent-start and child-runtime
	122	+records identify their Bun versions and executable paths independently. Its parent records the observed
	123	+exit code and signal separately from whether its own watchdog fired or cleanup
	124	+requested a kill. The settled `exited` status is distinct from the raw
	125	+`exitCode` property, which can remain null after a signal. Elapsed time includes process work and output handling, not
	126	+just a native check. Failed cases retain their private temporary directory,
	127	+stdout, stderr and lifecycle receipt when storage permits. A fixed warning
	128	+identifies each unavailable evidence file if its write fails; this does not
	129	+replace the captured child diagnostics or its original test verdict. Successful
	130	+cases remove their temporary files as before. CI sets `EXECUTION_LIFETIME_EVIDENCE`
	131	+to an owned runner directory and retains the failed synthetic database, process
	132	+configuration and captured diagnostics for seven days, using explicit file
	133	+patterns that exclude linked dependencies. No account tokens, scope identifiers or recording contents are included
	134	+in the scalar phase and lifecycle observations.
	135	+
	136	+These observations support [the unresolved Mac failure](https://github.com/stacksjs/drone/issues/60);
	137	+they do not establish its cause or repair it. The production authority-check
	138	+limit, actual 60-second retention, 90-second child watchdog and 100-second test
	139	+guard are unchanged. An observed signal or watchdog marker alone does not prove
	140	+an individually joined Worker thread or the cause of an earlier exit.

docs/runtime-updates.mdmodified+9-5

Changes to docs/runtime-updates.md

		⋯644 linesShow the 644 lines above this hunk@@ -645,9 +645,13 @@ EIO, sync and physical power-loss campaigns open.
645	645
646	646	The opt-in [full-filesystem qualification](adr/0031-full-host-filesystem-fixture.md)
647	647	uses a separate private Linux mount namespace and a new bounded tmpfs for the
648		-actual store, candidates and slots. It targets real ENOSPC during the image copy
649		-and SQLite failure-state persistence, then exercises explicit metadata recovery,
	648	+actual store, candidates and slots. Separate cases leave either 1 MiB or zero
	649	+free bytes after actual ENOSPC filling. They require exactly a 1 MiB authenticated
	650	+prefix or an existing empty image file, with no runtime launch or health claim.
	651	+Both target real ENOSPC during the image copy and SQLite failure-state
	652	+persistence, then exercise explicit metadata recovery,
650	653	exact-slot abandonment and a fresh host trial in new processes. Retained file
651		-archives are copied and hashed before unmount. This additional campaign is in
652		-qualification; a local skip is not positive ENOSPC evidence. Physical-media,
653		-EDQUOT, EIO, sync and power-loss acceptance remain separate.
	654	+archives are copied and hashed before unmount. CI executes both cases; a local
	655	+skip is not positive ENOSPC evidence. Failure before any image bytes are written
	656	+does not establish behavior for a write syscall that returns zero without an
	657	+error. Physical-media, EDQUOT, EIO, sync and power-loss acceptance remain separate.

tests/fixtures/public-safety-access/execution-lifetime-setup.tsmodified+3-2

Changes to tests/fixtures/public-safety-access/execution-lifetime-setup.ts

		@@ -1,4 +1,5 @@
1	1	import assert from 'node:assert/strict'
	2	+import type { FixturePhaseObserver } from './handover-setup'
2	3	import { setupHandoverFixture } from './handover-setup'
3	4	import { examplePublicSafetyConfig } from '../../../resources/functions/drone/public-safety/fixtures'
4	5
		⋯3 linesShow the 3 lines above this hunk@@ -8,8 +9,8 @@ export interface LifetimeSlot {
8	9	recording: string | null, completion: unknown, latestEvent: unknown,
9	10	candidate: unknown, pendingEvent: unknown, worker: Worker | null,
10	11	}
11		-export async function setupExecutionLifetime(mode: string) {
12		- const f = await setupHandoverFixture()
	12	+export async function setupExecutionLifetime(mode: string, observe?: FixturePhaseObserver) {
	13	+ const f = await setupHandoverFixture(observe)
13	14	const core = await import('../../../resources/functions/drone/public-safety/execution/service')
14	15	const { executionOwner } = await import('../../../resources/functions/drone/public-safety/execution/state')
15	16	const { executionWords } = await import('../../../resources/functions/drone/public-safety/execution/protocol')

tests/fixtures/public-safety-access/execution-lifetime.tsmodified+31-19

Changes to tests/fixtures/public-safety-access/execution-lifetime.ts

		⋯2 linesShow the 2 lines above this hunk@@ -3,21 +3,33 @@ import { setupExecutionLifetime } from './execution-lifetime-setup'
3	3
4	4	const mode = Bun.argv[2]
5	5	assert(mode === 'completed' || mode === 'canceled' || mode === 'quarantine')
6		-const h = await setupExecutionLifetime(mode)
	6	+const began = performance.now()
	7	+console.log(JSON.stringify({ oracle: 'lifetime-fixture-runtime', mode, bunVersion: Bun.version, executable: process.execPath, platform: process.platform, arch: process.arch }))
	8	+let observations = 0
	9	+async function observe<T>(phase: string, work: () => Promise<T>): Promise<T> {
	10	+ const emit = (outcome: 'began' | 'settled' | 'rejected') => {
	11	+ assert(observations < 128 && phase.length <= 64)
	12	+ console.log(JSON.stringify({ oracle: 'lifetime-phase', mode, ordinal: observations++, phase, outcome, elapsedMs: performance.now() - began }))
	13	+ }
	14	+ emit('began')
	15	+ try { const result = await work(); emit('settled'); return result }
	16	+ catch (error) { emit('rejected'); throw error }
	17	+}
	18	+const h = await observe('setup', () => setupExecutionLifetime(mode, observe))
7	19	let attachment: Awaited<ReturnType<typeof h.core.attachIncidentExecution>> | undefined
8	20	let expiredClose: string | null = null
9	21	try {
10		- const run = await h.start()
	22	+ const run = await observe('start', h.start)
11	23	const ack = h.captureFinished(run.slot, mode === 'quarantine')
12		- await h.sourceZero()
	24	+ await observe('source-zero', h.sourceZero)
13	25	if (mode === 'completed') {
14	26	let last = 0
15		- attachment = await h.f.as(h.f.senderToken, async () => h.core.attachIncidentExecution(h.core.captureExecutionContext(h.key), run.locator, value => {
	27	+ attachment = await observe('attach', () => h.f.as(h.f.senderToken, async () => h.core.attachIncidentExecution(h.core.captureExecutionContext(h.key), run.locator, value => {
16	28	if (!value.latestEvent || value.latestEvent.sequence === last) return
17	29	assert.equal(value.latestEvent.sequence, ++last)
18		- }))
19		- await attachment.activate()
20		- await h.wait(() => run.slot.phase === 'completed', 'The real eight-second producer must complete', 20000)
	30	+ })))
	31	+ await observe('activate', () => attachment!.activate())
	32	+ await observe('producer-completed', () => h.wait(() => run.slot.phase === 'completed', 'The real eight-second producer must complete', 20000))
21	33	assert(ack.event)
22	34	assert.equal(ack.event.data.outcome, 'completed')
23	35	assert.equal(ack.event.data.transferred, true)
		⋯3 linesShow the 3 lines above this hunk@@ -27,11 +39,11 @@ try {
27	39	attachment.release()
28	40	}
29	41	else {
30		- await h.wait(() => Atomics.load(run.slot.control, h.executionWords.emitted) === 1, 'The real producer must reach its first blocked callback')
	42	+ await observe('producer-first-callback', () => h.wait(() => Atomics.load(run.slot.control, h.executionWords.emitted) === 1, 'The real producer must reach its first blocked callback'))
31	43	assert.equal(Atomics.load(run.slot.control, h.executionWords.credit), 0)
32	44	assert.equal(Atomics.load(run.slot.control, h.executionWords.returned), 0)
33	45	const stoppingAt = performance.now()
34		- const stopping = await h.core.closeIncidentExecution(h.context, run.locator)
	46	+ const stopping = await observe('close', () => h.core.closeIncidentExecution(h.context, run.locator))
35	47	assert.equal(stopping.phase, 'stopping')
36	48	await h.wait(() => Boolean(ack.event), 'Capture a genuine finished ACK from the canceled actual Worker')
37	49	assert.equal(ack.event!.data.outcome, 'canceled')
		⋯17 linesShow the 17 lines above this hunk@@ -55,16 +67,16 @@ try {
55	67	const expiry = run.slot.metadataDeadline
56	68	const observedAt = Date.now(), observedMonotonic = performance.now()
57	69	assert(expiry > observedAt + 50000, 'The actual unshortened terminal TTL must still be mostly ahead')
58		- await h.sourceZero()
	70	+ await observe('source-zero', h.sourceZero)
59	71	console.log(JSON.stringify({ oracle: 'actual-retention-begins', mode, configuredMs: h.limits.retainedMs, remainingMs: expiry - observedAt, producer: run.slot.producer }))
60	72	// Scalar time/slot observation only, not repeated native authority polling.
61		- await h.untilWall(expiry - 1000)
	73	+ await observe('wait-before-expiry', () => h.untilWall(expiry - 1000))
62	74	assert.equal(h.executionOwner().slot, run.slot, 'Reservation remains during the actual retained lifetime')
63		- const retained = await h.core.readIncidentExecution(h.context, { scope: h.scope })
	75	+ const retained = await observe('retained-read', () => h.core.readIncidentExecution(h.context, { scope: h.scope }))
64	76	assert.equal(retained?.execution, run.view.execution)
65	77	assert.equal(retained.producer, mode === 'quarantine' ? 'unconfirmed' : 'finished')
66	78	assert.equal(run.slot.metadataDeadline, expiry, 'Reads do not renew retention')
67		- await h.untilWall(expiry + 100)
	79	+ await observe('wait-after-expiry', () => h.untilWall(expiry + 100))
68	80	if (mode === 'quarantine') {
69	81	assert.equal(h.executionOwner().slot, run.slot, 'Actual TTL cannot free an unconfirmed reservation')
70	82	try { assert.equal((await h.core.closeIncidentExecution(h.context, run.locator)).producer, 'unconfirmed'); expiredClose = 'unconfirmed-view' }
		⋯13 linesShow the 13 lines above this hunk@@ -84,20 +96,20 @@ try {
84	96	await h.wait(() => h.executionOwner().slot === null, 'The real retention timer must dispose the confirmed slot')
85	97	h.assertCleared(run.slot)
86	98	assert.equal(run.slot.worker, null)
87		- assert.equal(await h.core.readIncidentExecution(h.context, { scope: h.scope }), null)
	99	+ assert.equal(await observe('expired-read', () => h.core.readIncidentExecution(h.context, { scope: h.scope })), null)
88	100	if (mode === 'completed') {
89		- const next = await h.start()
	101	+ const next = await observe('next-start', h.start)
90	102	assert.notEqual(next.view.execution, run.view.execution)
91	103	assert.equal(next.view.phase, 'waiting-for-consumer')
92	104	await h.wait(() => Atomics.load(next.slot.control, h.executionWords.emitted) === 1, 'A new explicit start constructs a new real producer')
93		- await h.core.closeIncidentExecution(h.context, next.locator)
	105	+ await observe('next-close', () => h.core.closeIncidentExecution(h.context, next.locator))
94	106	await h.wait(() => next.slot.producer === 'finished', 'New explicit producer cooperatively finishes')
95		- await h.core.closeIncidentExecution(h.context, next.locator)
	107	+ await observe('next-release', () => h.core.closeIncidentExecution(h.context, next.locator))
96	108	assert.equal(h.executionOwner().slot, null)
97	109	}
98	110	}
99		- await h.sourceZero()
	111	+ await observe('source-zero', h.sourceZero)
100	112	console.log(JSON.stringify({ oracle: 'actual-retention-ended', mode, elapsedSinceObservationMs: performance.now() - observedMonotonic, expiryPassedMs: Date.now() - expiry, slotRetained: h.executionOwner().slot === run.slot, producer: run.slot.producer, sourceCount: 0, expiredClose, lateAckControlled: mode === 'quarantine', actualIndividualThreadJoinQualified: false }))
101	113	console.log(`PASS native execution lifetime ${mode}`)
102	114	}
103		-finally { attachment?.release(); await h.close() }
	115	+finally { attachment?.release(); await observe('cleanup', h.close) }

tests/fixtures/public-safety-access/handover-setup.tsmodified+18-9

Changes to tests/fixtures/public-safety-access/handover-setup.ts

		⋯1 lineShow the 1 line above this hunk@@ -2,8 +2,13 @@ import { randomUUID } from 'node:crypto'
2	2	import { setupIncidentDatabase } from './setup'
3	3	import { examplePublicSafetyConfig } from '../../../resources/functions/drone/public-safety/fixtures'
4	4	import { simulatePublicSafety } from '../../../resources/functions/drone/public-safety/runtime'
5		-export async function setupHandoverFixture() {
6		- const { database, User } = await setupIncidentDatabase()
	5	+export type FixturePhaseObserver = <T>(phase: string, work: () => Promise<T>) => Promise<T>
	6	+
	7	+export async function setupHandoverFixture(observe: FixturePhaseObserver = (phase, work) => {
	8	+ void phase
	9	+ return work()
	10	+}) {
	11	+ const { database, User } = await observe('fixture-database', () => setupIncidentDatabase())
7	12	const access = await import('../../../resources/functions/drone/public-safety/access/service')
8	13	const records = await import('../../../resources/functions/drone/public-safety/access/records')
9	14	const handover = await import('../../../resources/functions/drone/public-safety/access/handover')
		⋯10 linesShow the 10 lines above this hunk@@ -20,14 +25,18 @@ export async function setupHandoverFixture() {
20	25	for (const ability of incidentAbilities) Gate.define(`public-safety:${ability}`, (user, key) => publicSafetyIncidentGate(user, key, ability))
21	26	for (const ability of ['read', 'write'] as const) Gate.define(`public-safety:handover-${ability}`, (user, key) => policy.publicSafetyHandoverGate(user, key, ability))
22	27	const as = <T>(token: string, work: () => Promise<T>) => runWithRequest(enhanceRequest(new Request('https://incident.invalid/api/public-safety/incidents', { headers: { authorization: `Bearer ${token}` } }) as Parameters<typeof enhanceRequest>[0]), work)
23		- const sender = await User.create({ name: 'Outgoing Coordinator', email: 'handover-sender@example.invalid', password: 'temporary-fixture-password' })
24		- const recipient = await User.create({ name: 'Incoming Coordinator', email: 'handover-recipient@example.invalid', password: 'temporary-fixture-password' })
25		- const unrelated = await User.create({ name: 'Unrelated Observer', email: 'handover-unrelated@example.invalid', password: 'temporary-fixture-password' })
26		- for (const permission of ['public-safety:access', 'public-safety:create']) await Rbac.createPermission(permission, 'web')
27		- for (const user of [sender, recipient, unrelated]) await Rbac.givePermission(user.id, 'public-safety:access', 'web')
28		- for (const user of [sender, recipient]) await Rbac.givePermission(user.id, 'public-safety:create', 'web')
	28	+ const sender = await observe('fixture-sender-account', async () => User.create({ name: 'Outgoing Coordinator', email: 'handover-sender@example.invalid', password: 'temporary-fixture-password' }))
	29	+ const recipient = await observe('fixture-recipient-account', async () => User.create({ name: 'Incoming Coordinator', email: 'handover-recipient@example.invalid', password: 'temporary-fixture-password' }))
	30	+ const unrelated = await observe('fixture-unrelated-account', async () => User.create({ name: 'Unrelated Observer', email: 'handover-unrelated@example.invalid', password: 'temporary-fixture-password' }))
	31	+ await observe('fixture-rbac', async () => {
	32	+ for (const permission of ['public-safety:access', 'public-safety:create']) await Rbac.createPermission(permission, 'web')
	33	+ for (const user of [sender, recipient, unrelated]) await Rbac.givePermission(user.id, 'public-safety:access', 'web')
	34	+ for (const user of [sender, recipient]) await Rbac.givePermission(user.id, 'public-safety:create', 'web')
	35	+ })
29	36	const credential = async (userId: number, scopes = ['public-safety:read', 'public-safety:coordinate']) => (await tokens.createToken(userId, 'handover fixture', scopes, { withRefreshToken: false })).plainTextToken
30		- const senderToken = await credential(sender.id), recipientToken = await credential(recipient.id), unrelatedToken = await credential(unrelated.id)
	37	+ const senderToken = await observe('fixture-sender-token', () => credential(sender.id))
	38	+ const recipientToken = await observe('fixture-recipient-token', () => credential(recipient.id))
	39	+ const unrelatedToken = await observe('fixture-unrelated-token', () => credential(unrelated.id))
31	40	const seed = async (key: string, lifetime = 86400000) => as(senderToken, async () => {
32	41	const config = examplePublicSafetyConfig()
33	42	config.id = key

tests/fixtures/updates/host-volume-full.tsmodified+19-12

Changes to tests/fixtures/updates/host-volume-full.ts

		⋯50 linesShow the 50 lines above this hunk@@ -51,17 +51,20 @@ async function workerInput(path: string) {
51	51	const item = object(await Bun.file(path).json()), marker = object(await Bun.file(join(root, 'fixture.json')).json())
52	52	assert.equal(marker.kind, 'host-full-volume-v1'); assert.equal(item.nonce, marker.nonce)
53	53	assert.equal(item.namespace, namespace); identity(item.nonce, 32)
	54	+ assert.ok(item.imageWriteBudgetBytes === 0 || item.imageWriteBudgetBytes === reserveBytes)
	55	+ assert.equal(item.imageWriteBudgetBytes, marker.imageWriteBudgetBytes)
54	56	assert.equal(readFileSync(join(volume, 'nonce'), 'utf8'), item.nonce)
55	57	checkedVolume(volume)
56	58	const reference = object(item.reference)
57	59	assert.equal(reference.directory, join(volume, 'store')); assert.equal(reference.enrollmentDirectory, join(root, 'enrollment'))
58	60	assert.equal(item.project, join(root, 'project')); assert.equal(item.repository, join(root, 'repository'))
59		- return { root, volume, project: text(item.project), repository: text(item.repository), candidateId: identity(item.candidateId, 32), trialId: item.trialId === undefined ? undefined : identity(item.trialId, 32), reference: {
	61	+ return { root, volume, imageWriteBudgetBytes: item.imageWriteBudgetBytes, project: text(item.project), repository: text(item.repository), candidateId: identity(item.candidateId, 32), trialId: item.trialId === undefined ? undefined : identity(item.trialId, 32), reference: {
60	62	directory: text(reference.directory), enrollmentDirectory: text(reference.enrollmentDirectory), enrollmentSha256: identity(reference.enrollmentSha256, 64), storeSha256: identity(reference.storeSha256, 64),
61	63	} }
62	64	}
63	65
64		-function fillVolume(volume: string) {
	66	+function fillVolume(volume: string, imageWriteBudgetBytes: number) {
	67	+ assert.ok(imageWriteBudgetBytes === 0 || imageWriteBudgetBytes === reserveBytes)
65	68	const before = checkedVolume(volume), path = join(volume, 'filler.bin')
66	69	const fd = openSync(path, constants.O_WRONLY | constants.O_CREAT | constants.O_EXCL | constants.O_NOFOLLOW, 0o600)
67	70	let written = 0, full = false
		⋯4 linesShow the 4 lines above this hunk@@ -72,10 +75,11 @@ function fillVolume(volume: string) {
72	75	catch (error) { assert.ok(storageErrorCodes(error).includes('ENOSPC')); full = true; break }
73	76	}
74	77	assert.equal(full, true); assert.equal(checkedVolume(volume).freeBytes, 0)
75		- assert.ok(written > reserveBytes); ftruncateSync(fd, written - reserveBytes)
	78	+ assert.ok(written > reserveBytes)
	79	+ if (imageWriteBudgetBytes > 0) ftruncateSync(fd, written - imageWriteBudgetBytes)
76	80	const after = checkedVolume(volume)
77		- assert.equal(after.freeBytes, reserveBytes); assert.equal(after.device, before.device)
78		- return { before, after, written, fillerBytes: written - reserveBytes, fillError: 'ENOSPC' }
	81	+ assert.equal(after.freeBytes, imageWriteBudgetBytes); assert.equal(after.device, before.device)
	82	+ return { before, after, written, fillerBytes: written - imageWriteBudgetBytes, fillError: 'ENOSPC' }
79	83	}
80	84	finally { closeSync(fd) }
81	85	}
		⋯4 linesShow the 4 lines above this hunk@@ -86,7 +90,7 @@ async function limited(inputPath: string) {
86	90	try {
87	91	await trialRuntimeFirmwareCandidate({ ...input.reference, repository: input.repository, project: input.project, candidateId: input.candidateId, clock, async onProgress(event) {
88	92	assert.ok(phases.length < 512); phases.push(event.phase)
89		- if (event.phase === 'slot-selected') await writeFile(join(input.root, 'fill.json'), JSON.stringify(fillVolume(input.volume), null, 2))
	93	+ if (event.phase === 'slot-selected') await writeFile(join(input.root, 'fill.json'), JSON.stringify(fillVolume(input.volume, input.imageWriteBudgetBytes), null, 2))
90	94	} })
91	95	}
92	96	catch (error) { rejected = true; codes = storageErrorCodes(error) }
		⋯28 linesShow the 28 lines above this hunk@@ -121,13 +125,14 @@ async function archiveVolume(volume: string, output: string) {
121	125	}
122	126
123	127	/** Full bounded tmpfs, genuine ENOSPC/SQLite failure, and fresh public recovery. */
124		-export async function qualifyHostFullVolume(directory: string) {
	128	+export async function qualifyHostFullVolume(directory: string, imageWriteBudgetBytes: 0 | 1048576 = reserveBytes) {
125	129	const namespace = privateNamespace()
	130	+ assert.ok(imageWriteBudgetBytes === 0 || imageWriteBudgetBytes === reserveBytes)
126	131	directory = resolve(directory)
127	132	await mkdir(directory, { mode: 0o700 })
128	133	const volume = join(directory, 'volume'), nonce = randomBytes(16).toString('hex')
129	134	await mkdir(volume, { mode: 0o700 })
130		- await writeFile(join(directory, 'fixture.json'), JSON.stringify({ kind: 'host-full-volume-v1', nonce, namespace }), { mode: 0o600 })
	135	+ await writeFile(join(directory, 'fixture.json'), JSON.stringify({ kind: 'host-full-volume-v1', nonce, namespace, imageWriteBudgetBytes }), { mode: 0o600 })
131	136	async function command(argv: string[], label: string, timeoutMs = 10000) {
132	137	const result = await runBoundedHostProcess(argv, { cwd: directory, env: { ...process.env, BUN_RUNTIME_TRANSPILER_CACHE_PATH: '0' }, timeoutMs, stdoutBytes: 16384, stderrBytes: 16384, label })
133	138	assert.equal(result.exitCode, 0, result.stderr)
		⋯19 linesShow the 19 lines above this hunk@@ -153,7 +158,7 @@ export async function qualifyHostFullVolume(directory: string) {
153	158	const candidate = await reserveRuntimeFirmwareCandidate({ ...reference, repository: source, clock })
154	159	await signed.publish({ image, version: 3, custom: { releaseSequence: 9 }, rotateTargets: true })
155	160	const inputPath = join(directory, 'worker-input.json'), config = join(directory, 'worker.bunfig.toml')
156		- const input = { nonce, namespace, reference, project, repository: source, candidateId: candidate.candidate!.id }
	161	+ const input = { nonce, namespace, imageWriteBudgetBytes, reference, project, repository: source, candidateId: candidate.candidate!.id }
157	162	await writeFile(inputPath, JSON.stringify(input), { mode: 0o600 }); await writeFile(config, 'preload=[]\n', { mode: 0o600 })
158	163	async function worker(mode: 'limited' | 'recover' | 'abandon' | 'fresh') {
159	164	const result = await runBoundedHostProcess([process.execPath, '--no-env-file', `--config=${config}`, import.meta.path, mode, inputPath], { cwd: directory, env: { ...process.env, BUN_RUNTIME_TRANSPILER_CACHE_PATH: '0' }, timeoutMs: 60000, stdoutBytes: 16384, stderrBytes: 16384, label: `Full update filesystem ${mode}` })
		⋯17 linesShow the 17 lines above this hunk@@ -177,12 +182,14 @@ export async function qualifyHostFullVolume(directory: string) {
177	182	await writeFile(join(directory, 'failed-slots.json'), JSON.stringify(failed, null, 2))
178	183	assert.deepEqual(failed.knownGood, good.knownGood); assert.deepEqual(failed.installed, good.installed)
179	184	assert.equal(failed.trial!.phase, 'selected'); assert.equal(failed.trial!.health, null)
180		- assert.equal(failed.trial!.integrity, 'mismatch'); assert.equal(failed.store.attempt!.phase, 'pending')
	185	+ assert.equal(failed.trial!.integrity, imageWriteBudgetBytes === 0 ? 'unavailable' : 'mismatch')
	186	+ assert.equal(failed.store.attempt!.phase, 'pending')
181	187	assert.equal(failed.store.releaseFloor, 9)
182	188	const version = (role: string) => failed.store.metadata.find(row => row.role === role)!.version
183	189	assert.equal(version('root'), 2); assert.equal(version('targets'), 3)
184	190	const partialPath = join(reference.directory, 'host-slots', failed.trial!.id, 'image.bin'), partial = await readFile(partialPath)
185		- assert.ok(partial.length > 0 && partial.length < image.length); assert.deepEqual(partial, image.subarray(0, partial.length))
	191	+ assert.equal(partial.length, imageWriteBudgetBytes); assert.ok(partial.length < image.length)
	192	+ assert.deepEqual(partial, image.subarray(0, partial.length))
186	193	const previousImage = join(reference.directory, 'host-slots', good.knownGood!.id, 'image.bin')
187	194	assert.deepEqual(await readFile(previousImage), image)
188	195	await archiveVolume(volume, join(directory, 'failed-volume'))
		⋯19 linesShow the 19 lines above this hunk@@ -208,7 +215,7 @@ export async function qualifyHostFullVolume(directory: string) {
208	215	assert.notEqual(fresh.knownGood!.health!.sessionId, good.knownGood!.health!.sessionId)
209	216	assert.deepEqual(await readFile(partialPath), partial); assert.deepEqual(await readFile(previousImage), image)
210	217	assert.deepEqual(await readFile(join(reference.directory, 'host-slots', fresh.knownGood!.id, 'image.bin')), image)
211		- const result = { evidence: 'host-actual-enospc-v1', filesystemType: 'tmpfs', filesystemBytes: mounted.capacityBytes, freeBytesAfterFailure: fullVolume.freeBytes, sourceRevision, authorizedImageBytes: image.length, authorizedImageSha256: signed.imageHash, partialImageBytes: partial.length, partialImageSha256: digest(partial), errorCodes: rejection.errorCodes, interruptedPhase: failed.trial!.phase, interruptedAttempt: failed.store.attempt!.phase, acceptedRootVersion: version('root'), acceptedTargetsVersion: version('targets'), releaseFloor: fresh.store.releaseFloor, abandonedPhase: abandoned.trial!.phase, previousSlotId: good.knownGood!.id, failedSlotId: failed.trial!.id, confirmedSlotId: fresh.knownGood!.id, previousBootId: good.knownGood!.health!.bootId, freshBootId: fresh.knownGood!.health!.bootId, physicalDeploymentAllowed: fresh.physicalDeploymentAllowed }
	218	+ const result = { evidence: 'host-actual-enospc-v1', filesystemType: 'tmpfs', filesystemBytes: mounted.capacityBytes, imageWriteBudgetBytes, freeBytesAfterFailure: fullVolume.freeBytes, sourceRevision, authorizedImageBytes: image.length, authorizedImageSha256: signed.imageHash, partialImageBytes: partial.length, partialImageSha256: digest(partial), errorCodes: rejection.errorCodes, interruptedPhase: failed.trial!.phase, interruptedAttempt: failed.store.attempt!.phase, acceptedRootVersion: version('root'), acceptedTargetsVersion: version('targets'), releaseFloor: fresh.store.releaseFloor, abandonedPhase: abandoned.trial!.phase, previousSlotId: good.knownGood!.id, failedSlotId: failed.trial!.id, confirmedSlotId: fresh.knownGood!.id, previousBootId: good.knownGood!.health!.bootId, freshBootId: fresh.knownGood!.health!.bootId, physicalDeploymentAllowed: fresh.physicalDeploymentAllowed }
212	219	await writeFile(join(directory, 'qualification.json'), JSON.stringify(result, null, 2))
213	220	return result
214	221	}

tests/integration/runtime-host-storage.test.tsmodified+8-5

Changes to tests/integration/runtime-host-storage.test.ts

		@@ -1,21 +1,24 @@
1	1	import { expect, test } from 'bun:test'
2		-import { mkdtemp, rm } from 'node:fs/promises'
	2	+import { mkdir, mkdtemp, rm } from 'node:fs/promises'
3	3	import { tmpdir } from 'node:os'
4	4	import { join, resolve } from 'node:path'
5	5	import { qualifyHostFullVolume } from '../fixtures/updates/host-volume-full'
6	6
7	7	// Actual privileged Linux namespace qualification only. A skip is not ENOSPC evidence.
8		-test.skipIf(process.env.RUNTIME_HOST_ENOSPC_TEST !== '1')('a full host filesystem preserves accepted trust and requires explicit recovery before fresh confirmation', async () => {
	8	+for (const imageWriteBudgetBytes of [1048576, 0] as const) test.skipIf(process.env.RUNTIME_HOST_ENOSPC_TEST !== '1')(`a full host filesystem with ${imageWriteBudgetBytes} image bytes available requires explicit recovery before fresh confirmation`, async () => {
9	9	const evidence = process.env.RUNTIME_HOST_ENOSPC_EVIDENCE
10	10	const temporary = evidence ? undefined : await mkdtemp(join(tmpdir(), 'drone-host-enospc-'))
11		- const root = evidence ? resolve(evidence) : join(temporary!, 'campaign')
	11	+ if (evidence) await mkdir(resolve(evidence), { recursive: true, mode: 0o700 })
	12	+ const root = evidence ? join(resolve(evidence), imageWriteBudgetBytes === 0 ? 'zero-write' : 'short-write') : join(temporary!, 'campaign')
12	13	let completed = false
13	14	try {
14		- const result = await qualifyHostFullVolume(root)
	15	+ const result = await qualifyHostFullVolume(root, imageWriteBudgetBytes)
15	16	expect(result.filesystemType).toBe('tmpfs')
16	17	expect(result.filesystemBytes).toBe(16777216)
17	18	expect(result.freeBytesAfterFailure).toBe(0)
18		- expect(result.partialImageBytes).toBeGreaterThan(0)
	19	+ expect(result.imageWriteBudgetBytes).toBe(imageWriteBudgetBytes)
	20	+ expect(result.partialImageBytes).toBe(imageWriteBudgetBytes)
	21	+ if (imageWriteBudgetBytes === 0) expect(result.partialImageSha256).toBe('e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855')
19	22	expect(result.partialImageBytes).toBeLessThan(result.authorizedImageBytes)
20	23	expect(result.errorCodes).toContain('ENOSPC')
21	24	expect(result.errorCodes).toContain('SQLITE_FULL')

tests/unit/public-safety-execution-lifetime.test.tsmodified+37-7

Changes to tests/unit/public-safety-execution-lifetime.test.ts

		@@ -1,23 +1,53 @@
1	1	import { expect, test } from 'bun:test'
2		-import { mkdtemp, rm, writeFile } from 'node:fs/promises'
	2	+import { mkdir, mkdtemp, rm, writeFile } from 'node:fs/promises'
3	3	import { tmpdir } from 'node:os'
4		-import { join } from 'node:path'
	4	+import { join, resolve } from 'node:path'
5	5
6	6	for (const mode of ['completed', 'canceled', 'quarantine']) test(`native execution lifetime ${mode}`, async () => {
7		- const root = await mkdtemp(join(tmpdir(), 'drone-incident-access-execution-lifetime-'))
	7	+ const evidence = process.env.EXECUTION_LIFETIME_EVIDENCE
	8	+ if (evidence) await mkdir(resolve(evidence), { recursive: true, mode: 0o700 })
	9	+ const root = await mkdtemp(join(evidence ? resolve(evidence) : tmpdir(), 'drone-incident-access-execution-lifetime-'))
8	10	let child: ReturnType<typeof Bun.spawn> | undefined
	11	+ let settledExitCode: number | null = null
	12	+ const began = performance.now()
	13	+ let deadlineFired = false, cleanupKill = false, completed = false
	14	+ let stdoutWritten = false, stderrWritten = false
	15	+ const retain = async (artifact: 'fixture.stdout' | 'fixture.stderr' | 'lifecycle.json', value: string) => {
	16	+ try { await writeFile(join(root, artifact), value, { mode: 0o600, flag: 'wx' }); return true }
	17	+ catch { console.error('LIFETIME_FIXTURE_EVIDENCE_UNAVAILABLE', JSON.stringify({ mode, artifact })); return false }
	18	+ }
9	19	try {
10	20	const config = join(root, 'bunfig.toml'); await writeFile(config, 'preload=[]\n[install]\nauto="disable"\n')
11	21	const running = child = Bun.spawn([process.execPath, '--no-install', `--config=${config}`, '--no-env-file', `${import.meta.dir}/../fixtures/public-safety-access/execution-lifetime.ts`, mode], { cwd: root, env: { APP_ENV: 'test', APP_DEBUG: 'false', NODE_ENV: 'test', DB_CONNECTION: 'sqlite', DB_DATABASE_PATH: join(root, 'access.sqlite') }, stdin: 'ignore', stdout: 'pipe', stderr: 'pipe' })
12		- const timeout = setTimeout(() => { if (running.exitCode === null) running.kill('SIGKILL') }, 90000)
	22	+ const exited = running.exited.then(code => { settledExitCode = code; return code })
	23	+ const timeout = setTimeout(() => {
	24	+ if (settledExitCode !== null) return
	25	+ deadlineFired = true
	26	+ running.kill('SIGKILL')
	27	+ console.error('LIFETIME_FIXTURE_DEADLINE', JSON.stringify({ mode, pid: running.pid, timeoutMs: 90000, elapsedMs: performance.now() - began, requestedSignal: 'SIGKILL' }))
	28	+ }, 90000)
13	29	try {
14		- const [code, stdout, stderr] = await Promise.all([running.exited, new Response(running.stdout).text(), new Response(running.stderr).text()])
	30	+ console.log(JSON.stringify({ oracle: 'lifetime-fixture-start', mode, pid: running.pid, timeoutMs: 90000, parentBunVersion: Bun.version, childExecutable: process.execPath }))
	31	+ const [code, stdout, stderr] = await Promise.all([exited, new Response(running.stdout).text(), new Response(running.stderr).text()])
15	32	// eslint-disable-next-line no-console
16	33	for (const line of stdout.split('\n')) if (line.startsWith('{"oracle":')) console.log(line)
	34	+ stdoutWritten = await retain('fixture.stdout', stdout)
	35	+ stderrWritten = await retain('fixture.stderr', stderr)
17	36	expect(code, `${stdout}\n${stderr}`).toBe(0)
18	37	expect(stdout).toContain(`PASS native execution lifetime ${mode}`)
	38	+ completed = true
	39	+ }
	40	+ finally { clearTimeout(timeout); if (settledExitCode === null) { cleanupKill = true; running.kill('SIGKILL') }; settledExitCode = await exited }
	41	+ }
	42	+ finally {
	43	+ if (child && settledExitCode === null) {
	44	+ if (child.exitCode === null && !child.signalCode) { cleanupKill = true; child.kill('SIGKILL') }
	45	+ settledExitCode = await child.exited
19	46	}
20		- finally { if (running.exitCode === null) running.kill('SIGKILL'); await running.exited; clearTimeout(timeout) }
	47	+ const lifecycle = { mode, pid: child?.pid ?? null, exitCode: settledExitCode, exitProperty: child?.exitCode ?? null, signal: child?.signalCode ?? null, timeoutMs: 90000, deadlineFired, cleanupKill, elapsedMs: performance.now() - began, stdoutWritten, stderrWritten, completed }
	48	+ console.log(JSON.stringify({ oracle: 'lifetime-fixture-exit', ...lifecycle }))
	49	+ await retain('lifecycle.json', JSON.stringify(lifecycle, null, 2))
	50	+ if (completed) await rm(root, { recursive: true, force: true })
	51	+ else console.error('LIFETIME_FIXTURE_RETAINED', root)
21	52	}
22		- finally { if (child?.exitCode === null) { child.kill('SIGKILL'); await child.exited }; await rm(root, { recursive: true, force: true }) }
23	53	}, 100000)