ReviewOS

stacks/drone

Execution lifetime: diagnose Mac full-suite authority rejection and child timeout

#60
Open chris opened this 9 days ago ·
9 days ago

The retained Mac full Buddy run at 1e4622898c863d8ee57b5010c467bfc0c5606f0e reported two execution-lifetime failures before later explicit ENOSPC errors. The run subsequently stopped without a terminal receipt. Do not report it as a full pass or infer a single cause from the later disk exhaustion.

Observed with PATH Bun 1.4.3 on macOS arm64:

  • native execution lifetime completed: exit 1 after 83,106 ms. The child reached actual unshortened 60,000 ms retention with 59,985 ms remaining, then native authority checking rejected with PublicSafetyAccessError: Incident is unavailable at execution/policy.ts:83.
  • native execution lifetime canceled: outer 100,000 ms test timeout, followed by an unhandled assertion reporting child exit 137. The original child watchdog is 90,000 ms. Original observation did not retain its fired flag or actual signal, so the cause of 137 is unestablished.
  • quarantine passed with the actual stop grace and retained quarantine oracle.
  • Later cases reported actual ENOSPC and failed SQLite work. The original log is 180,223 bytes, SHA-256 6f6c45a60fcfa501835919dbf0f47490ff68f25490ba4425e3bf8d05d0f5caf6; terminal exit code is unavailable.

Focused unchanged cases after space became available:

  • bun test tests/unit/public-safety-execution-lifetime.test.ts --test-name-pattern 'native execution lifetime completed': passes in 66.07 s, 60,204 ms observed actual retention. Log SHA-256 62e7dfec7933a6b035b5d9bbad2768306725261ca113dfb3880d66ea13360c6d.
  • The corresponding canceled case passes in 63.00 s, 60,104 ms observed actual retention. Log SHA-256 fd4a62c95cbb4758d0262616773fdc92c16b66170855c095d9d38f2aa9c12cb7.
  • Exact-head Linux CI 39737879757 passed all five jobs, including the full Buddy suite (1,498 passed, 0 failed across 193 files).

The lifetime test and fixtures are unchanged by the maintenance continuation commit. These passing observations do not establish the original cause or repair it. Private logs and receipts are retained under campaigns 364, 366, 368 and 369.

Acceptance:

  • Reproduce the full-suite lifetime failure at the actual native authority/Worker/retention seam.
  • Retain bounded phase timing, real child signal and watchdog-fired state, and failed fixture evidence before cleanup.
  • Determine the cause with actual operand observations; distinguish native authority deadline rejection, child guard, process exit, event-loop delay and storage failures.
  • Fix only an established cause, keeping production authority, 60-second retention, stop grace and existing original test guards intact unless separately justified.
  • Qualify focused and full-suite behavior, preserving the original failed observations separately.

Pinned local diagnostic observation, October 22

The diagnostic worktree at 16458e50480008dd0fa4e5d13710ac85bd303029 now records parent and child runtime identity independently. This commit is still local, above the draft PR head, and is not published to main.

A fresh macOS ARM64 normal-lifetime run used /Users/chris/.bun/bin/bun 1.4.2. Parent and child both reported 1.4.2, and the executable SHA-256 remained 35d20dd0263e5c950194434b925454fdfa9ba6e4467da960410fa05b08a7a5b5 before and after the run. Actual result: 2 passed, 1 failed, 240.55 seconds, process exit 1. This is not a green normal-lifetime qualification.

The completed-mode child reported setup settled at 30,460.61 ms, activation settled at 30,574.97 ms, then entered the real 60-second retention wait at 49,267.22 ms. The unchanged 90-second parent watchdog fired while that wait was outstanding. The child exited with settled status 137, raw exit property null, SIGKILL, deadlineFired: true, cleanupKill: false; its failed fixture and output were retained. Canceled and quarantine modes passed separately.

These observations locate this run's guard exhaustion during retention after setup and producer work consumed about 49 seconds. They do not explain why setup/producer work was slow, establish the cause of the original native failure, or justify increasing a guard. The prior mixed-runtime failure and controlled diagnostic signal campaigns remain separate evidence. Original failures are retained.

9 days ago

The bounded lifetime diagnostics and evidence-write cleanup correction are now on main in 06547e6f292ed7e85571d6ab94e5fc694f3bf999 (PR #110). They record actual wall/monotonic/expiry operands, native settled exit and signal, parent-watchdog firing and cleanup-kill state. The production authority, real 60-second retention, stop grace and original guards/assertions are unchanged.

A real evidence-write failure reproduced a separate forensic cleanup bug: the old parent deleted a genuinely successful child's database despite failing to write its stdout evidence. The corrected cleanup retains the database and surviving evidence unless the child succeeds and all three evidence writes succeed. Original red/green logs and the failed-write fixture remain retained.

The final Mac full suite passes all three actual unshortened lifetime modes and reports 2,187 passing tests, 28 platform skips and zero failures overall. Exact-head CI passes all five jobs, including its complete Linux suite. Those passes do not establish the cause of the original Mac failures or repair the unexplained near-expiry null read. This issue remains open.

One remaining diagnostics transport gap is explicit: CI does not yet set EXECUTION_LIFETIME_EVIDENCE and upload retained failures before the ephemeral runner disappears. The intended upload is limited to fixture stdout/stderr, lifecycle JSON and access.sqlite plus its WAL/SHM companions. Do not upload the entire retained root: it contains dependency/default symlinks. A pending snippet is prepared but is not part of main yet.

Sign in to comment on this issue.