Execution browser: diagnose intermittent nominal Worker completion failure

Observed failure

The execution-browser job for enrollment candidate 416073e88b5649e74b1e98edc6368fc2bcfdfa11 failed Actual Worker recording must complete in run 39644754350. The unchanged browser scenario reached real nominal100 completion after real live progress, GET renewal, cooperative stop and finished-result discard. The completion DOM did not satisfy the original 40-second guard. The job's browser fixture reports 1 pass, 1 fail and 5 assertions in 69.11 seconds. That total includes setup and earlier steps; it is not a measured Worker execution duration.

The bounded failure report was only execution-browser-failure-diagnostics with available:false. It does not establish whether the second POST entered/settled, which DOM phase remained, or whether the Worker was still running, had failed, or had completed without rendering. The root cause is unknown. The failure occurs before source admission; it is distinct from the later admission symptom in #34.

Retained evidence

• Completed failed job's API log: 82,465 bytes, SHA256 440e8004fcfa9e5825ff0c91829da9e5b5f85835ca2051d4040868497bcd8b4e.
• Original CI artifact, 195,280-byte ZIP, SHA256 194858cdb081c88c074e59e8b93bda36aa27fc2b944dd83d1bea6025fd95e6a0. Download matches the logged digest.
• Compiled HTML and execution-live-desktop.png retained. The image was taken during the earlier live run, not at the later failure, and does not establish the failing DOM state.
• The full workflow is still running at this initial checkpoint. A failed completed job is not treated as a terminal full-suite result.

Acceptance

• Retain a red-capable loop for the actual nominal-completion scenario and every observed verdict, with Linux/Bun1.4.3 versus other-platform results distinguished.
• Preserve bounded nonsecret partial diagnostics when one server/DOM collection arm is unavailable; record known phase and separate second-start request/settlement/DOM/producer observations without bodies, credentials, account IDs or recordings.
• Reproduce and establish a causal prediction, then fix the demonstrated cause with a meaningful regression. Non-reproduction is not a fix.
• Qualify the original native Auth/SQLite/genuine-Worker browser scenario, preserving the 40-second completion guard, authority checks, explicit source admission and all original assertions. Record later attempts separately from the first failure.

Related to #23, #34 and qualification of #43. No product fix or acceptance-limit change is claimed. A local unchanged native browser observation is being collected separately; it cannot retroactively explain this CI failure.

Diagnostic-only correction and separate local observations

Commit 981e17c2cdd5beade7ea8619d2e0b5975d9b1801, in #43, corrects a reproduced reporting defect: the old all-settled race discarded an already available observation when its peer remained pending. The preserved RED failed that exact expectation. The helper now retains each settled arm and pending/rejected status independently, with the unchanged one-second budget. The driver retains phase, baseline and nonsecret counts, including execution POST count, within the existing 8,192-byte ceiling. Cleanup still aborts the owned fetch and closes the CDP socket. No original execution assertion, completion/admission guard or product control path changed.

Five diagnostic regression tests pass with 13 assertions. Combined enrollment/staging/diagnostic regression has 34 passes, zero failures and 220 assertions. App and runtime types and Pickier pass; independent Spec and Standards reviews have zero actionable findings.

Separate actual macOS arm64/Bun1.4.2 native browser observations: unchanged 416073e passed 2 tests/6 assertions in 32.90 seconds; the reporting change passed 2 tests/6 assertions in 31.99 seconds. Both preserve native Auth, SQLite, genuine Worker, the original 40-second completion guard and explicit admission. These are non-reproductions, not a causal explanation of the original Linux/Bun1.4.3 failure. New exact-head CI is pending. All original-cause and original-platform acceptance criteria remain open.

The execution-browser job for corrected diagnostic head 981e17c2cdd5beade7ea8619d2e0b5975d9b1801 has now passed on Linux/Bun1.4.3 in CI 39647836271. The full workflow is still running. This is a separate original-platform non-reproduction with the original assertions and guards, not a causal explanation or a Worker fix. The original failed job/artifact and all acceptance criteria remain retained and open. The later SDK prerequisite in #45 does not change the execution path or resolve this issue.

Original enrollment workflow terminal result

The original workflow 39644754350 for 416073e88b5649e74b1e98edc6368fc2bcfdfa11 is now terminal with overall failure. The execution-browser job failed as recorded above; the other four jobs succeeded. The native check reported 1,406 passes, 39 skips, zero failures and 488,354 assertions. That passing check does not override the failed browser job.

The full terminal workflow log was collected once and retained separately: 3,121,606 bytes, SHA256 8101aa2c0adc280a6e4c6f136afd5a134e5fcd87b5387baa32614bd5c36dfce2. The original failed-job log and artifact are also retained. Later head 981e17c2cdd5beade7ea8619d2e0b5975d9b1801 has a separate passing Linux browser observation and a still-running full workflow at this checkpoint. No original-cause fix or broader completion is claimed.

Qualified enrollment and diagnostics on main

CI 39647836271 for 981e17c2cdd5beade7ea8619d2e0b5975d9b1801 succeeded across all five jobs, including the native suite with 1,411 passes, 39 skips, zero failures and 488,367 assertions across 177 files. The exact commit is now on origin/main after a normal fast-forward push as Chris with no coauthor trailers; #43 is merged. The terminal log was collected once and retained (3,001,888 bytes, SHA256 45f92c23dcad57dc141dc1442033ef245ba23786b18011bf34210c0ef615cf14).

This publishes initial immutable trust enrollment and the test-only partial-diagnostic correction. The passing original-platform browser observation does not explain or fix the original intermittent Worker failure. Protected metadata/release floors, recovery, key/clock lifecycle, slots and boot/session health remain open. No broader acceptance checkbox is completed. The SDK prerequisite #45 now has a separately reviewed correction 32330728f853f9013a1845fdb15ea8bde15c4319 preserving operational causes and closing the Symbol/coercion cache-preference bypass; its new-head full CI is pending.

Additional main-run recurrence, October 22

The separate main workflow 39730098318, on 6d3a9ad4cf8f0b5f25c104cf121a4fb12f191fab, is terminal with failure in the execution-browser job. The other four jobs, including the full native suite and purge namespace case, succeeded. The original exact-head feature workflow had passed all five jobs; that is a separate observation.

The unchanged 40-second nominal-completion oracle failed again. Retained bounded diagnostics now establish: phase real nominal100 completion, start baseline 1, two execution POST requests, no admission requests, no browser errors, workspace current, DOM execution phase running with an execution ID, no completion or execution-error DOM, and Start disabled. The DOM arm fulfilled while the server observation remained pending within the one-second diagnostic budget. That does not establish Worker liveness, whether the second POST settled, or why the server observation was pending. The cause remains unknown.

The terminal log was collected once: 3,086,990 bytes, SHA-256 0c42eb97bbda245284c83d1f86ef1dfedb4a45f2e2018d01a13c31d8d9dfc1c6. The artifact was retained once: 196,018-byte ZIP, SHA-256 a846920bc18fd8e6d4f8bef6def404424420f97445d23ca5fb1e78cff8637e91, containing compiled HTML and the earlier live-run screenshot. That screenshot is not a failing-state capture. No assertion, guard or production control path was changed, and no rerun was started for this recurrence. All original-cause acceptance remains open.

Additional exact-head recurrence

CI 39810221070 execution browser job on 6ded83267d9bb9d0fe87c900f405314ec08a6119 again fails at real nominal100 completion: Actual Worker recording must complete. The test exits 1 after 57,964.48 ms. Bounded diagnostics retain workspace current, execution running, an execution ID, no completion/source/error, start disabled, zero browser errors, two execution starts and zero admissions. The server arm is pending/unavailable while DOM observation succeeds. This is the same observed symptom and still does not establish Worker liveness, POST settlement or the reason the server observer cannot answer.

Original job log retained once: 82,970 bytes, SHA-256 77c2e5d81fc7b78c62bbdeec74892874a430c1ab9c593b6567b5e4944563bccb. Artifact 12440341288 is downloaded and retained independently. The workspace job in this workflow has a separate unlabeled timeout; no common cause is established. No causal fix or rerun was substituted for this failed result.

Additional exact-head recurrence with both diagnostic arms

The completed execution-browser job in CI 39838472116, at 6bdf28b544fb821f9cab54088c7178196b3674c1, failed the same unchanged 40-second nominal-completion oracle. The case reports 54,564.52 ms total, including setup and earlier phases.

This time both bounded diagnostic arms fulfilled. The server's second start settled with HTTP 202 in 8.94 ms. At diagnostic capture the slot and DOM were completed, the producer was finished, finishedAck and recording/completion were present, native credit/emitted sequence was 2716, and simulated event time was 54,260 ms. No check or subscriber was pending then; two starts and zero admissions were observed, with no browser error. The slot's elapsedMs was 40,794 at that later capture. This is not a completion timestamp and does not prove whether completion/rendering happened before or after the original guard. It narrows the recurrence without establishing its cause or authorizing a deadline change.

The completed-job log is retained once: 85,326 bytes, SHA-256 0446adf45d8d41c76203a4b967ba784b9cf70af3f4ecfc868cd3efb4055df8ed. Artifact12452366867 is retained once, 196,133 bytes, SHA-256 f3aa283aeeb0d1f8e5e976cd18a71be2e85d7ada67391509b4d99cf06ea7b9eb, matching its logged digest. Its screenshot is the earlier live phase, not a failing-state capture. The full workflow remains in progress; no rerun has been started. The separate workspace startup failure has no established shared cause.

A new local investigation on 8477de907fc6f67b2682dc7d3ae2647486266962 demonstrates one reason a server diagnostic can remain pending, without diagnosing the original completion timeout. The actual parent result path synchronously digests, decodes, parses, performs the unchanged deterministic replay validator and checks canonical equality before completion becomes eligible. No validation or authority check was removed.

One fresh pure macOS arm64/Bun1.4.2 process measured that parent verification span at 1,452.93 ms for the actual nominal100 recording: 7,327,865 wire bytes, 2,716 events, 2,684 epochs and 32 captures. The unchanged full canonical/replay validation accounted for 1,212.29 ms and the subsequent canonical equality for 219.41 ms. This measurement excludes Auth, SQL, native streams and browser transport.

A separate fresh process tested a causal prediction with real HTTP. An independent health GET was sent after that same synchronous verification started. It timed out at the original one-second diagnostic bound, after 1,001.69 ms. Verification then returned HTTP 200 after 1,504.68 ms of synchronous work; a new health GET returned HTTP 200 in 3.25 ms. Both observations and the before-measurement predictions are retained. The exact owned server child was terminated during cleanup; no browser/CI rerun or application edit was made.

This establishes that this actual verification workload can delay HTTP beyond the diagnostic window on this local host. It does not establish that it caused a historical Linux 40-second completion failure, prove a Worker hang, or justify increasing any guard. Original-platform native parent-stage evidence is still needed before choosing a causal fix. The original acceptance criteria remain open. Separately, draft #85 corrects an atomic completed-DOM read gap and qualifies pre-admission authorization cancellation; its exact combined Linux workflow has passed both browser jobs but is still running the full suite at this update.

Draft #86, at c1b1569e1bf74a3cfe9c4d1b3b3687ee8026f9ce, adds bounded observation of the actual core-created Worker's synchronous callback. It preserves the genuine constructor, raw frames, callback receiver/argument/return/throw, all authority checks and the original completion/admission/stream guards. Five real-Worker contract tests pass with 58 assertions. A retained deterministic RED exposed marker latency being included in timing; the corrected timer brackets only the original callback, excluding marker and bookkeeping work, and limits marker output to four per Worker. Both independent source reviews are clean.

The actual Linux/Bun1.4.3 execution-browser job now passes on that exact source. Its digest-verified success artifact records 2,716 nominal event callbacks, one result callback taking 3,107.70 ms, and one finished callback taking 0.10 ms. Event callback synchronous work totals 163.02 ms; this excludes their asynchronous SQL/credit/delivery work. The completed-read and admission-renewal receipts also confirm private concealment/restoration, zero POSTs for the canceled decision and one POST from the fresh explicit decision. Chrome exits only after owned cleanup.

This is original-platform evidence that the actual parent result callback can synchronously occupy more than the one-second diagnostic window in a passing native case. It still does not establish the cause of a historical 40-second failure, measure async authorization duration or justify increasing guards. The new workflow's separate workspace-browser job failed Fresh browser must start: 400 DevTools probes returned ENOENT, no port was discovered and the owned Chrome child was terminated only during cleanup. That failure is retained under #72; the full workflow is still running and this draft is not qualified for main. No rerun, assertion weakening, production change or issue closure is claimed.