ReviewOS

stacks/drone

Workspace browser: diagnose Linux startup without a DevTools port

#72
Open chris opened this 8 days ago ·
8 days ago

Observed failure

Exact source 35b9bd8a8b21f7ca59031f30b111b08a09207960, CI 39856555655 workspace-browser job, Linux x64 and Bun 1.4.3, failed with Fresh browser must start. Native account setup and actual STX compilation completed before browser discovery failed. The initial page was never visited.

The retained launch observation reports:

  • phase: startup-unavailable, elapsed 20,103.90 ms.
  • 400 DevToolsActivePort probes, zero successful reads, 400 ENOENT failures, no discovered port.
  • No exit observed before cleanup. Cleanup requested SIGTERM; exit 143 / SIGTERM was then observed at 20,103.84 ms. This does not show an unsolicited browser exit.
  • Startup-only stderr is 191 bytes and contains one D-Bus address-parsing warning. That warning alone does not establish causality.
  • The exact Chrome version/hash, live startup milestones, descendant states and cgroup resource counters were not captured, so they cannot be reconstructed from this artifact.

Retained artifact 12459084082: ZIP 104,033 bytes, SHA-256 10798d63d4cdc7b5cbd8aadcc4164981a1175d0889ee61bba358f6c19a58d103. Contents: launch JSON, startup-only stderr, compiled HTML. Completed job raw log was collected once: 82,592 bytes, SHA-256 f86c5378242798c8cc787a4d28ab52b86b2b2252be5ed86e97e91cf3c33f8e1e.

The rest of this workflow and the final local full suite were still running when the job was retained. Do not restart the original failure or treat another source's passing workflow as its explanation.

Acceptance

  • Retain the original completed job log, fixed-shape launch observation and startup-only stderr, with exact head and artifact hashes.
  • Establish the startup cause with a bounded native Linux campaign using an identified Chrome binary/version, the same launch arguments and a fresh profile. Preserve failed attempts and controls.
  • Capture enough startup/process/resource evidence to distinguish a live stalled Chrome from a launcher/descendant failure or a missing/misplaced discovery file. Keep fixture credentials and post-page stderr private.
  • Test any proposed correction without increasing the existing 400-probe/50-ms discovery budget or bypassing browser qualification; retain exact-head CI evidence before main publication.

Relationship to existing work

Issue #64 records macOS Chrome exits during later browser activity. This Linux run instead has no discovered port and a cleanup-initiated termination. A common cause is unproven. The reporting added in #70 made this distinction observable.

Draft #71 remains unqualified for main publication while this failure and its final integrated qualification remain unresolved. Original #44/#60/#63/#64/#67/#68 causes remain open. No evidence here establishes a TCP-driver failure, successful application/browser behavior, firmware SITL or hardware acceptance.

Fixed native control campaign

Research campaign 39862439611, source 93d071aafd4bf4446d27d729b57f4fa77cae58fb, completed all 24 fixed native launch controls. All discovered a port within 100.76 to 2,060.42 ms using the original Chrome arguments and 400-read/50 ms guard. Every owned process group settled. Native Buddy/Pickier and dedicated TypeScript checks passed.

The actual runner image is ubuntu24 / 20261012.334.1, matching the original recorded image. The observed browser is Google Chrome 155.0.8059.39; the 297,798,984-byte backing binary hashes to 9bfb381296dffe75f3419d07b6cc64525105ed117d1453a4ef3b291cb97d2b58. The original failure did not retain its Chrome version/hash, so binary equivalence is unproven.

All 24 successful trials also emitted the D-Bus address warning. That warning by itself is therefore insufficient to explain the original missing discovery file. The first successful trial's root was observed in disk sleep after discovery; this is not evidence of the original startup cause.

This control removes app bootstrap and uses a detached Chrome group for ownership. It imports no Drone application or page and is not product qualification or reproduction of the original failure. The next campaign restores actual native account/SQLite/STX/server context and the original inherited Chrome process-group structure, with opt-in process observations after discovery and before any page visit. Its startup-only mode and manual research workflow stay off main.

Raw campaign log collected once: 35,550 bytes, SHA-256 d7126a6f5e52d61efd5331eab0168dbdadf6a49359f0c2681672b446a60a6bde. Artifact 12461852896, downloaded once: ZIP 85,776 bytes, SHA-256 e0a7345c89670fb076a8f23c670cc56a073b08bb880ff2027d52a30eafc901d3, 73 allowlisted files. The original failed artifacts are unchanged.

The original product CI has since completed with the separate two passive TCP test failures tracked in #73. Its local full suite passed, but PR #71 remains unqualified. All remaining acceptance items above stay open.

Restored workspace startup control campaign, 22 October 2026

Research-only run 39865101093, source 9c66ab0d44fae7d73223e244450164be0993861e, restored the actual Auth/SQLite/STX/server startup context, separate Bun browser driver and inherited Chrome process group. All 12 fixed trials discovered the port within the original 400 x 50 ms guard, completed the startup-only control and confirmed owned-group cleanup. No page was visited. Native Buddy lint and the dedicated types passed. Chrome 155.0.8059.39, Bun 1.4.3 and Ubuntu image 20261012.334.1 were recorded, with unchanged executable hashes across the campaign.

This clean control does not reproduce the original startup failure or qualify the full browser scenario. Exact binary equivalence to the original failed job remains unproven. The research workflow and startup-only mode will not be merged into main. Original failed run and evidence remain intact. Full raw log and allowlisted artifacts collected once in local evidence workspace-startup-context-campaign-488. Cause remains unresolved; keep this issue open.

Successful product control after the TCP fixture correction

Product CI 39870834185 at b18e1d63d344452cf45719cf3b4072a752b71bce passed both rendered browser jobs and all other jobs. The commit is published on main. This is a successful product control, not reproduction or an explanation of the original missing-DevTools-port startup. No startup-only research mode or speculative startup fix entered main. Keep #72 open until the original cause has evidence and an appropriate regression.

Recurrence in post-main execution-browser qualification

Source 137ed1b, post-main run 40067866417, execution-browser job, again failed before a browser page was visited. The exact-head pre-publication full workflow 40062707703 passed this browser job and all other jobs. That passing control does not explain the new failure.

The original startup artifact reports startup-unavailable after 22,086.61 ms, 400 probes, zero successful reads and 400 ENOENT failures. The root Chrome PID is 2194. No exit was observed before cleanup or within the retained cleanup observation; cleanup was requested, but settled exit and signal are null. This record does not establish owned-group cleanup or an unsolicited Chrome exit.

Startup-only stderr is 573 bytes and contains three D-Bus address-parsing warnings. Earlier successful controls emitted that warning too, so it remains insufficient causal evidence. The exact Chrome version/hash and live startup milestones are still absent from this production observation; they cannot be inferred from an earlier runner or browser campaign.

Artifact 12549693717 was downloaded once: ZIP 104,038 bytes, SHA-256 7baaa4ebdfff500823d7c480e2bb1f6c924313dab3b4dc859d8190697d8fecef. It contains launch JSON, startup-only stderr and compiled HTML. The original ZIP is retained locally under owned-vtol-back-planning-main-ci-580. After the workflow became terminal, its original logs were downloaded once and retained: ZIP 951,986 bytes, SHA-256 54b6aa02b894d9e0cd888b40f145e4ff37c1c10ba1ce0ff225bf3d362d9b4658. Analysis read the original archive, without a network refetch or a success-only verifier.

The whole run finished failure. The native check, workspace browser, firmware and purge-namespace jobs passed. The native suite recorded 1,982 passed, 41 skipped, zero failed and 502,616 assertions across 225 files in 3,062.59 seconds. The execution-browser startup failure remains the failing job; native success does not explain or clear it. No startup deadline, browser flag, sandbox, test assertion or qualification gate has been changed or bypassed. The remaining cause/process-observation/correction acceptance items above stay open.

Recurrence during scoped-session qualification

Source 5bf7ca9, PR #98 run 40078358290, execution-browser job, failed before any page visit. The original job artifact reports startup-unavailable after 22,109.01 ms: 400 probes, zero successful reads, 400 ENOENT failures and no discovered port. Root PID 2388 had no observed exit before cleanup; cleanup was requested, but settled exit, signal and exit elapsed remain null. This does not prove a live root process, spontaneous exit or complete owned-group cleanup.

The startup-only stderr retains all 955 bytes without truncation and contains five D-Bus address-parsing warnings. Previous successful controls emitted this warning too. Exact Chrome version/hash, live startup/process milestones and cgroup resource counters remain absent, so the cause cannot be established from this artifact.

Artifact 12554187046 was downloaded once: ZIP 104,054 bytes, SHA-256 f291dd91adc3c4c3312227c4def02110ba68511ddb166aba283893d936f7d0a3. It contains browser-launch.json, chrome-startup-stderr.log and compiled.html; the original archive and per-entry hashes are retained under local owned-cargo-fleet-ci-586 evidence. The whole workflow has now finished failure. The native check, workspace browser, firmware and purge-namespace jobs passed; execution-browser startup remained the failing job. The native suite recorded 2,090 passed, 41 skipped, zero failed and 506,826 assertions across 235 files in 4,673.76 seconds, and all 31 copied cargo cases passed. The original terminal workflow ZIP was downloaded once and retained: 920,293 bytes, SHA-256 052b5407e1beee2baac2ba469a5b8542b6d277f7aefd1d653b2f439ed64f2977. Native success does not explain or clear the browser startup failure.

The other rendered workspace-browser job, firmware and purge-namespace jobs passed. Neither those controls nor local shared-clock/cargo successes explain this startup failure. Keep the diagnosis and correction acceptance items open. No deadline increase, flag/sandbox bypass or blind retry has been used. The next production observation needs the identified binary, live root/descendant/process/resource evidence and confirmed cleanup while retaining the existing discovery budget.

Instrumented evidence checkpoint under qualification

PR #100, source f9b2917, adds startup-only native PID/Linux process, descendant, thread, discovery-file, cgroup-v2 and system-pressure observations. Actual executable descriptors are pinned without reading bytes before discovery; hashing follows discovery. Cooperative cancellation retains ownership until actual task settlement, prevents overlapping/later samples and hashing, and closes captured descriptors. An uninterruptible syscall is not given a fictitious hard deadline; finalization cannot allow a page behind unfinished observation work. Original Chrome arguments, inherited group and the 400 x 50 ms discovery guard are unchanged.

A held OS-adapter promise reproduced two overlapping observations before correction and passes after it. Final local types, native Buddy Pickier lint and docs passed; six focused tests passed with 65 assertions and one Linux-only case skipped. The actual execution-browser scenario and its guard passed locally with native routes and a genuine Worker. The separate earlier macOS workspace renewal failure remains retained under #64. Independent Spec and Standards reviews passed on all ten changed files, with source hashes unchanged through commit.

Exact-head CI 40083774805 is still running. Its early Linux native-fixture gate passed, including actual exited/live parent-child observations, independently checked executable digest, credential-sentinel privacy and actual pinned-descriptor closure on cancellation. Native artifact 12556286133 was retained once: ZIP 6,161 bytes, SHA-256 74e4ac2db44f8e518a3f7146a005585333bd4481f98428c1dad3b6d1bb4cb155. The live fixture retained seven Linux samples and actual root/child states; its 88,979,072-byte Bun backing executable hashed to 7ce7d6b654eddeec20afddf2396317fba6dc19ab52788a27e894e1806bf8f24b, after discovery, with closed descriptors.

Both actual rendered Linux browser jobs passed. The workspace artifact identifies Chrome/155.0.8059.39 from CDP and its 297,798,984-byte backing executable with SHA-256 9bfb381296dffe75f3419d07b6cc64525105ed117d1453a4ef3b291cb97d2b58. Workspace artifact 12556141810, retained once: ZIP 436,094 bytes, SHA-256 0bd1d2922fcb37ce1e191a6bd18d3afdcf877cac947465d31ad78dd15a10a0da. The execution artifact retained the same observed CDP version and backing file identity/size, but its Chrome digest is explicitly unavailable (ABORT_ERR) after the two-second hash budget; it must not be substituted with the workspace digest. Execution artifact 12556605863, retained once: ZIP 394,911 bytes, SHA-256 1b3cd7fb4b40420a524cf8b16f21239fa7c17e5954d3502b2f84329aee7ce467. Post-page stderr is withheld in both.

These are successful instrumented controls, not reproduction or explanation of the original failed startups. A configured launcher is not assumed to be the backing browser; hash timeout/permission/cap states remain explicit. Full workflow qualification and main publication are still pending. Root cause and correction acceptance remain open. The instrumented source includes pending shared-clock/intake prerequisites, but no fleet transfer or physical qualification is claimed.

Full workflow and publication checkpoint

Exact-source CI 40083774805 passed all five jobs. Native results: 2,101 passed, 41 skipped, zero failed and 507,073 assertions across 237 files, plus 13 Python bridge tests and all 31 copied cargo cases. All copied cargo report hashes match the qualified Linux intake release. The original workflow ZIP and each of the native/workspace/execution/cargo artifacts were retained once. The ten reviewed source hashes stayed unchanged. Source f9b2917 was published to origin/main as Chris, without coauthors and by normal fast-forward. The Linux workspace control captured a Chrome executable digest; the execution control retained a null Chrome hash after its actual 2 s budget abort. Do not substitute one control's binary identity for the other. This successful observed workflow does not reconstruct or explain the original failed startup. Issues #72 and #64 remain open for causal diagnosis.

8 days ago

The workspace-browser job on new test-instrumentation head c1b1569e1bf74a3cfe9c4d1b3b3687ee8026f9ce fails Fresh browser must start in workflow 40012425565. The private callback observer is used by the separate execution fixture; it does not change the shared Chrome launcher or this workspace fixture. No common cause is established.

The original uploaded workspace artifact was downloaded once and verified against its logged digest: 103,874 bytes, SHA-256 b786f5a6698d8e1fd4638288ac85eca11989f820f874a9ce7713dbb33333f10d. Its native launch report has 400 attempts, zero successful port-file reads, 400 ENOENT failures and discovered:false. The original 20-second bound elapsed; Chrome's exit 143/SIGTERM was observed only after owned cleanup began. Startup stderr is empty. This is not evidence of an early Chrome crash, a particular resource cause or a successful browser launch. No page was visited.

The separate execution-browser job passes and retains a successful closed launch report; that does not erase the workspace startup failure. The complete workflow is still running, its sole terminal-log collector has not downloaded logs yet, and #86 stays unqualified for main. No deadline/flag/authority changes or rerun were substituted. The startup cause and this issue's remaining criteria stay open.

Sign in to comment on this issue.