ReviewOS

reviewos/reviewos.org

Preserve runner log-delivery failure causes during step cleanup

#10
Closed chris opened this 3 days ago ·
3 days ago

HQ onboarding CI run 2026 failed during Pantry service setup before any unit steps started. The runner reported only: The step output could not be delivered; the command was stopped.

The executor currently uses that same message for an HTTP/retry failure and for an in-flight delivery at the fixed post-exit pipe grace. Preserve the underlying delivery reason and identify the grace case explicitly. Keep existing bounded retries, fail-closed conclusions, background service handling, and log redaction.

Acceptance: real HTTP/process recovery tests distinguish persistent 503 exhaustion from pending post-exit delivery; transient restart and lost-ack tests still retain the final output; background-process cleanup still passes. Deploy through the existing runner path and inspect the next actual failure before changing delivery semantics.

Related: https://github.com/stacksjs/hq.training/issues/318 and https://github.com/stacksjs/hq.training/issues/346 Run: https://reviewos.org/stacks/hq.training/run/2026

3 days ago

Diagnostics implemented and pushed as https://github.com/ReviewOS/reviewos.org/commit/4ea9889ace41a472a09d9ee0dab3ecc33a4f45d5.

The existing real HTTP/process recovery tests were red for the missing reason and are now green. Recovery plus orphan cleanup: 12 tests, 48 assertions passed; final recovery after cause-priority adjustment: 5 tests, 33 assertions passed. Buddy lint and Buddy typecheck passed. Normal buddy build:runner --target linux-x64 passed, and the existing installBinary upgrade path installed SHA256 prefix 067c8009eba6, retaining the old binary. SIGUSR2 reached only the main runner process, which acknowledged finishing its active job before stopping. Active job 5981 is preserved; restart onto the installed binary is pending its completion.

This improves diagnosis and preserves the existing fail-closed transport behavior. The original cause of run 2026 is not yet established, and this issue remains open until the new binary has taken over and actual delivery has been observed.

3 days ago

The diagnostic fix (4ea9889) is now installed on both existing runners through normal compiled binaries and graceful SIGUSR2 restart. The Linux running executable hash is 067c8009eba644500f1d8845c01f77af65525df4127d0256e99ba3e382d94347; the Mac binary is 416d7dddd7fa0b1791a4f424da3124e9fd9f949680925971f5a78f89066727ea. Existing capacity, retries and fail-closed behavior are unchanged. The 13 real HTTP/process recovery, cleanup and provisioning tests pass (58 assertions), with Buddy lint and types passing.

The next old-binary Mac failure, run 2045/job 6011, happened during checkout: Git reported a connection reset and early EOF before any workflow step started. Its generic output-delivery message cannot be retrospectively assigned a delivery cause. The new Mac binary is now active for the next run, and no delivery semantics or checkout checks have been weakened.

3 days ago

Implemented in 4ea9889, with 13 real HTTP/process regressions passing (58 assertions), Buddy lint/types passing, and both normal compiled runner binaries installed through graceful restart. The subsequent Linux failure in HQ run 2036 was inspected using its retained unit artifact: two private coaching SEO contract failures, not a delivery failure. Those are fixed on current HQ main and the next complete unit gate passed in run 2047. Delivery retries, pipe grace, capacity limits and fail-closed conclusions remain unchanged. See the preceding comment for binary hashes and the separately inspected old-binary Mac checkout failure.

Sign in to comment on this issue.