ReviewOS

stacks/drone

Payload IPC: diagnose camera clock synchronization failure in the full suite

#35
Open chris opened this 9 days ago ·
9 days ago

Observed failure

The first local full ./buddy test run while qualifying 8157713 failed the existing test bounded child IPC handles fragmented records and maps the child monotonic clock in tests/unit/payloads.test.ts:149.

The retained error is PayloadCaptureError, code invalid, message Camera clock synchronization exceeded uncertainty budget, from the composite sync guard in resources/functions/drone/payloads/process.ts:61. That guard checks packet kind, the child clock value and parent round-trip validity/budget. The log does not identify the failed predicate, packet fields, parent send/receive times or whether the first or second capture failed. The 218.21 ms test duration is not a measured sync round trip. Root cause is unknown.

The payload process, types, capture implementation and worker/test fixtures byte-match published f860f36. Its automatic main CI passed all five jobs. These facts establish source identity and a prior passing run; they do not establish the cause of this local failure.

Retained evidence

  • First full run remains in progress at the initial receipt; partial counts are not a terminal result.
  • Exact observed log prefix: 178,499 bytes, SHA256 e2fee59b5fa444a0d812458e05da814bfd9eac0de0f2d528edec3ce4e5bad1ee.
  • Full log, first failure context and five baseline-equal source snapshots are retained locally. No qualification retry or source change has occurred.

Acceptance criteria

  • Establish a focused red-capable loop for the actual existing IPC/capture seam and retain every observed verdict.
  • Capture bounded nonsecret sync-phase observations that distinguish packet-field, monotonic-clock and round-trip conditions. Preserve the existing 5 ms uncertainty budget and fail-closed behavior.
  • Apply a correction only after its cause is established, with meaningful regression evidence. An unreproduced failure remains explicitly unknown.
  • Qualify the resulting source with relevant focused/integrated checks and exact-head CI, preserving the original failure and separating controlled-clock, real-clock and hardware evidence.

This is local process-fixture evidence. Pi camera hardware support and moving-camera acceptance retain their own criteria in #29 and #26. Counted-encoder publication is held pending resolution of its full-suite qualification. No clock threshold, firmware or physical authority change is authorized by this diagnosis ticket.

Diagnostic checkpoint, 2026-10-20

The original full local run completed with exit 1, 1,145 passes, 24 skips and two failures. The second failure, in the separate 137-review purge scenario, is tracked in #36. The same candidate 8157713fec136f7d0ca463ba0ba56081268d6e39 passed all five jobs in CI run 39503133989, including 1,132 passes, 39 skips and zero failures in its Linux full suite. This does not explain the local failure.

The unchanged isolated IPC test passed once with six assertions in 81.07 ms. A separately recorded reproduction campaign, bounded to 64 attempts and stopping on the first failure, passed attempt 1 and failed attempt 2. That failure reached validateCaptureAck at capture.ts:29, with Capture ACK identity, time or capabilities invalid, rather than the original synchronization guard. It had zero successful capture assertions. The two rejecting guards remain distinguished; no actual failed ACK values were retained in that output. All receipts and original sources are preserved. The original synchronization failure remains unexplained. No causal fix, acceptance-limit change or qualification retry is claimed.

Reproduced guard observations

Temporary rejection-only probes on an isolated diagnostic branch captured two distinct failures of the unchanged original real-clock test. Both original rejection conditions, clock reads, 5 ms limit, test assertions and peer were preserved; the probes are excluded from product publication.

  • Synchronization: valid response kind and finite nonnegative worker timestamp; parent send 291.499 ms, receive 303.561041 ms, measured round trip 12.062040999999965 ms. Its half round trip exceeded 5 ms, so the adapter rejected before sending a capture command. The bounded campaign stopped at this failure on attempt 17.
  • ACK: valid identity and timing flags, dimensions and JPEG size; mapped capture/ACK time 210.3229165 ms exceeded parent receipt 209.979 ms, with combined uncertainty 2.7729585000000103 ms. The strict ordering check rejected this point estimate. A separately bounded campaign stopped at this ACK observation on attempt 19.

These observations explain the rejecting predicates in those specific attempts. They do not retroactively supply the missing values from the original full-run failure or establish which scheduling component delayed a message. The current test assumes unconditional real-clock success that these conservative checks cannot guarantee. A tests-only correction is being implemented with the existing public injected clock seam, real fragmented child IPC, fixed positive mapping samples, and explicit rejection/retirement tests for excessive round trip, future mapped ACK and exact combined uncertainty boundaries. Production acceptance checks and limits remain unchanged. No fix has yet been published or issue criterion marked complete.

Published tests-only correction and qualification, 2026-10-20

56363b5 is published to main as Chris without coauthors. Its IPC correction changes only tests/unit/payloads.test.ts and the new fixed clock peer tests/fixtures/payloads/clock-worker.ts. The existing positive retains two captures and all six capture assertions over actual fragmented child stdin/stdout. Literal injected clock samples make the mapping case deterministic. Added cases verify a 12 ms round trip rejects before capture, a future mapped ACK rejects, and combined uncertainty of exactly 5 ms is admitted while 5.25 ms is rejected. Rejection cases check closed session/request behavior. Production process/capture checks, the 5 ms limit and original unrelated worker/closure cases remain unchanged. Temporary diagnostic probes were restored and excluded.

Pinned Bun 1.4.2 focused checks passed four tests with 30 assertions; the complete payload file passed 21 tests with 194 assertions. Runtime/application types and Pickier passed. The first full run of the changed combined candidate finished with exit 0: 1,150 passed, 24 skipped, zero failed, 413,558 assertions, 1,174 tests across 165 files. Exact-head CI passed all five jobs; its Linux full suite reports 1,135 passed, 39 skipped, zero failed and 413,492 assertions. Compiled legacy output comparisons and five real-browser legacy module comparisons also passed. These are controlled-clock IPC and regression results, not arbitrary-host real-clock success or Pi hardware qualification.

The observed rejection predicates in the two bounded probed attempts establish why the unconditional success assumption was invalid. The exact values and scheduling component in the original failed full-run attempt remain unknown; this checkpoint does not invent them or claim a production timing fix. All first failures and subsequent verdicts stay retained, and the original checklist is preserved. #36 remains a separate unresolved purge diagnosis.

Sign in to comment on this issue.