ReviewOS

stacks/ts-cloud

Avoid holding the storage coordinator while waiting for maintenance exclusion

#204
Closed chris opened this 3 days ago ·
3 days ago

An actual shared-host deployment has an ensure-site process holding the compute-storage lock while waiting for the required host-backup lock. Another already placed site is blocked behind the compute lock even though its ensure operation would only validate existing mounts. Public cloud 0.16.89 fixes optional maintenance and unchanged apply, but a genuine pending placement can still monopolize the coordinator while its backup exclusion is unavailable. This is now delaying exact-source SDK CI and enterprise acceptance infrastructure updates.

Keep required backup/migration exclusion. Acquire the required maintenance locks without holding the compute coordinator, then reacquire it and reload/revalidate policy, mounts and pending entries before any copy or cutover. An already healthy site's no-op ensure must remain usable while a different placement waits for backup exclusion. Optional reclamation must also defer when an active deployment lock is busy, instead of waiting while holding global locks.

Prove this with real Linux flock and private mount-namespace tests: hold backup exclusion, start a pending placement, verify the compute lock remains available and an inherited site completes; preserve the unmounted pending source until exclusion is released. Preserve volume identity checks, migration opt-in, capacity checks, health gates and rollback. Publish normally and consume the public package; do not edit installed host agents or stop unrelated backups/deployments.

Related: https://github.com/ReviewOS/reviewos.org/issues/14, https://github.com/ReviewOS/reviewos.org/issues/13 and https://github.com/stacksjs/hq.training/issues/346.

3 days ago

Implemented in https://github.com/stacksjs/ts-cloud/commit/2f9fb88a5da25df2374da5ee3f82f30868c0e06e. The normal bun run release:patch created v0.16.90 at 5ab2977; its publisher is https://github.com/stacksjs/ts-cloud/actions/runs/41110895477. Publication and consumer rollout are still pending.

Real Linux proof uses private mount namespaces, shadowed configuration/lock paths and one owned temporary volume directory. The regression fails against the previous agent because the compute coordinator remains held during required backup exclusion. It passes after the fix: inherited ensure works while a different placement waits; two concurrent placements resume and record one journal entry; optional reclamation defers on an active deployment lock; original and newly placed data survive. Final real test: 1 pass, 0 failures/skips, 4 assertions. No production database, configuration, lock or existing backup/deployment process was changed.

The 76 focused storage/deployment tests pass with 249 assertions; full types and Pickier (913 files, zero errors/warnings) pass. Required exclusion, volume/mount checks, migration opt-in, capacity checks, health gates and rollback remain enforced. This ticket stays open until the public package is installed and the maintained helper is rolled out normally.

1 day ago

Published rollout verified on November 1 (Pacific time). The original fix shipped through normal v0.16.90, whose publisher succeeded. HQ and ReviewOS now pin and install public cloud 0.16.91, which contains it. The live ReviewOS API release 3e057f9 uses installed cloud 0.16.91.

The maintained host helper at /usr/local/lib/ts-cloud/compute-storage.mjs has SHA-256 ba34c48ebbbb114d318cec1e4e03f40bd64b1bc6410b4ca13051d74a0bd90610, exactly matching the helper emitted by the installed published 0.16.91 buildComputeStorageInstallScript API. This was read-only verification, without changing locks, mounts, backup operations or production configuration.

Combined with the retained real Linux namespace/flock regressions and source checks above, both source and consumer rollout requirements are complete. ReviewOS rollout: /reviewos/reviewos.org/issues/14. Enterprise acceptance remains tracked separately at /stacks/hq.training/issues/346.

Sign in to comment on this issue.