ReviewOS

also looking at this

stacks/ts-cloud

feat(deploy): make host cleanup a property of the host, not of deploying

#196
Closed glennmichael123 wants to merge feat/host-cleanup-timer into main
6 files +374 -21
9 days ago

Closes #195. Also fixes #194, folded in rather than landed separately — see below.

The problem

buildHostCleanupScript() was invoked from exactly one place: the tail of a successful deploy. Everything else followed from that.

  • A box that stops deploying never cleans again, however full it gets.
  • A deploy that fails skips cleanup entirely — the run most likely to have stranded something.
  • Nothing reacted to the disk filling. The windows were fixed constants whether root was at 20% or 95%. The box in #194 sat at 84% and was found by looking.
  • Retention windows were literals in the returned strings, with no way for an operator to say that 140 sites on 150 GB is a different problem from three.
  • The path list was closed. /root/rpx-backups at 3.0 GB, plus cert-backup-*, unit-backups-* and rpx-rollback-* in /root, had no way to participate.

What changed

The pass is installed as /usr/local/sbin/ts-cloud-cleanup plus a daily ts-cloud-cleanup.timer, and still runs inline at the end of a deploy so a deploy's own leftovers don't wait up to a day for the timer.

It reads root usage before doing anything, and behaves by band:

Root usageBehavior
Below relaxedBelowPercent (50%)Deploy leftovers only. The Bun cache, container images and apt — things that cost network to rebuild — are left alone.
Between the thresholdsFull pass at configured windows.
At or above escalateAtPercent (85%)Full pass, every configurable window divided by escalationFactor (4).

Bounds that exist for correctness rather than retention never move with pressure: an in-flight upload belonging to a concurrent deploy on a shared box keeps its hour however full the disk is.

Windows are configurable via infrastructure.compute.cleanup, and other components register their own rules rather than leaking:

cleanup: {
  artifactMaxAgeMinutes: 720,
  rules: [
    { path: '/root/rpx-backups', pattern: 'cert-backup-*', maxAgeMinutes: 4320, directories: true },
  ],
}

On #194

Its two-line fix is here as two entries in the new rule table rather than as a separate commit — the artifact rules are now ordinary rules, so there was no smaller version of this change to land first. If you'd still rather have the narrow fix on its own, say so and I'll split it out.

The underlying bug is fixed: artifact retention was inlined in the Hetzner driver's upload path, which put it on the cache-miss branch (the box reusing the cache most pruned it least), and its *.tar.gz filter could never match the .<digest>-<uuid>.tmp files a died-mid-SCP upload strands. Both now run unconditionally, and the inline find is gone from driver.ts.

Not failing deploys

This runs after the release is already live, so every step is guarded. A host with no systemd or a read-only /etc degrades to the inline pass. The root-usage assignment carries || true — under the deploy script's set -euo pipefail, a bare assignment from a failing df pipeline aborts the shell. Verified both ways with a stubbed failing df:

guarded:   root at 0% — keeping rebuildable caches ... exit=0
unguarded: (aborts)                                    exit=1

Verification

  • bun test — 2589 pass / 0 fail (ts-cloud), 1727 pass / 0 fail (core). 8 new tests.
  • bunx --bun pickier on changed files — clean. (The repo's 13 pre-existing errors are unrelated files, unchanged on main.)
  • Generated script sh -n/bash -n clean, outer and inner.
  • All three pressure bands exercised against a stubbed df.
  • Functionally verified in a sandbox: an aged orphan named like the real one from the production box (.960be54ea6efabb…tmp) and an aged *.tar.gz were deleted; a fresh in-flight upload and fresh cached tarball survived. Directory rules verified the same way.

Not verified on a real host — worth one deploy to a non-production box to confirm the timer installs and systemctl list-timers shows it.

🤖 Generated with Claude Code

6 changed files on the files tab, with 0 review threads.

Sign in to comment on this pull request.