Closes #195. Also fixes #194, folded in rather than landed separately — see below.
The problem
buildHostCleanupScript() was invoked from exactly one place: the tail of a successful deploy. Everything else followed from that.
- A box that stops deploying never cleans again, however full it gets.
- A deploy that fails skips cleanup entirely — the run most likely to have stranded something.
- Nothing reacted to the disk filling. The windows were fixed constants whether root was at 20% or 95%. The box in #194 sat at 84% and was found by looking.
- Retention windows were literals in the returned strings, with no way for an operator to say that 140 sites on 150 GB is a different problem from three.
- The path list was closed.
/root/rpx-backupsat 3.0 GB, pluscert-backup-*,unit-backups-*andrpx-rollback-*in/root, had no way to participate.
What changed
The pass is installed as /usr/local/sbin/ts-cloud-cleanup plus a daily ts-cloud-cleanup.timer, and still runs inline at the end of a deploy so a deploy's own leftovers don't wait up to a day for the timer.
It reads root usage before doing anything, and behaves by band:
| Root usage | Behavior |
|---|---|
Below relaxedBelowPercent (50%) | Deploy leftovers only. The Bun cache, container images and apt — things that cost network to rebuild — are left alone. |
| Between the thresholds | Full pass at configured windows. |
At or above escalateAtPercent (85%) | Full pass, every configurable window divided by escalationFactor (4). |
Bounds that exist for correctness rather than retention never move with pressure: an in-flight upload belonging to a concurrent deploy on a shared box keeps its hour however full the disk is.
Windows are configurable via infrastructure.compute.cleanup, and other components register their own rules rather than leaking:
cleanup: {
artifactMaxAgeMinutes: 720,
rules: [
{ path: '/root/rpx-backups', pattern: 'cert-backup-*', maxAgeMinutes: 4320, directories: true },
],
}On #194
Its two-line fix is here as two entries in the new rule table rather than as a separate commit — the artifact rules are now ordinary rules, so there was no smaller version of this change to land first. If you'd still rather have the narrow fix on its own, say so and I'll split it out.
The underlying bug is fixed: artifact retention was inlined in the Hetzner driver's upload path, which put it on the cache-miss branch (the box reusing the cache most pruned it least), and its *.tar.gz filter could never match the .<digest>-<uuid>.tmp files a died-mid-SCP upload strands. Both now run unconditionally, and the inline find is gone from driver.ts.
Not failing deploys
This runs after the release is already live, so every step is guarded. A host with no systemd or a read-only /etc degrades to the inline pass. The root-usage assignment carries || true — under the deploy script's set -euo pipefail, a bare assignment from a failing df pipeline aborts the shell. Verified both ways with a stubbed failing df:
guarded: root at 0% — keeping rebuildable caches ... exit=0
unguarded: (aborts) exit=1Verification
bun test— 2589 pass / 0 fail (ts-cloud), 1727 pass / 0 fail (core). 8 new tests.bunx --bun pickieron changed files — clean. (The repo's 13 pre-existing errors are unrelated files, unchanged onmain.)- Generated script
sh -n/bash -nclean, outer and inner. - All three pressure bands exercised against a stubbed
df. - Functionally verified in a sandbox: an aged orphan named like the real one from the production box (
.960be54ea6efabb…tmp) and an aged*.tar.gzwere deleted; a fresh in-flight upload and fresh cached tarball survived. Directory rules verified the same way.
Not verified on a real host — worth one deploy to a non-production box to confirm the timer installs and systemctl list-timers shows it.
🤖 Generated with Claude Code