ReviewOS

stacks/ts-cloud

Implement scheduled backups, S3-compatible destinations, retention, and tested restores

#157
Closed chrisbbreuer opened this 24 days ago · 0 comments
24 days ago

Roadmap relationship

Outcome

Give databases, volumes, files, and supported infrastructure resources a unified backup policy and destination model with encryption, retention, integrity verification, restore previews, and regularly tested recovery.

Current gap

ts-cloud includes AWS Backup commands, on-box ts-backups, serverless database restore, and S3-compatible storage work, but these are fragmented. Dashboard backups are limited, database CLI backup/restore is largely placeholder, destination credentials/policies are not centrally managed, and successful backup does not prove restoreability.

User experience

  • Configure destinations with provider/endpoint/bucket/prefix/region, credential reference, encryption, immutability options, health test, and last success/failure.
  • Attach policies to resources with schedule/timezone, retention tiers, compression/encryption, include/exclude, and expected RPO/RTO.
  • Backup inventory shows resource, type, point-in-time, size, checksum, destination, expiry/lock, status, duration, and verification state.
  • Restore wizard selects target/new-vs-overwrite behavior, previews conflicts/downtime/data loss, runs preflight, streams progress, and validates health.
  • Dashboard highlights resources with no policy, missed RPO, unverified backups, or expiring credentials.

Scope

  • Define destination, policy, backup artifact/recovery point, job, restore job, verification, and retention schemas.
  • Support AWS S3/Backup plus generic S3-compatible destinations using endpoint/path-style capability from #116.
  • Cover managed database snapshots/PITR, logical DB dumps, persistent/named volumes, application shared files, and configuration/control-plane backups where supported.
  • Implement scheduled/manual backup, list, verify checksum/manifest, restore to new target, controlled in-place restore, retention cleanup, and legal/immutable hold metadata.
  • Add periodic restore drills into isolated targets with health validation and cleanup.

Technical direction

  • Use adapter interfaces for snapshot, dump, volume archive, destination, and restore strategies; record tool/engine versions.
  • Encrypt client-side where provider encryption is insufficient and keep keys as secret references.
  • Use multipart/resumable transfer, checksums, manifests, idempotent job IDs, and safe partial-upload cleanup.
  • Coordinate restore locks with deployments, jobs, data connections, and health gates.

Safety and operability

  • Never infer a successful restore from upload success; verification state is separate.
  • Overwrite restores require recent auth, exact target confirmation, fresh safety backup, and explicit acknowledgement of downtime.
  • Retention cleanup cannot delete active, held, pinned, or restore-in-progress artifacts.
  • Destination tests/logs redact keys and sensitive endpoints; SSRF/private endpoint policy is explicit.

Acceptance criteria

  • A policy produces scheduled recovery points, enforces retention, detects missed RPO, and reports delivery/verification failures.
  • A selected database and volume backup can restore into an isolated target and pass configured health checks.
  • S3-compatible destination configuration works without AWS-specific assumptions.
  • In-place restore is locked, audited, cancellability-aware, and leaves a clear recovery record.

Verification

  • Schedule/retention/hold, multipart/resume, checksum/corruption, credential expiry, partial cleanup, and idempotency tests.
  • Adapter fixtures for managed snapshot, logical dump, volume archive, AWS S3, and custom S3 endpoint.
  • Automated restore-drill end-to-end test with injected corruption/failure.

Non-goals

  • Claiming zero data loss beyond configured provider capabilities and RPO.
  • Backing up ephemeral logs/caches by default.
  • Deleting provider recovery points that ts-cloud did not adopt/manage.

Sign in to comment on this issue.