Roadmap relationship
Outcome
Run deployments and other long-lived mutations as durable background operations with ordered events, streaming output, explicit concurrency, safe cancellation, retries, and recovery after process restarts.
Current gap
Dashboard actions currently spawn CLI processes inside requests, while deployments across server/serverless/container paths have separate progress/history models. There is no persisted queue, resource lock, retry policy, queue visibility, or robust distinction between cancelling queued work and interrupting a provider operation.
User experience
- Show global and scoped queue views with status, target, actor/trigger, commit, priority, queued/start/duration times, attempt, and current step.
- Deployment detail streams ordered structured steps plus raw expandable logs, reconnects from a cursor, and clearly marks redacted/truncated content.
- Allow cancel queued, request cancellation of running work, retry failed/cancelled work, and clear completed history according to retention policy.
- Display why work is blocked (concurrency, dependency, resource lock, maintenance) and its estimated position without fake precision.
Scope
- Create persistent queue/lease/worker primitives using the operation/event schema.
- Add per-project, environment, server/provider, and build concurrency controls with configurable defaults.
- Define step checkpoints, heartbeats, timeouts, cancellation tokens, retry/backoff, idempotency, and orphan reconciliation.
- Adapt server, serverless, static, and container deployment entry points to enqueue shared jobs.
- Provide authenticated Server-Sent Events or WebSocket progress with cursor resume and polling fallback.
- Add queue management CLI/API/dashboard surfaces and deployment retention.
Technical direction
- Use transactional claim/lease semantics and resource-scoped locks to prevent conflicting deploy/destroy/restore work.
- Separate job orchestration from provider driver calls; drivers report structured progress and declare cancellability.
- Persist sanitized stdout/stderr chunks with sequence numbers and bounded storage.
- Propagate request, operation, deployment, provider, and audit correlation IDs.
Safety and operability
- Retries must be opt-in by error class and never blindly repeat destructive non-idempotent steps.
- Cancellation records what finished, what could not be cancelled, and required reconciliation.
- Secrets are redacted before persistence/streaming; log authorization matches target resource permission.
- Production concurrency and queue-clearing changes require elevated permission and audit.
Acceptance criteria
- A queued deployment survives dashboard restart and either resumes safely or ends in an explicit recoverable state.
- Conflicting operations on the same target cannot run concurrently.
- A disconnected browser resumes log streaming without gaps or duplicate visible entries.
- Cancel, retry, timeout, and provider failure produce deterministic terminal states and audit records.
Verification
- Lease expiry, duplicate worker, crash/restart, ordering, lock, retry, timeout, cancellation, and cursor-resume tests.
- Secret redaction and authorization tests for persisted and streamed logs.
- End-to-end concurrent deploy, restart-mid-deploy, cancel, and retry scenarios.
Non-goals
- Distributed multi-region queue workers in the first release.
- Claiming provider operations are cancellable when their API is not.
- Replacing CI provider job queues outside ts-cloud.