Roadmap relationship
Outcome
Detect service and platform failures with explicit health checks and alert rules, then route deduplicated lifecycle notifications to configured channels with ownership, quiet hours, escalation, and delivery diagnostics.
Current gap
ts-cloud has CloudWatch alarms, host monitoring, health-check primitives, and Slack/Discord/Telegram/email/webhook notification config, but no unified rule model, routing policy, alert lifecycle, test delivery, deduplication, silence, escalation, or incident context.
User experience
- Health configuration supports HTTP/TCP/command checks with interval, timeout, expected response, thresholds, regions/agents, and preview/test.
- Alert list shows firing/pending/resolved/silenced states, affected scope, severity, first/last seen, duration, owner, related deployment, and evidence.
- Notification channels can be created, tested, paused, rotated, and scoped; routing rules preview which events reach which destinations.
- Support maintenance windows, quiet hours/timezones, grouping, deduplication, reminders, escalation, acknowledgement, and resolved messages.
- Failures to deliver notifications are visible and retryable without hiding the original alert.
Scope
- Normalize health results and alert rules for uptime, latency, error rate, saturation, certificate expiry, deployment, queue, backup, security, cost, and provider health.
- Implement alert evaluation state machine with consecutive thresholds, no-data policy, hysteresis, dedup/group keys, and recovery.
- Support Slack, Discord, Teams-compatible webhook, Telegram, email, generic webhook, and extensible channel adapters.
- Add routing by organization/project/environment/service/severity/event, channel templates, signed webhooks, and delivery history.
- Feed health gates into deployment activation and fleet status.
Technical direction
- Evaluate rules from normalized observability queries or provider-native alarms while retaining source IDs.
- Persist alert and delivery events with idempotency and bounded retries/dead-letter visibility.
- Run synthetic checks from explicit agents/regions and record DNS/connect/TLS/TTFB timings.
- Version templates/rules and sanitize payloads before adapter dispatch.
Safety and operability
- Encrypt channel credentials and never include secret/environment values or sensitive URLs in notification payloads.
- Prevent alert storms through grouping/rate limits without dropping terminal resolution state.
- Signed outgoing webhooks include replay-resistant timestamps; inbound acknowledgements require auth.
- Production health/alert disablement and broad silences require elevated permission and audit.
Acceptance criteria
- A failed health check transitions pending → firing after configured thresholds and resolves after recovery.
- Routing sends one grouped notification to expected channels, records delivery, retries transient failure, and surfaces permanent failure.
- A deployment can wait on a named health gate and automatically fail/roll back according to release policy.
- Channel tests and rule previews do not create fake production incidents.
Verification
- State transition, no-data, hysteresis, grouping, silence/DST, escalation, retry, and dedup tests.
- Adapter contract fixtures and credential/redaction/signature tests.
- End-to-end synthetic outage → alert → notification → acknowledgement → recovery.
Non-goals
- Staffing an incident response service.
- Guaranteeing delivery through third-party channel outages.
- Replacing provider-native alarms where they are the better source.