Add health checks, alerts, and multi-channel notification routing

Roadmap relationship

• Parent: #135
• Master roadmap: #132
• Depends on: #139, #142, #153

Outcome

Detect service and platform failures with explicit health checks and alert rules, then route deduplicated lifecycle notifications to configured channels with ownership, quiet hours, escalation, and delivery diagnostics.

Current gap

ts-cloud has CloudWatch alarms, host monitoring, health-check primitives, and Slack/Discord/Telegram/email/webhook notification config, but no unified rule model, routing policy, alert lifecycle, test delivery, deduplication, silence, escalation, or incident context.

User experience

• Health configuration supports HTTP/TCP/command checks with interval, timeout, expected response, thresholds, regions/agents, and preview/test.
• Alert list shows firing/pending/resolved/silenced states, affected scope, severity, first/last seen, duration, owner, related deployment, and evidence.
• Notification channels can be created, tested, paused, rotated, and scoped; routing rules preview which events reach which destinations.
• Support maintenance windows, quiet hours/timezones, grouping, deduplication, reminders, escalation, acknowledgement, and resolved messages.
• Failures to deliver notifications are visible and retryable without hiding the original alert.

Scope

• Normalize health results and alert rules for uptime, latency, error rate, saturation, certificate expiry, deployment, queue, backup, security, cost, and provider health.
• Implement alert evaluation state machine with consecutive thresholds, no-data policy, hysteresis, dedup/group keys, and recovery.
• Support Slack, Discord, Teams-compatible webhook, Telegram, email, generic webhook, and extensible channel adapters.
• Add routing by organization/project/environment/service/severity/event, channel templates, signed webhooks, and delivery history.
• Feed health gates into deployment activation and fleet status.

Technical direction

• Evaluate rules from normalized observability queries or provider-native alarms while retaining source IDs.
• Persist alert and delivery events with idempotency and bounded retries/dead-letter visibility.
• Run synthetic checks from explicit agents/regions and record DNS/connect/TLS/TTFB timings.
• Version templates/rules and sanitize payloads before adapter dispatch.

Safety and operability

• Encrypt channel credentials and never include secret/environment values or sensitive URLs in notification payloads.
• Prevent alert storms through grouping/rate limits without dropping terminal resolution state.
• Signed outgoing webhooks include replay-resistant timestamps; inbound acknowledgements require auth.
• Production health/alert disablement and broad silences require elevated permission and audit.

Acceptance criteria

• A failed health check transitions pending → firing after configured thresholds and resolves after recovery.
• Routing sends one grouped notification to expected channels, records delivery, retries transient failure, and surfaces permanent failure.
• A deployment can wait on a named health gate and automatically fail/roll back according to release policy.
• Channel tests and rule previews do not create fake production incidents.

Verification

• State transition, no-data, hysteresis, grouping, silence/DST, escalation, retry, and dedup tests.
• Adapter contract fixtures and credential/redaction/signature tests.
• End-to-end synthetic outage → alert → notification → acknowledgement → recovery.

Non-goals

• Staffing an incident response service.
• Guaranteeing delivery through third-party channel outages.
• Replacing provider-native alarms where they are the better source.

Implemented end to end on main in small conventional commits:

• bc516f9 persists schema v22 health checks/results, alert rules/lifecycle/events/silences, encrypted channels/routes/deliveries, telemetry and health evaluators, retries, signing, quiet hours, grouping, reminders, escalation, and tests.
• 18affe0 makes named health checks release gates and automatically fails/rolls back unhealthy activations.
• 20ff755 adds scoped dashboard APIs, scheduled evaluation/check/retry sweeps, recent-auth enforcement for sensitive mutations, delivery diagnostics, channel tests, route previews, and SES-backed email delivery.
• 5b83e07 adds CLI health, alert, channel-test, evaluation, acknowledgement, and retry operations.
• a5f0c4b adds the responsive Alerts & Health workspace with explicit empty states and no sample fallbacks.
• 5b0cb8e adds schema v23 route templates and per-minute storm limits while ensuring resolved notifications bypass the limit.

Verification:

• Focused alert/control-plane/release/route suites: 32 passing, 0 failing after the final changes.
• Full @stacksjs/ts-cloud suite: 1,064 passing, 0 failing (plus one intentional skip and one todo).
• Production @stacksjs/ts-cloud build: 49/49 dashboard routes generated.
• Standalone UI build: 49/49 routes generated.
• Lint check: 0 errors.

Acceptance evidence includes pending → firing → recovery, no-data policy, metric hysteresis, normalized telemetry evidence, deduplication, grouping waits, silence/DST, reminders/escalation, bounded retry/dead delivery, encrypted credentials, signed webhooks, route templates/rate limiting, named release-gate rollback, and the synthetic outage → acknowledgement → recovery journey. Channel and check previews explicitly create no incident.