ReviewOS

stacks/ts-cloud

Investigate: April 2026 AWS bill spike (~$700, suspect S3 egress from registry publishes)

#103
Closed glennmichael123 opened this 2 months ago ·
2 months ago

Context

April 2026 AWS bill came in around $700 — significantly higher than baseline. Working hypothesis: S3 data transfer (egress) from publishing packages to our private registry, since publish frequency has been high across the ecosystem.

What to investigate

  1. Cost Explorer breakdown by service — confirm S3 is the dominant line, not surprise EC2/CloudFront/data-transfer-out elsewhere.
  2. S3 by bucket — which bucket is doing the most egress? The registry bucket should be the suspect.
  3. S3 by usage type — separate DataTransfer-Out-Bytes from Requests-Tier1/2. Egress is what scales with downloads, requests scale with publish count.
  4. Cross-reference with publish frequency — count npm publish / pantry releases in April. If 10x last month, we found it.
  5. Object size audit — any tarballs >50MB pushed to the registry would inflate egress on every download. (.pantryignore was added to ts-countries for exactly this reason.)

Why this matters

Even if April turns out to be a one-off, we don't have a shared answer to "what's costing us money this month." That's the gap the cost-reporting CLI (separate issue) closes — but this issue is the one-time forensic dig that confirms or refutes the registry-egress hypothesis and tells us where to look first when the CLI gets built.

Acceptance

  • Root cause identified and posted in this issue (with a CSV/screenshot from Cost Explorer)
  • If registry egress confirmed: short-term mitigation logged (CDN in front of registry? bigger pantryignore sweep? colder storage class for old versions?)
  • Findings feed into the cost-reporting CLI scope (linked issue)
9 days ago

Findings

Ran `cost:analyze` against both profiles for April 2026:

ProfileTotalTop driver
`default`$352.16 across 16 servicesEC2 + VPC + EC2-Other = ~$309 (88%) — compute/network heavy
`stacks`$307.63 across 19 servicesS3 $191.91 (62.4%) — registry-egress hypothesis confirmed
Combined$659.79≈ the "around $700" sticker shock

Key drivers

`stacks` profile:

  • Amazon Simple Storage Service: $191.91 (62.4%, 50 buckets)
  • EC2 - Compute: $41.78 (13.6%)
  • VPC: $18.00 (5.9%)
  • WAF: $15.00 (4.9%)

`default` profile:

  • EC2 - Compute: $158.07 (44.9%)
  • VPC: $82.80 (23.5%)
  • EC2 - Other: $68.55 (19.5%) — typically NAT gateway data processing / EBS / inter-AZ
  • WAF: $23.00 (6.5%)

Conclusion

Hypothesis from #103 (registry egress) holds for the `stacks` profile — S3 is the dominant cost there, and $191/mo of S3 spend on a registry-style workload is consistent with high egress. The `default` profile bill is a separate story: EC2 + NAT/transfer ate ~$309. That's the always-on infra; not registry-related.

Follow-ups (already filed)

  • #104 — cost-reporting CLI tracker (top-services done; top-buckets / top-egress filed below)
  • #105 — `listBuckets()` returns 0 anomaly (closed; was a parser bug, not IAM scoping)
  • #106 — Cost Explorer response cache (closed; ships in #76316e2)
  • #107 — stub commands return mock data (closed; warn-and-track + #108–#112 sub-issues)

Closing this investigation issue. Cost-savings work continues under the implementation issues above.

Sign in to comment on this issue.