ReviewOS

stacks/ts-cloud

cost:analyze: top S3 buckets by spend (per-bucket cost breakdown)

#113
Closed glennmichael123 opened this 24 days ago · 0 comments
24 days ago

Context

`cost:analyze` ranks AWS services for the last full month — at the service granularity — and shows the bucket count next to the S3 line. When S3 dominates the bill (e.g. $191.91 / 62.4% on the `stacks` profile in April 2026, see #103), the service-level number doesn't tell us which buckets are eating the spend.

Proposal

Add a per-bucket breakdown for S3, surfaced either as a new subcommand or a flag on `cost:analyze`:

```sh cloud cost:analyze --service s3 --by bucket

or

cloud cost:buckets ```

Output sketch:

``` S3 buckets by spend — April 2026 (profile: stacks)

Bucket Storage Transfer-Out Total ───────────────────────────────────────────────────────────────────── pantry-registry $ 12.40 $148.20 $160.60 stacks-production-s3-public $ 8.10 $ 14.30 $ 22.40 … ```

Implementation challenges

Cost Explorer's `GetCostAndUsage` does not support grouping by individual bucket out of the box. Two viable paths:

Option A — Cost & Usage Report (CUR) ingestion

  • AWS publishes detailed line items (per-bucket, per-usage-type) to an S3 bucket if CUR is enabled.
  • One-time setup: create a CUR, point it at a bucket, wait for the first report.
  • Once enabled, parse CUR Parquet/CSV and aggregate per `resource_id`.
  • Pros: authoritative, includes resource IDs, no extra request charges.
  • Cons: requires opt-in setup; first report can take ~24h; CUR storage itself costs cents/month.

Option B — Resource tagging + cost allocation tags

  • Tag every bucket with a `Project` / `App` / etc. tag.
  • Activate the tag as a cost allocation tag in Billing settings.
  • Group Cost Explorer by that tag (it does support tag grouping).
  • Pros: no CUR setup; tag groupings are first-class in CE.
  • Cons: requires every bucket to be tagged consistently — which is fine for newly-provisioned ones via ts-cloud, but legacy buckets need backfill. Also: not per-bucket but per-tag.

Option C — Approximate via S3 metrics + storage classes

  • Pull `BucketSizeBytes` and `NumberOfObjects` from CloudWatch (free).
  • Multiply by published S3 pricing for storage tier.
  • Skip transfer-out entirely (CloudWatch doesn't expose per-bucket egress).
  • Pros: zero AWS-side setup.
  • Cons: misses the most important cost driver for registry-style workloads (egress), so it's misleading.

Recommended: Option A for accuracy. Option B as a fallback if CUR is not set up. Option C is too lossy for the common case (egress-dominated buckets).

Acceptance

  • Detect whether CUR is enabled; if yes, use it; otherwise warn with setup instructions.
  • Output ranks buckets by total cost descending.
  • Includes storage vs. transfer-out split where the source allows it.
  • Documented in `docs/features/cost.md`.
  • #103 — original bill investigation that surfaced this
  • #104 — cost-reporting CLI tracker
  • #106 — response cache (CUR ingestion is locally cacheable too)

Sign in to comment on this issue.