ReviewOS

also looking at this

craft-native/craft

Gate SBOM findings on whether they ship

#321
Merged glennmichael123 wants to merge fix/sbom-shipping-scope into main
5 files +230 -1
9 days ago

Closes #279.

#316 closed three of this issue's five asks: no scanner invocation swallows its exit code, scanning is scoped to the exact shipped archives, and the installer downloads are pinned. What remained was the part that made the scan meaningless — 30 High findings and a permanently green job.

Why High was ungated

I pulled the actual Grype report from the last successful SBOM run and classified every High and Critical finding by its recorded path. All 27, across 8 packages, come from somewhere that publishes nothing:

PackagePath
lodash, brace-expansion, minimatch (5.1.6 and 9.0.5)/benchmarks/apps/electrobun/bun.lock
electron, extract-zip/benchmarks/apps/electron/bun.lock
stdlib@go1.26.4, golang.org/x/text/node_modules/@typescript/typescript-linux-x64/lib/tsc

Not one comes from Craft. That explains the flag: the scan could not fail on High without going permanently red over a benchmark's dependencies, so it failed on nothing instead. The two Go findings are the runtime inside the TypeScript compiler binary, which is a neat illustration of how little the repository-wide count says about what Craft ships.

Scope, not a list

scripts/sbom-scope.ts fails when a High or Critical finding is not confined to /benchmarks/, /node_modules/, /examples/, /artifacts/ or /.github/, and otherwise prints the tooling inventory so the number stays visible without deciding the job. It runs unconditionally; enforce_high survives as the stricter opt-in.

Three deliberate edges:

  • "Every location", not "any" — a package in both a benchmark and Craft is Craft's problem.
  • No location means not tooling — an unlocatable High gets looked at, never waved through.
  • A malformed or absent report throws, matching scan-sbom.ts's existing refusal to read a failed scan as good news.

I chose a path rule over a retention list on purpose, which differs from the approach I just used for lockfile advisories in #320. Benchmark lockfiles and node_modules versions move constantly, so entries pinned to package-and-version would be stale within days and their own anti-rot check would fire at every benchmark bump — a gate that cries wolf gets switched off. Lockfile advisories are few and slow-moving, so retentions suit them. The docs say which instrument applies where, and why.

Evidence

  • 9 unit tests: benchmark-only passes, the compiler's Go runtime passes, a Craft path fails, a both-places package fails, an unlocatable High fails, Medium and below are inventory anywhere, Critical is gated exactly like High, a malformed report throws.
  • Against the real report from run 36139517229: No High or Critical finding reaches a published artifact. Tooling inventory: 27 findings across 8 packages — lodash@4.17.23 (1), brace-expansion@2.0.2 (3), stdlib@go1.26.4 (7), minimatch@5.1.6 (3), minimatch@9.0.5 (3), electron@33.4.11 (7), extract-zip@2.0.1 (2), golang.org/x/text@v0.38.0 (1).
  • Mutation on that same real report: re-pointing one genuine finding at /packages/typescript/package.json exits 1 and names the path. Checked without a pipe.
  • bunx --bun pickier . 30 warnings / 0 errors (unchanged), bun run typecheck clean, bun run verify:docs valid, bun test scripts/ 164 pass / 0 fail.

Note

The SBOM workflow does not run on pull requests, so this gate's first live execution is the push to main after merge. The classifier itself is covered on every PR by the unit tests, and I verified it against the real report locally.

5 changed files on the files tab, with 0 review threads.

Sign in to comment on this pull request.