epic: publish-time malware scanning and dual-use package policy

Outcome

Every package artifact accepted by the Pantry registry/API is automatically malware-scanned before it becomes installable. Malware is blocked, scanner failures fail closed, dual-use packages carry durable disclosure metadata, and operators can observe and rehearse the control.

This tracks parity with npm's July 28, 2026 publish-time scanning behavior: https://github.blog/changelog/2026-07-28-npm-publish-time-malware-scanning-and-dual-use-metadata/

Workstreams

• #222 — scanner abstraction, clamd integration, limits, fail-closed behavior
• #223 — gate core, commit, Zig, and PHP publish routes; atomic batch scanning; persist verdict metadata
• #221 — contentPolicy, root text-only DISCLOSURE, persistence across versions, auth-policy groundwork
• #220 — deployment, fresh signatures, health/readiness, metrics/audit events, alerts, rollout and incident runbook

Security invariants

1. No new installable package bytes are persisted before a clean scan verdict.
2. Scanner timeout, outage, or malformed response cannot degrade into allow.
3. A blocked member prevents an entire multi-package commit batch from being published.
4. Dual-use declarations provide context only; they never override a malware verdict.
5. No package contents, credentials, or disclosure text enter logs/metrics.
6. Tests prove all public API publication paths pass through the gate.
7. Any remaining direct object-storage publication path is treated as a bypass and must be migrated or equivalently gated before rollout completes.

Definition of done

• All child acceptance criteria pass.
• Registry tests, typecheck, lint, and docs contract checks pass.
• Production service runs a private, signature-updated scanner.
• EICAR is rejected end-to-end and leaves no installable artifact.
• Clean fixtures publish through every supported endpoint.
• Scanner outage returns a retryable failure and writes nothing.
• Operator runbook covers false positives, stale signatures, outage, and recovery.
• Release notes explain the publish-time delay and dual-use metadata requirements.

Rollout note

Code completion is not production completion. Keep this epic open until scanner deployment and the EICAR/outage rehearsals are confirmed in the production-equivalent environment.

Progress update (2026-07-28): the fail-closed scanner, dual-use policy, API gates, persistence, tests, deployment automation, and operator documentation are implemented in the current worktree.

Verification completed:

• Registry: 440/440 tests.
• Zig: 731/731 tests.
• TypeScript typechecks and production server bundle pass.
• Documentation contracts, Zig formatting, targeted lint, and git diff checks pass.

The epic remains open until the code is reviewed/merged, #220 is deployed and rehearsed in production, and #223 closes the trusted native-binary direct-object-storage path. This preserves the core invariant honestly: no registry API publish becomes installable without a clean verdict; broader artifact-pipeline enforcement is still outstanding.

Epic implementation update: all logically implementable repository work is complete in the current local worktree. Core, commit, Zig, PHP, native binaries, pkgx fallback, legacy helpers, and storage-layer writes are covered by shared scan-before-commit rules. Native publication uses reusable stage/seal/stream-scan/promote primitives; read-side attestations and infrastructure policy provide defense in depth; dual-use disclosure, tests, docs, backfill, and rollout tooling are included. Verification: registry typecheck; 452/452 registry tests; 731/731 Zig tests; root TypeScript typecheck; server production bundle; documentation contracts; shell syntax; CloudFormation YAML parse; git diff check; and repository lint with no errors. Keep the epic open until review/merge and issue #220 production rollout are complete.

The complete repository implementation is now on main as five small conventional commits: a65b2bda8, 673517231, 02b16e2ff, 70f7a50bf, and d0d77f59d. Issues #221, #222, and #223 are code-complete and closed. Keeping the epic open solely for the production evidence tracked by #220.

Rollout is in its final migration phase. Production now has the variant-stream fix, bounded ClamAV resources, one-at-a-time retained scans, and a rehearsed systemd recovery path for package-maintenance stops. Evidence and the active full backfill are tracked in #220: https://github.com/pantry-pm/pantry/actions/runs/30427892727

Final rollout remains in progress under #220. Latest hardening: 030c2344b closes each backfill HTTP connection and 3aa793bb5 regression-tests the one-shard/two-stream/connection-close capacity contract. Production deploy 30446175977 passed readiness plus EICAR blocking; full reconciliation 30446132818 is actively draining the final moving tail. Strict legacy read enforcement will only be enabled after a zero-gap dry run.

Final migration checkpoint: latest-main production deploy 30451901764 passed readiness/EICAR, and the formerly failing Azure CLI 2.83 scan now succeeds end-to-end in 30451160482. Digest-bound durable attestation restoration landed as 97a3c2577. The authoritative 32-shard plus object-storage reconciliation is running in 30452374832; the epic remains open until its zero-gap proof and strict enforcement deployment complete.

Rollout update: commit bea9068 repaired the legacy checksum edge case without weakening live publishes. Production deploy 30486580622 and exact rucio-client@40.4.0 proof 30486659012 both passed; all four retained archives are now clean and durably attested, with health/readiness unaffected. Full inventory reconciliation and strict cutover remain in progress.

Authoritative retained-object reconciliation is running at https://github.com/pantry-pm/pantry/actions/runs/30487020985 on exact main commit bea90681e. Checkpoint: 33 successful matrix shards, 8 running at the configured max parallelism, 0 failed. Completed attestations are durable. Strict enforcement remains intentionally gated on this run, its independent storage reconciliation, and the subsequent dry --require-complete zero-gap proof.

Oversized retained-artifact reconciliation is now proven on production data.

Implementation:

• 2c6dd94e8 (fix(ops): scan oversized legacy archives) — raises only the cutoff-gated legacy preparation bound to 8 GiB; streams oversized gzip tar entries through ClamAV below its 2 GiB per-file limit; overlaps segments at boundaries; preserves whole-archive size/SHA-256 verification; and maps ClamAV coverage limits to fail-closed errors instead of malware quarantines.
• 9d63a80de (fix(ops): bound hosted archive traversal) — aligns hosted runners with the bounded production traversal policy (MaxFiles=100000, MaxRecursion=30).

Real-data proof:

• Initial run 30502425595 cleanly scanned and attested the 2,977,871,586-byte Linux arm64 archive (25,346 entries / 5,034,178,658 unpacked bytes) and the 2,861,917,988-byte Linux x86-64 archive (25,181 entries / 5,175,377,705 unpacked bytes). It then failed closed on the hosted daemon's former 10,000-file cap for Darwin x86-64.
• Follow-up run 30503037389 succeeded and attested Darwin x86-64 after the bounded traversal fix.
• All four openai.com/whisper@20250625.0.0 platform records now have clean ClamAV 1.5.3/database 28076 attestations whose artifact SHA-256 exactly matches metadata.
• Idempotence run 30503353230 completed in 44 seconds with openai.com/whisper: no unattested artifacts.

Production health/readiness remained HTTP 200 throughout. Next: latest-main full catalog reconciliation, dry --require-complete proof, then strict enforcement cutover.

Long-running retained-archive scans are now proven under a bounded migration-only timeout.

• Full run 30503470109 durably completed 99 shards, then failed closed in three shard-22 jobs on Heuristics.Limits.Exceeded.MaxScanTime for .NET SDK archives. No artifact was attested from an incomplete scan.
• 653fee78c (fix(ops): bound long legacy scans) raises only the hosted backfill daemon to a 30-minute engine limit and its client to 35 minutes. The production live-publish scanner remains at 60 seconds.
• Targeted proof 30504579563 completed successfully in 24m15s for all four dotnet.microsoft.com@10.0.200 platforms. Clean scan durations were 603,554–796,347 ms, confirming the former 600,000 ms cutoff was the blocker.
• All four metadata records now carry ClamAV 1.5.3/database 28076 clean attestations whose artifact SHA-256 exactly matches the platform SHA-256.
• Idempotence run 30505766942 completed in 47 seconds with dotnet.microsoft.com: no unattested artifacts.
• Deploy, CI, and Health Monitor for 653fee78c succeeded; production health/readiness stayed HTTP 200.

Next: restart latest-main full actual reconciliation, then run the dry --require-complete zero-gap proof.

Reusable zero-gap audit mode is now on main in 528337ff4 (feat(ops): add retained-artifact audit mode).

The backfill workflow now accepts dry_run=true + reconcile_only=true, skips all 256 catalog workers, and runs the retained-object --require-complete proof directly. It rejects mutation mode and incompatible targeted/quarantine inputs. The workflow contract and runbook are covered by 13 malware-backfill tests, 6 publication-boundary tests, YAML parsing, and git diff --check.

Refs #220 Refs #224

Live resource-isolation incident and remediation (2026-07-30)

During authoritative reconciliation, the unrelated package publisher run 30529324673 submitted the newly updated llama.cpp artifact. Its isolated Bun scanner worker reached ~500 MiB RSS inside the Registry service cgroup, filled that cgroup's 512 MiB swap allowance, and put both worker and serving process into mem_cgroup_handle_over_high. Public health returned an empty reply, while scan mutations stayed fail-closed. Four backfill workers exhausted their bounded five-minute retry window; no unscanned artifact was promoted. Killing the exact scanner child immediately restored health and dropped the Registry cgroup from ~550 MiB + 512 MiB swap to ~130 MiB + 0 swap.

Remediation is on main in 1afe07e42 (fix(registry): isolate scan worker resources):

• signed-object workers run in collected transient systemd services outside the Registry cgroup;
• each worker has 192 MiB reclaim, 256 MiB hard memory, zero swap, TasksMax=32, idle I/O, and low CPU/I/O weights;
• systemd enforces the 270-second whole-control-group deadline even if the parent event loop is pressured;
• only allowlisted non-secret scanner environment is forwarded;
• deploy and pantry registry setup install the same reusable isolation settings;
• the one-minute recovery unit now probes local /health three times and restarts an active-but-unresponsive Registry instead of only calling systemctl start;
• 29 focused scanner/boundary tests, Registry typecheck, production bundle build, YAML parsing, and a real transient-unit host probe pass.

Exact production deploy 30530413384 succeeded; /health and /ready are 200. The exact failed publisher batch is being rerun under isolation, and the four migration shards will be rerun idempotently before the zero-gap proof.

Refs #220 Refs #224

Production isolation proof passed

The exact publisher batch that triggered the pre-fix resource stall was rerun as attempt 2 of run 30529324673 after deploy 1afe07e42; all three platform jobs succeeded.

While the three signed-object scans ran concurrently, production showed three separate pantry-scanner-*.service units. Each applied:

• MemoryHigh=201326592 (192 MiB)
• MemoryMax=268435456 (256 MiB)
• MemorySwapMax=0
• RuntimeMaxUSec=4min 30s
• whole-control-group collection after completion

The Registry service stayed at ~139–147 MiB with zero swap and /ready remained 200. Production logged clean verdicts for all updated llama.cpp and CLIProxyAPI artifacts, the transient units were automatically collected, and no worker leaked into the Registry cgroup. Exact-main CI 30530413492 also passed on retry, including the service-install job affected by the original outage.

Refs #220 Refs #224

High-file-count retained archive support

Authoritative run 30521434475, job backfill (shard:8/32:1/8) (90802548242), safely failed closed on elizaOS.github.io@1.7.1: ClamAV 1.5.3/database 28077 returned Heuristics.Limits.Exceeded.MaxFiles after 983,731 ms because the historical vendored dependency tree exceeded the hosted runner's 100,000-file ceiling. No attestation or installable mutation was accepted.

Commit a5678339a (fix(ops): resume high-file-count scans) raises only ephemeral reconciliation runners to a one-million-file ceiling. The existing 8 GiB scan-size, 30-minute engine, 35-minute client, and six-hour job bounds remain; production publish scanning keeps its stricter 100,000-file limit.

The same commit adds a reusable catalog_shards workflow input for exact, validated 32x8 shard resumes. It rejects malformed/mixed inputs, uses the idempotent quarantine-aware path, and intentionally skips whole-storage reconciliation; the final reconcile_only dry proof remains the authority. 19 focused backfill/boundary tests, workflow YAML parsing, valid/invalid selector checks, and git diff --check pass.

The final resume will include every failed shard from the completed authoritative run, including shard:8/32:1/8.

Refs #220 Refs #224

Follow-up coverage boundary: job backfill (shard:8/32:3/8) (90802548318) safely failed closed on elizaOS.github.io@2.0.1. The whole-archive scan required the existing MaxFileSize per-entry fallback; that fallback scanned 100,000 regular files in ~17 minutes and then hit Pantry's own 100,000-entry parser cap.

56927df14 (fix(ops): deep-scan vendored archives) aligns the hosted streaming fallback with the one-million-entry reconciliation ceiling. It still scans every regular member individually, never extracts package contents to disk, caps declared unpacked bytes at 64 GiB, and retains the six-hour hosted job bound. Production limits are unchanged. The focused 19-test backfill/boundary suite and git diff --check pass. This exact shard will be included in the validated resume set.

Refs #220 Refs #224

Current-main reconciliation recovery

The first authoritative retained-artifact run (30521434475, head 1d326b906) produced 61 successful shards and six preserved failures before cancellation:

• four shards failed only while the Registry was unavailable during the pre-isolation memory incident;
• two high-file-count elizaOS shards hit the old hosted ClamAV/archive-entry coverage limits;
• no failed scan was attested and no blocked artifact became installable.

That run could never reach reconciliation after any matrix failure. Current main now includes the bounded recovery control in 19b34d195 (feat(ops): scale retained scan recovery (#220 #224)): safe default 4, explicitly selectable 8/16 tiers, one local ClamAV worker per hosted job, readiness/capacity runbook, and boundary coverage.

Validation:

• focused backfill tests: 13/13
• publication-boundary tests: 6/6
• exact-main CI 30535119195: success
• exact-main deploy 30535119209: success
• production during 16-job load: /health 200, /ready 200, scanner required/ready, Registry ~135 MiB, zero swap

The idempotent full current-main recovery is run 30535130330. It skips exact current clean attestations, rescans only gaps, uses the corrected one-million-member deep-scan bounds, and will run whole-storage reconciliation only after every shard succeeds. At this checkpoint, the two earlier outage shards shard:5/32:1/8 and shard:5/32:3/8 have already recovered successfully with clean digest-bound ClamAV 1.5.3/DB 28077 attestations.

Strict enforcement remains intentionally pending until this run and the independent dry --require-complete zero-gap proof both succeed.

Complete ClamAV coverage-limit fallback

Commit 61dfe8fcd (fix(ops): deep-scan every coverage limit (#220 #224)) now routes every explicit whole-archive ClamAV Heuristics.Limits.Exceeded coverage verdict—MaxFileSize, MaxFiles, and MaxScanTime—through the existing bounded per-entry scanner. Malware verdicts and ordinary scanner errors remain fail-closed. The fallback still:

• scans every regular tar member without extracting the package tree to disk;
• permits at most one million entries and 64 GiB of declared unpacked data;
• verifies the SHA-256 of the original compressed object;
• accepts no clean attestation until all members scan clean.

Verification:

• backfill tests: 13/13 (including all three explicit coverage limits)
• publication-boundary tests: 6/6
• full TypeScript typecheck: success
• exact-main CI 30537518030: success
• exact-main deploy 30537517910: success

Current reconciliation 30535130330 remains failure-free. All four outage-caused shard failures have recovered. Production audit events now include clean SHA-bound ClamAV 1.5.3/database 28077 attestations for the formerly failing elizaOS.github.io@1.7.1 high-file-count family, including individual scans lasting 887–1,089 seconds. The 2.0.1 per-entry scan remains in progress and cannot mutate installable state before complete clean coverage.

Strict enforcement remains gated on successful catalog completion plus the independent dry --require-complete retained-object proof.

Bounded long-scan attestation and observed recovery set

Current reconciliation 30535130330 produced full clean coverage for elizaOS.github.io@2.0.1: 325,791 regular archive members and 5,098,101,739 bytes were deep-scanned with ClamAV 1.5.3/database 28077 in 2,958,594 ms. The old Registry snapshot correctly rejected that result because its external-evidence duration cap was 30 minutes; no attestation or installable mutation was accepted.

Commit 39bce2ef5 (fix(registry): accept bounded long scan evidence (#220 #224)) aligns external evidence with the existing six-hour hosted job:

• scan duration is capped at six hours;
• submission age is capped at six hours plus a 15-minute grace;
• durationMs must be temporally coherent with scannedAt;
• digest, size, object key, and storage identity are still revalidated under the metadata lock immediately before mutation;
• overlong, stale, future-skewed, changed-object, malformed, and non-ClamAV evidence remains rejected.

The same run then observed Heuristics.Limits.Exceeded.MaxScanSize for mysql.com@8.4.7 and mysql.com@8.0.43. The generic Heuristics.Limits.Exceeded.* fallback already on main covers that verdict; a21b2a4db adds the exact observed limit to tests and the runbook.

Validation:

• binary publication tests: 30/30
• backfill tests: 13/13, including MaxFileSize, MaxFiles, MaxScanSize, and MaxScanTime
• publication-boundary tests: 6/6
• full TypeScript typecheck: success
• exact-main CI 30539103516: success
• exact-main deploy 30539103518: success
• post-deploy production /health and /ready: 200/200

Current exact recovery set is shard:8/32:3/8, shard:10/32:5/8, and shard:10/32:7/8; any later failures will be added. Every failed result remained fail-closed. The formerly failing high-file-count shard:8/32:1/8 succeeded after 58 minutes.

Strict enforcement remains gated on current catalog completion, exact latest-main shard recovery, and the independent dry --require-complete retained-object proof.

Current-main recovery update (2026-07-30)

The bounded prepared-object download retry fix is now deployed from 5f7bfd786 (fix(ops): retry retained object downloads (#220) (#224)).

Validation on the exact deployed SHA:

• CI 30542889378: success (lint, typecheck, JS/Zig tests, and continuous release all green).
• Deploy 30542889163: success, including post-deploy verification.
• Local focused verification before push: backfill tests 15/15, publish-boundary tests 6/6, and registry typecheck.

The authoritative catalog pass remains fail-closed. Eight shards have failed so far and will be resumed on this exact current implementation after the pass finishes:

shard:8/32:3/8, shard:10/32:5/8, shard:10/32:7/8, shard:13/32:4/8, shard:13/32:5/8, shard:14/32:2/8, shard:14/32:3/8, shard:14/32:4/8.

The newest failure, job 90846579002, produced many valid clean attestations and then stopped at the next prepared-object fetch with the same raw transient Unable to connect boundary covered by 5f7bfd786. No unscanned artifact was accepted or reconciled.

I will continue collecting every failed shard, resume only that exact set on latest main, and require every resumed job to succeed before the independent dry --require-complete zero-gap proof and strict cutover.

Current-main race hardening shipped in 12c45c4a8 and CI is green: run 30559664931.

The two shard 26 failures in the long catalog run were not a post-cutoff write bypass. A concurrent Publish Changed Packages run clean-scanned and atomically replaced the legacy Hugging Face records after backfill discovery but before prepare/attest. The Registry correctly rejected applying legacy external evidence to those new objects with BINARY_EXTERNAL_ATTESTATION_NOT_LEGACY.

The reusable recovery now:

• preserves structured Registry status/code failures;
• recognizes only the two explicit concurrent replacement codes (BINARY_EXTERNAL_ATTESTATION_NOT_LEGACY, BINARY_RESCAN_ARTIFACT_CHANGED);
• refetches current Registry metadata;
• converges only when every requested platform has a clean verdict whose artifactSha256 exactly matches its current sha256;
• remains fail-closed for partial/stale metadata, unrelated errors, or verification fetch failures;
• handles races both before preparation and after a long external scan.

Verification:

• focused malware-backfill tests: 17 passed / 197 assertions;
• full TypeScript typecheck: passed;
• repository CI, including lint, unit/Zig tests, Redis action contract, and continuous release: passed.

The authoritative audit run 30535130330 is still draining. At the last checkpoint it had 230 completed, 16 running, 11 queued, and the same 12 already-diagnosed failed shards; no new failure class has appeared. Once it finishes, I will resume the exact final failure set on latest main.

A second current-main hardening pass is complete after shard:28/32:0/8 exposed a metadata reconstruction edge case.

Evidence for nodejs.org@25.5.0 darwin-arm64:

• the installable object's metadata had a post-cutoff uploadedAt and no embedded malwareScan;
• the matching .scan.json sidecar existed and contained a clean ClamAV verdict bound to the exact current SHA-256;
• the original publish log shows the Registry scan ran, while the publisher's socket closed before it received completion;
• a later direct metadata rebuild reconstructed the platform from object state but only carried scan data from the prior metadata snapshot, discarding the durable sidecar;
• external backfill then checked the legacy cutoff before consulting durable evidence and correctly refused new external evidence.

Two small fixes are now on main:

1. 03ca9e86c makes prepare consult the exact digest-bound durable sidecar first, so current Registry-published bytes can repair metadata without accepting external evidence past the cutoff. CI 30561202431 and production deploy 30561203530 succeeded.
2. 2c67e1347 makes every metadata-repair workflow restore only structurally valid, digest-bound clean sidecars; reject post-cutoff objects without one; and require every retained object to be attested once strict mode is enabled. CI 30561476389 succeeded.

Regression coverage now includes:

• post-cutoff durable-sidecar recovery through the Registry;
• sidecar recovery during object-state metadata rebuild;
• fail-closed refusal to advertise an unattested post-cutoff object;
• strict-mode refusal of any unattested retained artifact.

Local verification across the Registry publisher, backfill, and metadata-repair suites: 56 tests / 341 assertions, plus full typecheck and lint (0 errors).

The authoritative catalog run is still draining; its recovery set is now 13 exact shards, with this new shard categorized as durable-evidence restoration rather than a malware-scan bypass.

Retained-scan URL lifetime hardening

Recovery run 30563392582 exposed a boundary between the configured scanner limit and the Registry's presigned capability lifetime:

• MySQL whole-archive scans ran for 17–19 minutes and correctly returned Heuristics.Limits.Exceeded.
• The bounded archive-entry fallback then fetched the same identity-bound object again.
• The prepare URL expired after 15 minutes, so the second request failed with HTTP 403 before any attestation or metadata mutation.
• No bytes were accepted without a clean verdict.

Commit 79c9b45be keeps retained-rescan download capabilities bounded to one hour. This exceeds ClamAV's 30-minute MaxScanTime and allows the fallback request while remaining temporary. Registry publication tests pass 31/31; typecheck and targeted lint pass. Production deploy 30568209691 succeeded.

Exact recovery for shard:10/32:5/8 and shard:10/32:7/8 is queued in run 30568381193.

Epic complete

Every tracked workstream and production gate is complete.

• Child issues #221, #222, #223, and #220 are closed; quarantine review/release support #226 is closed; the Vault security incident #225 is closed with fail-closed containment.
• Every supported package, commit-batch, Zig, PHP, and native-binary publication boundary passes through the reusable scan gate before installable persistence. Batch rejection is atomic; errors, malformed responses, timeouts, and outages fail closed.
• Dual-use contentPolicy/root DISCLOSURE, durable malware attestations, private quarantine evidence, exact-digest review, and metadata-authorized download enforcement are implemented.
• Strict production enforcement has no compatibility cutoff. Deploy 30588527008 proves EICAR rejection on package and binary surfaces plus retryable outage/zero-write recovery.
• Production ClamAV is private, signature-updated, resource-isolated, readiness-gated, observable, and covered by runbook/rollout/rollback/release documentation.
• Audit 30589326169 proves no retained installable artifact lacks an attestation across the entire catalog after strict rollout.
• CI 30587350923 is fully green.

Final linked hardening commits on main: 12c45c4a8, 03ca9e86c, 2c67e1347, 79c9b45be, 335014e84, and f295058ab. All were pushed directly to origin/main as small conventional commits, as requested.