ops(registry): deploy, observe, and rehearse publish-time malware scanning

Goal

Make publish-time scanning an operable production control, not only an API code path.

Deployment

• Run a pinned ClamAV/clamd service beside the registry with freshclam signature updates.
• Keep the daemon private to the host/network namespace and set resource limits.
• Configure scanner host/port, timeout, maximum bytes, and required/fail-closed mode.
• Add health/readiness reporting that distinguishes process health from scanner readiness.
• Document a staged rollout: shadow/observe in non-production, block EICAR, enforce fail-closed, then monitor.
• Document rollback that does not silently publish unscanned packages.

Observability

• Counters by verdict and publish surface.
• Scan latency and error/timeout counts.
• Signature/database age and scanner readiness.
• Structured audit event containing package/version/commit, publisher identity, artifact hash, verdict, engine, signature, and timestamp.
• Alerts for scanner unavailable, stale definitions, abnormal blocked/review rate, and latency saturation.
• No tokens, tarball bytes, or disclosure text in logs/metrics.

Verification/runbook

• EICAR canary is blocked end-to-end.
• A clean fixture publishes through every route.
• Scanner outage yields 503 and zero new installable artifacts.
• Batch atomicity and retry behavior are rehearsed.
• Operators have steps for triage, false-positive review, definition updates, and safe recovery.

Implementation status (2026-07-28): deployment automation and runbook are ready in the current worktree; production rollout has not been performed and this issue remains open.

Prepared:

• pantry registry setup installs/configures ClamAV on supported apt-based hosts, refreshes signatures, starts clamd/freshclam, writes required scanner environment, and makes the registry service depend on clamd.
• Docker readiness now checks /ready; /health remains liveness-only.
• Operations documentation covers configuration, sidecar/daemon topology, health/metrics, audit events, alert conditions, EICAR rehearsal, rollback, and break-glass behavior.

Before closing:

1. Merge and deploy the implementation.
2. Confirm fresh signatures and ready=true in production.
3. Run clean and EICAR publish rehearsals and verify no object/metadata write occurs for EICAR.
4. Confirm blocked/error alerts and latency dashboards.
5. Restrict or migrate native-binary direct object-storage publishers as tracked in #223.

Repository-side rollout work is now included locally: clamd provisioning and signature refresh, fail-closed/read-attestation flags, a persistent independent staging-signing secret, private object storage policy restricting installable/sealed writes to the Registry role, one-day staging/sealed lifecycle cleanup, observability/readiness endpoints, an operator backfill command, and a deployment/EICAR rehearsal runbook. Remaining work is operational and intentionally not claimed complete here: merge, deploy infrastructure and service, verify role policy, backfill legacy binaries during the documented compatibility window, enable attestation filtering, run clean/EICAR/unavailable-scanner rehearsals, observe metrics/logs, and revoke obsolete builder write credentials.

Rollout implementation landed on main in d0d77f59d. This includes scanner provisioning/configuration, infrastructure write boundaries and cleanup, backfill tooling, documentation, and rehearsal procedures. Leaving this issue open until the actual production deploy, legacy backfill, EICAR/unavailable-scanner rehearsal, observation window, and credential cleanup are confirmed.

Production hardening update (2026-07-29):

• Stream compatibility fix: https://github.com/pantry-pm/pantry/commit/acbdd86c23fac631cf256274a057c0949dd1d2ec
• Serialized backfill capacity: https://github.com/pantry-pm/pantry/commit/9b59f5238662083ecc628ed452157acad047d5b6
• Post-maintenance service recovery: https://github.com/pantry-pm/pantry/commit/95eb83e40878baadce8e6fd33bb4071bacec323a
• Correct package-upgrade detection: https://github.com/pantry-pm/pantry/commit/d66e6caaefe29fe42749122c38a769fc5b4eec3e
• Successful deployment and EICAR rehearsal: https://github.com/pantry-pm/pantry/actions/runs/30427812927
• Full serialized backfill in progress: https://github.com/pantry-pm/pantry/actions/runs/30427892727

Live recovery was rehearsed by stopping pantry-registry and invoking the managed recovery unit; ClamAV, the Registry, and scanner-backed /ready all returned healthy. Strict legacy attestation enforcement remains staged until all 32 shards and a zero-gap dry run succeed.

Rollout update (2026-07-29):

• 030c2344b now sends Connection: close on every operator rescan request, preventing long catalog passes from accumulating proxy-to-registry CLOSE_WAIT sockets.
• 3aa793bb5 locks the one-shard/two-stream production capacity and explicit HTTP teardown into the publication-boundary contract test.
• Targeted retry run https://github.com/pantry-pm/pantry/actions/runs/30445678026 cleanly attested the prior final-shard failure (apollographql.com/rover@0.37.2).
• Production deploy run https://github.com/pantry-pm/pantry/actions/runs/30446175977 succeeded, including readiness and the EICAR rejection rehearsal. Current host evidence: registry + clamd active, scanner ready/required, two bounded clamd streams under load, zero registry restarts, zero CLOSE_WAIT sockets.
• Full reconciliation https://github.com/pantry-pm/pantry/actions/runs/30446132818 is draining the new multi-version build tail. One shard received HTTP 502 during the deliberate deploy restart; targeted retry https://github.com/pantry-pm/pantry/actions/runs/30446336920 is queued behind the serialized full run.

Strict legacy-attestation enforcement remains intentionally disabled until the stable-deployment retry and zero-gap dry run both succeed.

Latest rollout checkpoint (2026-07-29):

• 97a3c2577 restores missing metadata from a structurally valid, digest-bound durable clean sidecar; absent/malformed/mismatched evidence still goes through ClamAV.
• Production deploy 30451901764 passed scanner readiness and the EICAR rejection rehearsal.
• The Azure CLI 2.83 large-artifact regression retry 30451160482 completed cleanly in 4m21s, proving the 300-second proxy timeout remains connected beyond the former 60-second failure.
• The obsolete reconciliation run was canceled only after the latest deployment; it already had restart/old-timeout failures and could never become completion evidence. Its durable sidecars remain useful.
• Authoritative latest-main reconciliation 30452374832 is now running across all 32 catalog shards followed by direct object-storage reconciliation.

Strict legacy read enforcement remains staged until this actual run and a subsequent --require-complete dry run both prove zero gaps.

Migration reliability follow-up:

• 9fcc47a3a expands full catalog reconciliation from 32 to the GitHub matrix maximum of 256 shards while preserving max-parallel: 1 and the two-stream ClamAV limit. This bounds each hosted-runner job and makes retries much smaller without increasing production scanner load.
• The prior latest-main run was stopped after its completed scans had written durable sidecars. Replacement authoritative run 30454140602 is on the new shard plan; discovery and shard 0 have already succeeded with no failures.

Backfill throughput hardening (2026-07-29):

• 5e3b0e7ab moves only retained-byte scanning outside the per-domain metadata lock. Before applying any clean/block verdict it re-reads and verifies the tarball key, SHA-256, size, and S3 version/ETag under the lock; object replacement returns BINARY_RESCAN_ARTIFACT_CHANGED instead of persisting stale evidence.
• e39e3e8f3 feeds a global queue capped at exactly two artifacts, allowing a single large domain to use both provisioned ClamAV threads without raising the tested stream budget.
• Concurrency and replacement-race tests pass, as do the focused 23-test suite and repository typecheck.
• Production deploy 30459817747 passed readiness and EICAR.
• Latest-main reconciliation 30460037092 resumes from all durable sidecars produced by earlier runs.

Production rollout update (2026-07-29): the first latest-main reconciliation run 30460037092 proved that two sustained retained-artifact scan streams could starve the shared registry host: the scan requests continued, but /health, /ready, and the public site received empty upstream replies. I cancelled the run, restored production through deploy 30462249700, and verified health/readiness plus the EICAR canary.

Commit 08ef2db7f (Refs #220, Refs #224) now defaults the reusable backfill tool to one artifact worker, pins production reconciliation to one worker, reserves the second clamd thread for live publishes/readiness, and retains an explicitly bounded PANTRY_BACKFILL_SCAN_CONCURRENCY=1..16 override for dedicated scanner hosts. Focused boundary/backfill tests (10) and full TypeScript typecheck pass. Reconciliation will restart after this commit deploy gate and will reuse the durable clean sidecars already written.

Production load-hardening update (2026-07-29):

• 6e7851201 (Refs #220, Refs #224) installs reusable clamd systemd resource priorities (Nice=10, CPUWeight=25, IOWeight=25) so CPU/I/O-heavy archive scans yield to the Registry and reverse proxy.
• 771b4bb0e (Refs #220, Refs #224) gives readiness probes a separate five-second fail-closed timeout while retaining the 240-second artifact scan budget, and applies both the probe setting and capacity priorities to hosted deploys and self-host provisioning.
• Deploy 30464558929 passed readiness and the EICAR publish canary; CI 30464559069 is fully green.
• Under the known heavy reconciliation load, public /health probes remained 200 at roughly 0.54s. When clamd was fully occupied, /ready returned a structured fail-closed 503 in 5.63s (scanner timed out after 5000ms) instead of the former empty upstream connection, then automatically recovered to 200/ready between scans.
• Authoritative latest-main reconciliation 30464724527 is active. Shards 0–6 completed successfully; shard 7 is scanning. At this checkpoint the production process recorded 345 clean verdicts, one expected EICAR block, zero review verdicts, and zero scanner errors.

Production liveness follow-up

• During actual reconciliation run 30464724527, a CPU-heavy retained artifact scan made /health, /ready, and the security metrics endpoint return empty upstream responses for about 12 seconds. The run was canceled immediately; durable clean attestations from completed work remain reusable.
• A restore deploy on current main completed successfully in 30475668696.
• Commit 7b3a90695 (fix(ops): isolate scanner cpu capacity) now reserves CPU 0 for rpx/Registry, constrains clamd to CPUs 1..N-1, retains lower CPU/I/O scheduling weights, lowers clamd OOM preference, installs the drop-in atomically, and verifies the effective affinity during both managed deploys and reusable pantry registry setup provisioning.
• The production deploy and EICAR rehearsal passed in 30475951137.
• Actual reconciliation restarted on that exact commit in 30476048187. Shard 8 is now exercising the previously problematic workload while /health and /ready are probed continuously. Strict read-side attestation enforcement remains off until actual reconciliation plus a latest-main dry --require-complete proof both succeed.

Serving-loop isolation deployed

• Reconciliation 30476048187 was canceled after dense probes observed /health at 7.52s despite CPU affinity.
• Adding idle CPU/I/O scheduling in b2cf85472 reduced ordinary contention, but reconciliation 30476806318 still caused simultaneous 11.6–12.0s liveness/readiness timeouts. That isolated the remaining fault to the Registry in-process S3→SHA-256→clamd relay rather than clamd scheduling.
• 3ac7fd652 now performs native object scans in a separate low-priority worker. The worker receives a short-lived read-only object URL plus the expected size/digest, streams directly to clamd, and returns a bounded verdict. Registry remains the only authority that revalidates identity and promotes, attests, or quarantines.
• 559b90081 provisions the worker dependency and extends deployment with a real native EICAR staging/upload/completion rehearsal.
• Production deploy 30477692569 passed both package and isolated-native EICAR fail-closed rehearsals.
• Actual reconciliation is now running on the exact deployed commit in 30477791344, with dense liveness probes through the previously problematic shard. Strict read-side enforcement remains off pending actual completion and a latest-main dry --require-complete proof.

Memory-contained reconciliation evidence

• 02b99cee5 adds verified clamd cgroup containment: MemoryHigh=55%, MemoryMax=65%, and MemorySwapMax=0 in managed deploys and reusable CLI provisioning.
• Deploy 30478267517 confirmed the 2-CPU production host is enforcing memory_max=2603532288 and swap_max=0; both package and isolated-native EICAR rehearsals passed.
• Actual reconciliation 30478400295 has passed the previously failing shard-8 workload so far: 62 native scans are clean plus one expected native EICAR block, the longest scan is 139.4s, and dense /health probes remained fast HTTP 200 without the prior host-wide stalls. During scanner saturation, /ready correctly returned structured HTTP 503 after its independent five-second bound and recovered automatically.
• The run remains authoritative and is continuing serially. Durable attestations make it restart-safe. Strict read-side enforcement remains off until this run and a latest-main dry --require-complete proof both complete successfully.

Off-host retained-artifact scanning smoke proof

The production-host backfill path is now disabled. Legacy artifacts are prepared by Registry as exact-object, short-lived read-only URLs; a hosted runner streams those bytes through its own bounded ClamAV daemon; Registry then re-loads and compares tarball, SHA-256, size, and object identity before it alone writes the durable attestation or quarantine state. External evidence is restricted to the temporary legacy cutoff and requires current clean/blocked ClamAV metadata.

Targeted proof: Backfill run 30481702653 on nx.dev@18.3.5, the artifact family involved in the earlier host-pressure incident. All four platform archives were scanned off-host and attested clean with ClamAV 1.5.3 / database 28076. Scan durations were 20.5s, 24.8s, 6.8s, and 6.2s, with exact SHA-256 evidence. Ten production /health probes immediately after the run remained HTTP 200 at 0.55–0.73s.

Implementation landed on main in small linked commits through c2f08d81e, including 3a4b7a55d (off-host legacy scanning), 19c76ed96 (Debian Unix socket transport), b373629ab (real-file readiness probe), 4fc7d6bf6 (identity preflight), 0a84b54d0 (identity binding), and c2f08d81e (local engine/database identity).

The authoritative 64-shard retained-object run is the next gate; strict attestation enforcement remains disabled until it and a zero-gap --require-complete dry proof succeed.

Legacy checksum repair is implemented and proven in production.

• Commit: bea9068 (fix(registry): repair scanned legacy digests)
• Deploy: https://github.com/pantry-pm/pantry/actions/runs/30486580622 (success; readiness + EICAR verification passed)
• Exact retained-artifact proof: https://github.com/pantry-pm/pantry/actions/runs/30486659012 (success)
• rucio.cern.ch/rucio-client@40.4.0: all four platform archives scanned clean with ClamAV 1.5.3 / database 28076 and were durably attested. Observed digests: 529fc457…, bce3ff04…, 9d73ab5a…, 205bc2d7….
• Production /health and /ready: HTTP 200 throughout the off-host scan (sampled ~0.54–0.72s).

The repair capability is cutoff-gated to the legacy runner. New/live publish scans remain strict and fail closed on any declared digest mismatch. The Registry rechecks tarball key, size, and stable object identity under the domain lock before replacing legacy digest/checksum metadata with the clean scan’s observed digest.

Next: idempotence rerun, full retained inventory reconciliation, zero-gap proof, then strict enforcement.

Oversized retained-artifact reconciliation is now proven on production data.

Implementation:

• 2c6dd94e8 (fix(ops): scan oversized legacy archives) — raises only the cutoff-gated legacy preparation bound to 8 GiB; streams oversized gzip tar entries through ClamAV below its 2 GiB per-file limit; overlaps segments at boundaries; preserves whole-archive size/SHA-256 verification; and maps ClamAV coverage limits to fail-closed errors instead of malware quarantines.
• 9d63a80de (fix(ops): bound hosted archive traversal) — aligns hosted runners with the bounded production traversal policy (MaxFiles=100000, MaxRecursion=30).

Real-data proof:

• Initial run 30502425595 cleanly scanned and attested the 2,977,871,586-byte Linux arm64 archive (25,346 entries / 5,034,178,658 unpacked bytes) and the 2,861,917,988-byte Linux x86-64 archive (25,181 entries / 5,175,377,705 unpacked bytes). It then failed closed on the hosted daemon's former 10,000-file cap for Darwin x86-64.
• Follow-up run 30503037389 succeeded and attested Darwin x86-64 after the bounded traversal fix.
• All four openai.com/whisper@20250625.0.0 platform records now have clean ClamAV 1.5.3/database 28076 attestations whose artifact SHA-256 exactly matches metadata.
• Idempotence run 30503353230 completed in 44 seconds with openai.com/whisper: no unattested artifacts.

Production health/readiness remained HTTP 200 throughout. Next: latest-main full catalog reconciliation, dry --require-complete proof, then strict enforcement cutover.

Long-running retained-archive scans are now proven under a bounded migration-only timeout.

• Full run 30503470109 durably completed 99 shards, then failed closed in three shard-22 jobs on Heuristics.Limits.Exceeded.MaxScanTime for .NET SDK archives. No artifact was attested from an incomplete scan.
• 653fee78c (fix(ops): bound long legacy scans) raises only the hosted backfill daemon to a 30-minute engine limit and its client to 35 minutes. The production live-publish scanner remains at 60 seconds.
• Targeted proof 30504579563 completed successfully in 24m15s for all four dotnet.microsoft.com@10.0.200 platforms. Clean scan durations were 603,554–796,347 ms, confirming the former 600,000 ms cutoff was the blocker.
• All four metadata records now carry ClamAV 1.5.3/database 28076 clean attestations whose artifact SHA-256 exactly matches the platform SHA-256.
• Idempotence run 30505766942 completed in 47 seconds with dotnet.microsoft.com: no unattested artifacts.
• Deploy, CI, and Health Monitor for 653fee78c succeeded; production health/readiness stayed HTTP 200.

Next: restart latest-main full actual reconciliation, then run the dry --require-complete zero-gap proof.

Reusable zero-gap audit mode is now on main in 528337ff4 (feat(ops): add retained-artifact audit mode).

The backfill workflow now accepts dry_run=true + reconcile_only=true, skips all 256 catalog workers, and runs the retained-object --require-complete proof directly. It rejects mutation mode and incompatible targeted/quarantine inputs. The workflow contract and runbook are covered by 13 malware-backfill tests, 6 publication-boundary tests, YAML parsing, and git diff --check.

Refs #220 Refs #224

Live resource-isolation incident and remediation (2026-07-30)

During authoritative reconciliation, the unrelated package publisher run 30529324673 submitted the newly updated llama.cpp artifact. Its isolated Bun scanner worker reached ~500 MiB RSS inside the Registry service cgroup, filled that cgroup's 512 MiB swap allowance, and put both worker and serving process into mem_cgroup_handle_over_high. Public health returned an empty reply, while scan mutations stayed fail-closed. Four backfill workers exhausted their bounded five-minute retry window; no unscanned artifact was promoted. Killing the exact scanner child immediately restored health and dropped the Registry cgroup from ~550 MiB + 512 MiB swap to ~130 MiB + 0 swap.

Remediation is on main in 1afe07e42 (fix(registry): isolate scan worker resources):

• signed-object workers run in collected transient systemd services outside the Registry cgroup;
• each worker has 192 MiB reclaim, 256 MiB hard memory, zero swap, TasksMax=32, idle I/O, and low CPU/I/O weights;
• systemd enforces the 270-second whole-control-group deadline even if the parent event loop is pressured;
• only allowlisted non-secret scanner environment is forwarded;
• deploy and pantry registry setup install the same reusable isolation settings;
• the one-minute recovery unit now probes local /health three times and restarts an active-but-unresponsive Registry instead of only calling systemctl start;
• 29 focused scanner/boundary tests, Registry typecheck, production bundle build, YAML parsing, and a real transient-unit host probe pass.

Exact production deploy 30530413384 succeeded; /health and /ready are 200. The exact failed publisher batch is being rerun under isolation, and the four migration shards will be rerun idempotently before the zero-gap proof.

Refs #220 Refs #224

Production isolation proof passed

The exact publisher batch that triggered the pre-fix resource stall was rerun as attempt 2 of run 30529324673 after deploy 1afe07e42; all three platform jobs succeeded.

While the three signed-object scans ran concurrently, production showed three separate pantry-scanner-*.service units. Each applied:

• MemoryHigh=201326592 (192 MiB)
• MemoryMax=268435456 (256 MiB)
• MemorySwapMax=0
• RuntimeMaxUSec=4min 30s
• whole-control-group collection after completion

The Registry service stayed at ~139–147 MiB with zero swap and /ready remained 200. Production logged clean verdicts for all updated llama.cpp and CLIProxyAPI artifacts, the transient units were automatically collected, and no worker leaked into the Registry cgroup. Exact-main CI 30530413492 also passed on retry, including the service-install job affected by the original outage.

Refs #220 Refs #224

High-file-count retained archive support

Authoritative run 30521434475, job backfill (shard:8/32:1/8) (90802548242), safely failed closed on elizaOS.github.io@1.7.1: ClamAV 1.5.3/database 28077 returned Heuristics.Limits.Exceeded.MaxFiles after 983,731 ms because the historical vendored dependency tree exceeded the hosted runner's 100,000-file ceiling. No attestation or installable mutation was accepted.

Commit a5678339a (fix(ops): resume high-file-count scans) raises only ephemeral reconciliation runners to a one-million-file ceiling. The existing 8 GiB scan-size, 30-minute engine, 35-minute client, and six-hour job bounds remain; production publish scanning keeps its stricter 100,000-file limit.

The same commit adds a reusable catalog_shards workflow input for exact, validated 32x8 shard resumes. It rejects malformed/mixed inputs, uses the idempotent quarantine-aware path, and intentionally skips whole-storage reconciliation; the final reconcile_only dry proof remains the authority. 19 focused backfill/boundary tests, workflow YAML parsing, valid/invalid selector checks, and git diff --check pass.

The final resume will include every failed shard from the completed authoritative run, including shard:8/32:1/8.

Refs #220 Refs #224

Follow-up coverage boundary: job backfill (shard:8/32:3/8) (90802548318) safely failed closed on elizaOS.github.io@2.0.1. The whole-archive scan required the existing MaxFileSize per-entry fallback; that fallback scanned 100,000 regular files in ~17 minutes and then hit Pantry's own 100,000-entry parser cap.

56927df14 (fix(ops): deep-scan vendored archives) aligns the hosted streaming fallback with the one-million-entry reconciliation ceiling. It still scans every regular member individually, never extracts package contents to disk, caps declared unpacked bytes at 64 GiB, and retains the six-hour hosted job bound. Production limits are unchanged. The focused 19-test backfill/boundary suite and git diff --check pass. This exact shard will be included in the validated resume set.

Refs #220 Refs #224

Current-main reconciliation recovery

The first authoritative retained-artifact run (30521434475, head 1d326b906) produced 61 successful shards and six preserved failures before cancellation:

• four shards failed only while the Registry was unavailable during the pre-isolation memory incident;
• two high-file-count elizaOS shards hit the old hosted ClamAV/archive-entry coverage limits;
• no failed scan was attested and no blocked artifact became installable.

That run could never reach reconciliation after any matrix failure. Current main now includes the bounded recovery control in 19b34d195 (feat(ops): scale retained scan recovery (#220 #224)): safe default 4, explicitly selectable 8/16 tiers, one local ClamAV worker per hosted job, readiness/capacity runbook, and boundary coverage.

Validation:

• focused backfill tests: 13/13
• publication-boundary tests: 6/6
• exact-main CI 30535119195: success
• exact-main deploy 30535119209: success
• production during 16-job load: /health 200, /ready 200, scanner required/ready, Registry ~135 MiB, zero swap

The idempotent full current-main recovery is run 30535130330. It skips exact current clean attestations, rescans only gaps, uses the corrected one-million-member deep-scan bounds, and will run whole-storage reconciliation only after every shard succeeds. At this checkpoint, the two earlier outage shards shard:5/32:1/8 and shard:5/32:3/8 have already recovered successfully with clean digest-bound ClamAV 1.5.3/DB 28077 attestations.

Strict enforcement remains intentionally pending until this run and the independent dry --require-complete zero-gap proof both succeed.

Complete ClamAV coverage-limit fallback

Commit 61dfe8fcd (fix(ops): deep-scan every coverage limit (#220 #224)) now routes every explicit whole-archive ClamAV Heuristics.Limits.Exceeded coverage verdict—MaxFileSize, MaxFiles, and MaxScanTime—through the existing bounded per-entry scanner. Malware verdicts and ordinary scanner errors remain fail-closed. The fallback still:

• scans every regular tar member without extracting the package tree to disk;
• permits at most one million entries and 64 GiB of declared unpacked data;
• verifies the SHA-256 of the original compressed object;
• accepts no clean attestation until all members scan clean.

Verification:

• backfill tests: 13/13 (including all three explicit coverage limits)
• publication-boundary tests: 6/6
• full TypeScript typecheck: success
• exact-main CI 30537518030: success
• exact-main deploy 30537517910: success

Current reconciliation 30535130330 remains failure-free. All four outage-caused shard failures have recovered. Production audit events now include clean SHA-bound ClamAV 1.5.3/database 28077 attestations for the formerly failing elizaOS.github.io@1.7.1 high-file-count family, including individual scans lasting 887–1,089 seconds. The 2.0.1 per-entry scan remains in progress and cannot mutate installable state before complete clean coverage.

Strict enforcement remains gated on successful catalog completion plus the independent dry --require-complete retained-object proof.

Bounded long-scan attestation and observed recovery set

Current reconciliation 30535130330 produced full clean coverage for elizaOS.github.io@2.0.1: 325,791 regular archive members and 5,098,101,739 bytes were deep-scanned with ClamAV 1.5.3/database 28077 in 2,958,594 ms. The old Registry snapshot correctly rejected that result because its external-evidence duration cap was 30 minutes; no attestation or installable mutation was accepted.

Commit 39bce2ef5 (fix(registry): accept bounded long scan evidence (#220 #224)) aligns external evidence with the existing six-hour hosted job:

• scan duration is capped at six hours;
• submission age is capped at six hours plus a 15-minute grace;
• durationMs must be temporally coherent with scannedAt;
• digest, size, object key, and storage identity are still revalidated under the metadata lock immediately before mutation;
• overlong, stale, future-skewed, changed-object, malformed, and non-ClamAV evidence remains rejected.

The same run then observed Heuristics.Limits.Exceeded.MaxScanSize for mysql.com@8.4.7 and mysql.com@8.0.43. The generic Heuristics.Limits.Exceeded.* fallback already on main covers that verdict; a21b2a4db adds the exact observed limit to tests and the runbook.

Validation:

• binary publication tests: 30/30
• backfill tests: 13/13, including MaxFileSize, MaxFiles, MaxScanSize, and MaxScanTime
• publication-boundary tests: 6/6
• full TypeScript typecheck: success
• exact-main CI 30539103516: success
• exact-main deploy 30539103518: success
• post-deploy production /health and /ready: 200/200

Current exact recovery set is shard:8/32:3/8, shard:10/32:5/8, and shard:10/32:7/8; any later failures will be added. Every failed result remained fail-closed. The formerly failing high-file-count shard:8/32:1/8 succeeded after 58 minutes.

Strict enforcement remains gated on current catalog completion, exact latest-main shard recovery, and the independent dry --require-complete retained-object proof.

Current-main recovery update (2026-07-30)

The bounded prepared-object download retry fix is now deployed from 5f7bfd786 (fix(ops): retry retained object downloads (#220) (#224)).

Validation on the exact deployed SHA:

• CI 30542889378: success (lint, typecheck, JS/Zig tests, and continuous release all green).
• Deploy 30542889163: success, including post-deploy verification.
• Local focused verification before push: backfill tests 15/15, publish-boundary tests 6/6, and registry typecheck.

The authoritative catalog pass remains fail-closed. Eight shards have failed so far and will be resumed on this exact current implementation after the pass finishes:

shard:8/32:3/8, shard:10/32:5/8, shard:10/32:7/8, shard:13/32:4/8, shard:13/32:5/8, shard:14/32:2/8, shard:14/32:3/8, shard:14/32:4/8.

The newest failure, job 90846579002, produced many valid clean attestations and then stopped at the next prepared-object fetch with the same raw transient Unable to connect boundary covered by 5f7bfd786. No unscanned artifact was accepted or reconciled.

I will continue collecting every failed shard, resume only that exact set on latest main, and require every resumed job to succeed before the independent dry --require-complete zero-gap proof and strict cutover.

Current-main race hardening shipped in 12c45c4a8 and CI is green: run 30559664931.

The two shard 26 failures in the long catalog run were not a post-cutoff write bypass. A concurrent Publish Changed Packages run clean-scanned and atomically replaced the legacy Hugging Face records after backfill discovery but before prepare/attest. The Registry correctly rejected applying legacy external evidence to those new objects with BINARY_EXTERNAL_ATTESTATION_NOT_LEGACY.

The reusable recovery now:

• preserves structured Registry status/code failures;
• recognizes only the two explicit concurrent replacement codes (BINARY_EXTERNAL_ATTESTATION_NOT_LEGACY, BINARY_RESCAN_ARTIFACT_CHANGED);
• refetches current Registry metadata;
• converges only when every requested platform has a clean verdict whose artifactSha256 exactly matches its current sha256;
• remains fail-closed for partial/stale metadata, unrelated errors, or verification fetch failures;
• handles races both before preparation and after a long external scan.

Verification:

• focused malware-backfill tests: 17 passed / 197 assertions;
• full TypeScript typecheck: passed;
• repository CI, including lint, unit/Zig tests, Redis action contract, and continuous release: passed.

The authoritative audit run 30535130330 is still draining. At the last checkpoint it had 230 completed, 16 running, 11 queued, and the same 12 already-diagnosed failed shards; no new failure class has appeared. Once it finishes, I will resume the exact final failure set on latest main.

A second current-main hardening pass is complete after shard:28/32:0/8 exposed a metadata reconstruction edge case.

Evidence for nodejs.org@25.5.0 darwin-arm64:

• the installable object's metadata had a post-cutoff uploadedAt and no embedded malwareScan;
• the matching .scan.json sidecar existed and contained a clean ClamAV verdict bound to the exact current SHA-256;
• the original publish log shows the Registry scan ran, while the publisher's socket closed before it received completion;
• a later direct metadata rebuild reconstructed the platform from object state but only carried scan data from the prior metadata snapshot, discarding the durable sidecar;
• external backfill then checked the legacy cutoff before consulting durable evidence and correctly refused new external evidence.

Two small fixes are now on main:

1. 03ca9e86c makes prepare consult the exact digest-bound durable sidecar first, so current Registry-published bytes can repair metadata without accepting external evidence past the cutoff. CI 30561202431 and production deploy 30561203530 succeeded.
2. 2c67e1347 makes every metadata-repair workflow restore only structurally valid, digest-bound clean sidecars; reject post-cutoff objects without one; and require every retained object to be attested once strict mode is enabled. CI 30561476389 succeeded.

Regression coverage now includes:

• post-cutoff durable-sidecar recovery through the Registry;
• sidecar recovery during object-state metadata rebuild;
• fail-closed refusal to advertise an unattested post-cutoff object;
• strict-mode refusal of any unattested retained artifact.

Local verification across the Registry publisher, backfill, and metadata-repair suites: 56 tests / 341 assertions, plus full typecheck and lint (0 errors).

The authoritative catalog run is still draining; its recovery set is now 13 exact shards, with this new shard categorized as durable-evidence restoration rather than a malware-scan bypass.

Retained-scan URL lifetime hardening

Recovery run 30563392582 exposed a boundary between the configured scanner limit and the Registry's presigned capability lifetime:

• MySQL whole-archive scans ran for 17–19 minutes and correctly returned Heuristics.Limits.Exceeded.
• The bounded archive-entry fallback then fetched the same identity-bound object again.
• The prepare URL expired after 15 minutes, so the second request failed with HTTP 403 before any attestation or metadata mutation.
• No bytes were accepted without a clean verdict.

Commit 79c9b45be keeps retained-rescan download capabilities bounded to one hour. This exceeds ClamAV's 30-minute MaxScanTime and allows the fallback request while remaining temporary. Registry publication tests pass 31/31; typecheck and targeted lint pass. Production deploy 30568209691 succeeded.

Exact recovery for shard:10/32:5/8 and shard:10/32:7/8 is queued in run 30568381193.

Production completion

Publish-time malware scanning is now an enforced, operable production control.

• Strict mode: PANTRY_REQUIRE_BINARY_SCAN_ATTESTATION=true; no legacy cutoff remains in GitHub variables or /opt/pantry-registry/registry.env.
• Scanner: private ClamAV 1.5.3/database 28077, clamav-freshclam enabled/active, Registry readiness coupled to scanner readiness.
• Timeout ladder: clamd 225s < Registry 240s < Bun 255s < rpx 300s. Isolated workers have a 270s runtime cap, 192/256 MiB memory thresholds, zero swap, 32-task cap, and low CPU/I/O priority; live systemd worker properties match the deployment contract.
• Deploy 30588527008: package EICAR blocked, isolated-binary EICAR blocked, and scanner outage returned retryable 503, wrote no installable artifact, then recovered readiness.
• Clean production proof: den.sh@0.1.0 linux-x86-64 has a clean digest-bound ClamAV attestation, its tarball redirects to an authorized object, its checksum matches cbc0e08c...cfc9ce2, and a ranged download succeeds.
• Blocked production proof: all four Vault Darwin tarball/checksum routes return 404 without redirects; rehearsal package/binary install routes are empty.
• Live audit events contain surface, identity, artifact digest, verdict, engine, timing, and optional signature/reason without artifact bytes or credentials. The mainline publisher has produced clean events and fail-closed timeout/size-integrity errors under strict mode.
• Final retained-object audit 30589326169 passes with no unattested artifacts across every catalog domain.
• Full CI 30587350923 is green, including lint, docs contracts, typecheck, unit/Zig tests, and the pinned Redis service contract.

The final reusable reconciliation/download/quarantine hardening landed in small linked commits: 12c45c4a8, 03ca9e86c, 2c67e1347, 79c9b45be, 335014e84, and f295058ab.

Runbook, rollout/rollback, alerting, readiness, capacity, false-positive review, and recovery contracts are implemented and tested. Refs #224 and closed incident #225.