ReviewOS

zig-utils/zig-js

Memory: reconcile owned bytes across allocator, heap, and native-code domains

#974
Closed chris opened this 9 days ago ·
9 days ago

Parent: #480 Depends on: #461

Goal

Publish a versioned, allocation-free Context memory snapshot whose non-overlapping ownership domains reconcile exactly, including shared-realm ownership and executable mappings, without presenting logical GC subsets as additional allocated bytes.

Scope

  • Report current and peak Context-allocator backing from the opt-in runtime attribution profile.
  • Report owned native live/retired mappings separately from shared mappings merely observed by a sibling realm.
  • Expose heap logical live bytes, net issued GC-cell bytes, managed string/ArrayBuffer/object/generator/environment side-storage bytes, and Promise reaction entries as explicitly overlapping diagnostic subsets.
  • Record whether this realm owns the precise heap and JIT owner; never double count shared storage.
  • Preserve heap-budget data and identify allocator domains outside the reconciled subtotal, including collector auxiliary scratch and profiler/control allocations.
  • Add a teardown checkpoint that proves every tracked owned domain returns to zero after finalizers, code retirement, and allocator wrapper destruction.

Acceptance

  • Schema version and coverage flags make partial accounting fail visible.
  • Checked arithmetic proves each published subtotal and net counter.
  • Private arena, private precise-GC, budgeted, JIT, shared-realm, and teardown fixtures reconcile exact ownership without allocations in the snapshot path.
  • Shared heap/code bytes appear once as owned and only as observed in sibling snapshots.
  • The disabled production path allocates no profiling state and changes no hot counter path.
  • Full unit, moving-GC, no-GIL/TSan, ABI, and code-retirement gates pass.

No-workaround rules

  • No RSS estimate substituted for owned bytes, allocator capacity inferred from logical live cells, shared mapping counted per realm, saturating arithmetic hiding counter corruption, or opaque "other" bucket presented as complete accounting.
9 days ago

First inventory slice landed on main in 77b38325.

Context.memoryInventorySnapshot() is opt-in through the existing attribution profile and publishes schema v1 with a checked, non-overlapping subtotal for Context allocator backing, profiler/budget/serialized controls, live recovery reserve, and owned live/retired native mappings. Logical heap/cell/string/ArrayBuffer/environment/object/generator/Promise fields are labeled diagnostic overlaps. Shared precise realms report heap/JIT ownership explicitly and never add observed shared mappings to their owned subtotal.

Coverage fails closed: a precise heap owner sets accounted_owned_bytes_complete = false and collector_auxiliary_owned_but_untracked = true, because zig-gc auxiliary scratch still bypasses the Context allocator. That is the next concrete gap; this commit does not claim #974 complete.

Validation: 2,482 full unit tests passed, 1 expected skip, 0 failures/leaks; focused inventory normal and TSan tests passed; the existing JIT attribution fixture reconciles mapped code bytes; focused frontend passed 79/79; production library build passed. No JavaScript evaluation path changed.

9 days ago

Two more slices landed on main:

  • 198f78d6 wraps zig-gc auxiliary worklists in attribution mode and records their allocation/growth/release/current/peak requested bytes as a distinct domain. It bypasses the public heap budget and existing Context allocation replay counters, preserving both semantics. Inventory schema v2 now includes current scratch in its checked subtotal and can mark precise-owner checkpoints complete.
  • efee87e6 adds destroyWithMemoryInventory(). After finalizers, heap/slab teardown, code retirement, arena release, recovery-reserve release, and control destruction, it returns a versioned checkpoint proving every tracked current domain is zero. Ordinary destroy() does not take the reporting snapshot.

Validation after the teardown refactor: 2,483 full units passed, 1 expected skip, 0 failures/leaks; the focused inventory/teardown fixture passes under normal and TSan. #974 remains open for finer owner/category partitions, shared-realm teardown reporting, and the broader no-GIL/ABI/code-retirement gate set.

9 days ago

Pushed dcfbc7ff (feat(memory): emit inventory in attribution rows (#974)). The benchmark runner now emits schema-v3 memory inventory records for configuration, warmup, and invocation. Matrix V40 versions the sidecar as schema 14, classifies the inventory as diagnostic/non-scored, and validates exact top-level ownership, generation partitions, and budget reconciliation fail-closed. Evidence: matrix structural tests 77/77, tier-attribution self-test, live 3/3 phase JSON validation, and a 50-page docs build/manifest check. The umbrella harness passed every affected validator before reaching the existing Home evaluator step-budget limit.

9 days ago

Completed on main through dcfbc7ff.

Acceptance evidence:

  • schema v3 and fail-closed coverage flags; checked exact reconciliation for Context backing, collector scratch, controls/recovery reserve, and owned native mappings;
  • arena, precise-GC, budgeted, JIT, sibling shared-realm, owner/sibling teardown, and generation-partition fixtures;
  • ordinary disabled snapshots remain null and allocate no profiler state;
  • benchmark attribution emits configuration/warmup/invocation records under matrix V40 and sidecar schema 14, explicitly diagnostic and non-scored;
  • full units: 2,483 passed, 1 expected skip, 0 failed, 0 leaked;
  • TSan no-GIL parallel_js: 74 passed, 0 failed/leaked, including shared moving GC, budget recovery, code invalidation, and lifecycle cases;
  • focused JIT/retirement: 430 passed, 1 platform skip, 0 failed/leaked;
  • relocation inventory: 9 cell kinds, 27 pointer surfaces, 21 boundary tags;
  • public ABI audit: 117/117 declarations complete, all symbols present; C/C++ host fixtures passed;
  • matrix structural tests 77/77, tier-attribution reconciliation self-test, live 3/3 phase JSON validation, and 50-page docs build/manifest check.

The umbrella benchmark harness separately retains the pre-existing Home evaluator step-budget limitation tracked by #614; all #974-affected validators passed before that unrelated limit.

Sign in to comment on this issue.