Shapes: scale shared transition publication beyond two lanes

Parent: #478 Split from: #530 Related: #461, #480, #493

Problem

#530 replaces deep linear lookup and predictable transition hashing with collision-free persistent AVL indexes. Its functional four-lane and suppression-free TSan witnesses pass, but a dedicated shared-realm publication diagnostic shows that total transition throughput stops scaling beyond two lanes even when lanes publish beneath independent parent shapes.

The pre-implementation fixture is intentionally split into:

• partitioned base shapes: each lane publishes unique fanout edges beneath its own prepared parent shape in one shared Context;
• anti-specialization variant: different base/key layout with identical logical work and checksum;
• contended control: every lane publishes beneath one parent, preserving the serialized worst case.

Each invocation uses a fresh base-shape generation and reuses one object per lane, deleting back to the base after every unique publication. This removes per-iteration object construction while preserving real transition publication, exact own-key order, deletion, and deterministic checksums.

Scope

• Attribute 1/2/4/8-lane time to Shape transition locks, Context arena allocation, object property locks, GC, and Thread lifecycle using shared attribution.
• Remove the causal independent-parent bottleneck with bounded Context-owned storage and an explicit synchronization design.
• Consider per-thread arena tranches, batched immutable-node allocation, or optimistic persistent-root publication only when exact profiles justify them.
• Preserve same-name convergence, collision-independent exact keys, stable slots, delete/re-add undo, OOM atomicity, moving GC, and Context teardown.
• Keep the same-parent contended row visible; do not describe intentional serialization as scalable.

No-workaround rules

• No GIL, hidden serialization outside the measured lock, prepublishing scored keys, reusing a result cache, reducing logical work at higher lane counts, unbounded per-thread reserves, benchmark-name/source recognition, or excluding the 4/8-lane rows.
• Do not trade exact same-name convergence or OOM rollback for duplicate visible shapes.

Acceptance

• Benchmark source, base/variant/contended identities, jobs, checksums, warmup, and timed boundary are committed before the runtime fix.
• Clean 1/2/4/8-lane order-balanced evidence reports transitions/second, instructions/transition, lock contention/wait, allocation bytes, GC, peak/retained RSS, and dispersion.
• Partitioned total publication throughput is monotonic across 1/2/4/8 lanes and the anti-specialization variant agrees; every row retains identical per-lane work.
• The contended row remains bounded and exact, with wait/serialization attributed rather than hidden.
• Same-name convergence, disjoint fanout, deep delete/re-add, allocation-failure rollback, 32 MiB bounded heap, moving/parallel GC, threadfuzz, and suppression-free TSan pass.
• Shallow classes, mixed application, dense Array, and #530 deep lookup rows do not regress materially; retained memory remains bounded and published.

Security boundary

Property names, fanout, and thread schedules are attacker-controlled. The fix must improve independent publication without reintroducing predictable collision behavior, races, half-published roots, or unbounded thread-local retention.

Benchmark-first commit landed on main as d9d54b5b: bench(shape): add shared transition publication rows (#531).

Frozen rows:

• representative_shape_transition_fanout: one fresh prepared base shape per lane;
• _variant: different base/key layout with identical logical work/checksum;
• _contended: all lanes deliberately share one parent shape.

Each lane reuses one object, publishes a unique edge, validates exact own-key order/value, deletes back to the prepared base, and checks absence. For 1,000 publications/lane the exact 1/2/4/8 checksums are 503,500 / 1,507,500 / 5,017,000 / 18,042,000. The 18/18 benchmark structural gates pass.

No runtime fix is included in this commit. Quick diagnostics confirm the partitioned row stops scaling beyond two lanes; accepted measurements and causal attribution will follow from this frozen parent.

Benchmark boundary correction is now pushed in d30790f.

The committed row no longer charges Reflect.ownKeys or membership scans to each transition publication. Every operation still publishes a unique computed edge, validates the observed value, and validates deletion back to the prepared base shape; exhaustive key-order and absence semantics remain in the production unit suite.

For jobs=1000, the frozen exact checksums are now:

• 1 lane: 501500
• 2 lanes: 1503500
• 4 lanes: 5009000
• 8 lanes: 18026000

A quick parent diagnostic still loses aggregate throughput beyond two lanes (about 333k, 303k, 255k, and 137k publications/s at 1/2/4/8 lanes), so this commit is benchmark isolation only, not a runtime fix. I am tracing the remaining shared-realm environment/property ownership and synchronization costs before changing the engine.

A second benchmark isolation correction is pushed in 6236f4b.

Unique property-name strings are now constructed in the existing untimed __benchmarkPrepare phase, one immutable key array per lane. The scored worker boundary remains one unique Shape-edge publication, exact value observation, and delete-to-base validation per operation. This removes integer/string formatting and its GC/backing allocations from the claimed transition-publication work.

The benchmark matrix structural tests pass 18/18, and all 20 quick samples across 1/2/4/8 lanes retain the frozen checksums (501500, 1503500, 5009000, 18026000). No runtime fix is included in this commit.

Shape transition attribution is now first-class in 6ced428. Tier-attribution rows record requests, hits, misses, and writer yields alongside the existing Environment/Object/allocation/worker counters.

On the isolated 1,000-publication-per-lane row, the candidate records:

• 1 lane: 1,002 requests / 1,001 misses / 0 yields
• 2 lanes: 2,004 / 2,002 / 0
• 4 lanes: 4,008 / 4,004 / 0
• 8 lanes: 8,016 / 8,008 / 0

That evidence rules out contended per-Shape writer locks in the partitioned row. It instead exposed persistent-AVL path-copy amplification and the surrounding ordinary-property synchronization costs. The benchmark comparison self-tests and 18/18 representative matrix structural tests pass.

The Shape publication implementation is pushed in 2708a80.

The persistent AVL remains the deep-shape property lookup index. Transition fanout now uses a separate append-only concurrent skip list:

• exact byte-key equality;
• one node per published edge instead of copying an AVL spine;
• one existing per-Shape writer lock for same-name convergence;
• lock-free readers over release-published immutable forward links;
• SipHash-2-4 secret level selection for attacker-controlled property names;
• secure-entropy failure is fail-closed, matching collection hash seeding.

Validation:

• Shape-focused units: 27 pass / 0 fail / 0 leak, including allocation-failure rollback, bounded + moving GC, concurrent same-name convergence, and shared-realm publication.
• focused no-GIL TSan publication: pass (23.56 s), no race/leak report.
• exact Shape-only test262: Object 3,411 + Array prototype 2,810 + Proxy 311 = 6,532 pass / 0 fail / 0 flips.

The partitioned JS row still does not meet the issue’s monotonic 1/2/4/8 throughput acceptance because surrounding ordinary-property synchronization remains dominant. I am keeping #531 open and continuing that work; the Shape writer-lock diagnosis itself is now closed by measured zero yields.

The measured non-Shape bottleneck is split into #532: ordinary named-property synchronization and its private/publication proof. #531 remains the parent end-to-end scaling issue; #532 owns the lock-amortization work needed to satisfy its remaining monotonic 1/2/4/8 acceptance. This split does not relax or remove any #531 gate.

The Environment portion of the measured synchronization cost is complete in #533 via 6a6b508a: exact invocation Environment locks fell about 90.4% at both 4 and 8 lanes with unchanged checksums, full units/TSan/threadfuzz/test262 gates green. #531 remains open because end-to-end monotonic 1/2/4/8 throughput is still not proven; ordinary named-property synchronization remains tracked in #532.

#532 made another measured step in 58d73f42: exact property-lock counts fell about 38.5% at both 4 and 8 lanes with unchanged workload checksums and full semantic/concurrency gates green. This is attribution evidence only. #531 stays open because its monotonic 1/2/4/8 throughput acceptance has not been satisfied.

Post-58d73f421558d3918de2e37af0e7ebbb6305f4c3 clean-main scaling check (ReleaseFast, representative_shape_transition_fanout, shared realm, 1,000 jobs, 7 samples/lane):

The checksums are exact and all seven rows were valid at every lane count. The 1→2 lane result improves, but 4 and 8 lanes collapse, so #531 acceptance is not met and the issue remains open.

The same run shows remaining synchronization acquisition counts scale roughly linearly while recorded contention is low: 4 lanes had 64,230 property-lock and 36,299 environment-lock acquisitions; 8 lanes had 128,422 and 72,531 respectively. System CPU also rose from 16.987 ms at 4 lanes to 103.136 ms at 8 lanes. The aggregate environment metric currently mixes root, captured, other reads, writes, and marker synchronization; the next step is to split that attribution before changing another lock path.

These are diagnostic /tmp samples, not publication evidence.

Environment lock ownership attribution landed in 3577d307, and the private-write protocol landed in 2d895edb / #535.

Exact 8-lane ownership before #535 accounted for all 72,531 Environment acquisitions: 16,334 reads + 56,197 writes + 0 traces. Of the writes, 56,191 were uncaptured private activations. After #535, actual Environment acquisitions are 16,340 and those 56,191 private writes are recorded as guarded elisions (-77.47% locks), with checksum 18,026,000 preserved.

Seven-sample ReleaseFast representative_shape_transition_fanout, 1,000 jobs per lane:

Compared with the prior clean-main medians, wall time improved by approximately 6.4% / 7.7% / 31.7% / 36.4% at 1/2/4/8 lanes. The optimization is material, but #531 acceptance is not met: throughput still peaks at two lanes and declines at four and eight. Keeping this issue open for the next measured bottleneck.

#532 advanced the parent end-to-end row in dc919306. Exact 8-lane property-lock acquisitions fell 128,422 -> 80,384 (-37.4%) with checksum 18,026,000; Environment/backing/element counts stayed 16,340 / 33,048 / 24,073.

Final-code seven-sample medians are 3,584,292 / 6,225,166 / 14,681,083 / 30,657,959 ns at 1/2/4/8 lanes, or 278,995.1 / 321,276.6 / 272,459.5 / 260,943.7 aggregate edges/s. Relative to the prior exact parent in this issue, elapsed time changes by +1.35% / -2.74% / -19.75% / -23.88%.

This is a material four/eight-lane improvement, but #531 acceptance remains unmet because aggregate throughput still peaks at two lanes. The exact attribution and semantic/concurrency evidence are in the #532 update. I am keeping both issues open; I am also leaving Thread/Worker/job lifecycle changes to the separately owned #482/#502 work rather than folding them into the Shape/Object change.

Profiling the current representative fanout workload found that a substantial share of scheduler yields is outside shape publication: direct-mapped statement-location cache aliases repeatedly enter the shared debugger registry. That independent bottleneck is now tracked in #541. I’ll add opt-in counters before changing the cache, then re-run the same #531 attribution matrix so subsequent shape conclusions are based on separated costs.

#541 is complete in 59b34db6. The 8-lane base invocation now performs 814 debugger-registry acquisitions instead of 440,593, with identical checksum, object-lock counts, and shape counts. Seven exact-parent diagnostic pairs measured wall 0.568×, instructions 0.6706×, cycles 0.5872×, and peak RSS 1.0004×; variance is explicitly too high for publication evidence.

The shape attribution is now much cleaner: base/variant still publish 8,008 misses with zero shape-lock yields, while the deliberately contended control isolates writer yields. I’m returning to the independent object-synchronization child #532 before drawing another conclusion about the remaining >2-lane scaling ceiling.

Post-d437b5fc diagnostics confirm that #532 removed the measured property-lock contention, but #531's end-to-end monotonic gate is still not met.

Seven clean-main samples per lane, ReleaseFast, 1,000 jobs/lane:

All checksums remained exact. These are noisy diagnostics, not publication evidence.

Exact invocation attribution on the partitioned 8-lane row records 8,016 transition requests / 8,008 misses / 0 Shape yields, 48,312 property-lock acquisitions with only 2 contentions / 2 spins, and zero backing/element contention. The benchmark's cooperative-GC telemetry records zero scored collections at every lane count, ruling out the sampled post-invocation nursery relocation work as the timed ceiling.

I tested the remaining apparent Shape.lookupState leaf separately in #543 by carrying proven immutable lookup state through OrdinarySet and named Delete. Seven order-balanced exact-parent pairs across all three controls showed no causal improvement and an eight-lane contended regression, so the patch was reverted and #543 closed as not planned. I am keeping #531 open, but will not make another Shape/Object change from this profile alone.

A current-main 1/2/4/8 diagnostic still scales through four lanes and falls at eight (about 282k / 391k / 487k / 377k edges/s). An amplified sample now identifies GcCellBacking single-cell allocation/free as the dominant collapsed path rather than Shape publication or named-property contention. I split that causal allocation problem into #559; #531 remains the end-to-end parent and none of its acceptance gates are relaxed.

Correction to the #559 split above: b3d41abe proved that the apparent GcCellBacking hotspot was generated by fixture preparation still running inside __benchmarkRunShared, plus preparation-triggered GC deferred into the next evaluation. #559 is now closed with that causal correction.

Under the corrected boundary, the 8-lane x 100000-job scored invocation preserves checksum 180002600000, performs zero collections, and records only 82 slab acquisitions across 800000 published edges (41 allocation, 33 publication, 8 ownership, no relocation/unpublication/free). The GC allocator is therefore ruled out for this row.

Corrected three-sample medians are 253,732,709 / 279,234,834 / 319,724,375 / 907,345,417 ns at 1/2/4/8 lanes, or approximately 394k / 716k / 1.25M / 882k aggregate edges/s. Scaling is substantially cleaner and remains strong through four lanes, but the four-to-eight fall still violates #531 acceptance. This issue again owns that residual end-to-end shape/publication analysis; no allocator sharding will be carried forward from the invalid profile.

An unlanded Object key-order candidate establishes a causal path to monotonic 1/2/4/8 partitioned throughput, but it is not ready to land because the required mid-GC lifecycle fuzz gate exposes a differential failure tracked on #482.

Candidate design: creation-order entries borrow immutable Shape operation-name slices owned by the Context arena; accessor/native names retain owned copies. Object and GC teardown free only owned entries. This removes one duplicate/free pair per ordinary add/delete without adding a cache, reserve, or alternate semantic path.

Exact 8034cdfa parent and candidate ReleaseFast binaries were pinned by SHA-256 (0e00bb5f… parent, 162bc678… candidate). Seven alternating fresh-process pairs, 100,000 exact publications per lane:

Every sample preserved the frozen checksums 5000150000, 15000350000, 50000900000, and 180002600000. Three-sample candidate diagnostics also remained monotonic for the anti-specialization variant (about 376k / 722k / 1.31M / 1.87M edges/s). The deliberately same-parent control remained exact and visibly serialized (about 387k / 676k / 694k / 504k edges/s).

Exact eight-lane attribution preserved checksum 180002600000, zero scored collections, 82 slab acquisitions, 800,008 transition requests / 800,000 misses / 0 Shape yields, and the same Object lock counts. Delegated Object-backing allocations/releases fell from 1,725,043 / 1,725,016 to 125,035 / 125,008: 1.6 million fewer of each, or about 92.7%.

Green candidate gates:

• full units: 1,634 pass / 1 skip / 0 fail / 0 leak;
• suppression-free TSan named deletion, moving relocation, and parallel shared mutation: 5/5;
• threadfuzz default: 400/400;
• threadfuzz amplify: 400/400;
• threadfuzz broad: 400/400.

Blocking differential:

• clean 8034cdfa: full mid-GC seed-10 replay 45/45; focused midgcmicrotask 1 10 3/3;
• candidate: focused seed 10 fails 2/2 with TypeError: Thread constructor requires a callable argument; the long mid-GC profile reported the same failure at seeds 10 and 46 before being stopped;
• exact reproducer and ownership boundary are documented in #482.

The candidate is stashed, not committed or pushed. #531 remains open: the performance result is diagnostic, the full efficiency/RSS/non-regression publication matrix has not been accepted, and the no-workaround lifecycle gate must be green before this optimization can land.

#764 removes deterministic adversarial shard/bucket placement from shared string interning in 288b4893, while preserving the 16-way lock topology. This is a bounded security/contention hardening slice only; it does not satisfy #531's independent shape-transition scaling acceptance.

Acceptance metadata reconciliation only. The benchmark-first boundary is proven by d9d54b5b, d30790f2, and 6236f4be: all three identities, equal work, warmup/preparation boundary, and frozen 1/2/4/8 checksums are committed. The same-parent control is also exact and visibly serialized in every retained diagnostic, including the candidate matrix recorded above, so that acceptance row is proven. I checked only those two boxes. Clean publication-grade 1/2/4/8 evidence, landed monotonic partitioned scaling, the complete current-code lifecycle matrix, and published control/RSS evidence remain open; the causal key-order candidate is still correctly gated by #482.

The previously blocked borrowed-key implementation is now landed on main as 88cde689: perf(object): borrow immutable shape order keys (#531, #532).

Design and safety boundary:

• ordinary data add/delete creation-order entries borrow the exact immutable Shape operation-name slice owned by the Context arena;
• accessor/native names remain self-owned;
• rollback, replacement, explicit teardown, and GC finalization free only owned entries;
• no cache, prepublication, reserve, global lock change, or benchmark-recognizing path was added.

The historical #482 seed-10 blocker is resolved on the current candidate: full mid-GC is 45/45 normally and under TSan, plus three focused seed-10 reproductions. Full units are 2,236 passed / 1 expected skip / 0 failed / 0 leaked. Default, amplified, broad, verify, and lifecycle threadfuzz profiles pass; the focused named-delete/OOM/moving-GC TSan group is clean. Exact Test262 remains Object 3,411 -> 3,411, Array 3,081 -> 3,081, Proxy 311 -> 311, with zero failures and zero flips.

Current exact-parent diagnostics (100,000 publications/lane, frozen checksums) show the expected causal separation:

At eight lanes, exact invocation attribution for both independent fixtures changes backing allocation/release calls from 1,725,044 / 1,725,008 to 125,036 / 125,000 and removes 41,422,408 bytes of matched allocation/free traffic. Net live backing-byte change, transition work, property-lock acquisitions, and scored GC collections remain unchanged; the serialized control is wall-neutral at scale. Shallow classes and the mixed application row are neutral-to-better; the dense-Array control is +0.92% wall with neutral-to-better instructions/cycles/energy/RSS.

These are clean diagnostics, not publication evidence. #531 remains open until the seven-pair quiet-reference 1/2/4/8 base/variant/control artifacts, exact attribution/RSS report, and retained deep-lookup control are checked in. No public performance claim is made from these /tmp results.

The standard publication path can now carry this issue exact Shape evidence: #844 landed as a6feac57, with supporting fail-closed runtime-profiler repairs #845 (928d5797) and #846 (61ec0d95). A complete V33 quick collection accepted 510/510 snapshots and rendered request/hit/miss/writer-yield tables without relaxing allocation, synchronization, GC, RSS, tier, or checksum gates. This issue remains open for the quiet-reference publication artifact and retained deep-lookup row.