ReviewOS

home-lang/home

Native recursive readdir stress deadlines and post-timeout descriptor accounting

#896
Open chris opened this 8 days ago ·
8 days ago

Native readdir validation checkpoint (2026-10-14): walker source 26a428426, corpus/control source a8be1de85, and 837 Git-verified execution/diagnostic artifacts are on main through ecd6216cd. Native build 56/56 and all owned ALPN, fetch-timing, init and recursive traversal controls pass. The final 140-file unchanged filesystem/FileHandle batch and every validator pass: 393 cases, 126 process checks, zero failures, 36 original case skips and three complete-file skips. The earlier twelve-file batch’s one synchronous string-result deadline failure remains recorded; a fresh complete-file recheck passes 262 cases with seven skips, and the later wider batch also passes that file. Original iterations/assertions/deadlines remain unchanged. Timing measurements are diagnostic, vary with host/runtime conditions, and are not attributed wholly to the algorithm; installed Bun is not a rebuilt pinned reference. #896 retains robust stress/pinned-reference acceptance. New #897 tracks observed undefined buffer Dirent names and ignored recursive async name encodings. Full feature/suite/all-platform acceptance remains open.

The unchanged pinned js/node/fs/fs.test.ts does not currently pass consistently within its original recursive readdir stress deadlines on Darwin arm64.

During native FileHandle transfer validation, the 119-file filesystem batch had 390 passing cases, 105 process checks and three failures in fs.test.ts: an async recursive x100 timeout, a subsequent fd-count mismatch while earlier work remained outstanding, and a recursive withFileTypes sync x100 timeout. Resource-probe deadlines also failed under host load.

A separate fresh-report complete-file recheck, with original bytes, iterations and deadlines unchanged, produced 260 passing cases, seven original skips and two synchronous recursive x100 deadline failures. The fd-count mismatch did not recur. Earlier source/evidence checkpoints ran this full file successfully, including the initial native-transfer implementation before the expanded lifecycle fixes; cause remains unproven. Native FileHandle/stream/transfer changes do not directly modify recursive readdir implementations.

Acceptance:

  • Establish performance and descriptor-lifetime behavior against the pinned runtime on the same host and fixture tree.
  • Diagnose the complete native recursive traversal and allocation paths, including directory closing and pending async work after test timeout.
  • Implement genuine algorithm/ownership fixes where needed; preserve x100 iteration counts and the original 10-second deadlines.
  • Verify the entire unchanged fs.test.ts and wider filesystem batch; retain stressed failures and separate subsequent evidence.
  • Do not claim all-filesystem or full-suite acceptance from the passing FileHandle selection.

Part of #66; related #878 directory ownership and #895 native FileHandle transfer. The final #895 controls and all 21 unchanged FileHandle originals pass, including early worker termination cleanup and the previously crashing worker-read original.

8 days ago

Native FileHandle handoff checkpoint (2026-10-14): #895 replaces workerData public-string markers/graph rewriting with native structured-clone branding, descriptor tickets, receiver adoption, aliases, rollback and discarded/unused payload cleanup. Native worker exit releases unconsumed tickets before parent notification. FileHandle streams delegate through handle methods; Node's public-fd I/O behavior is preserved while ownership/close uses the native descriptor. Source 79dcb3d01 and 1,628 Git-verified evidence artifacts are on main through f5b1af966. Build 56/56, final Node/Home lifecycle controls, all 21 unchanged FileHandle originals/validators (21 process checks, no skips/failures), five affected ownership checks and Pickier passed. The final wider 119-file batch had 390 passing cases/105 process checks, three fs.test.ts failures, 36 case skips and three complete-file skips. The unchanged full-file recheck passed 260 cases/seven skips with two recursive readdirSync x100 deadline failures; the prior fd-count failure did not recur. Those failures and host resource-probe logs are retained in #896; they receive no pass credit and are not resolved by the passing transfer subset. Full suite/all-platform acceptance, remaining skips, independent builds, #894 Temporal harness, #893/#878 callback traces and #891 integration remain open.

8 days ago

Recursive readdir investigation is in progress under #896. A same-tree 200-traversal baseline counted 5,597 entries on Darwin arm64: Home took 36.0 s for string entries and 95.5 s for Dirents; installed Bun 1.4.2 took 42.1 s and 53.7 s; installed Node v26.10.0 took 13.1 s and 4.18 s. This diagnostic separates traversal from sorting and does not reproduce the complete original test or confer pass credit. The installed Bun reference is not a rebuilt pinned-source binary.

The pending native batch removes repeated descendant path parsing, encodes each opened directory's parent path once, and returns an allocation error while freeing copied queue paths when queue growth fails. Node integration checks and formatting pass. Native rebuild and twelve unchanged pinned original files are pending; original iteration counts and deadlines remain unchanged. Full-suite acceptance remains open.

8 days ago

Recursive readdir checkpoint (2026-10-14): source 26a428426 and 27 Git-verified baseline/build artifacts are on main through e6ee69738. The native batch removes repeated descendant path parsing and parent-path encoding, and repairs queue-allocation error/cleanup behavior. Node integration, Pickier, Zig formatting and diff checks pass. Home compilation and twelve unchanged pinned originals are pending: two native rebuilds were stopped by the resource monitor after repeated kernel-probe deadlines, and a fourth guarded build is queued behind a live Home job. Same-tree 200-traversal timings are diagnostic only, with no original-test pass credit. Original iterations/deadlines and the full Bun-suite/all-platform goal remain unchanged; #896 tracks completion.

8 days ago

Corpus/native build checkpoint (2026-10-14): source a8be1de85 and 23 Git-verified evidence artifacts are on main through cf6d284db. The new full inventory audit passes all 12,996 pinned entries, explicitly reporting three adopted upstream TLS revisions from #710 and four removed failure exclusions. The pinned CLI init original is restored; its Home entry-preservation regression moved to an owned control. All six verifier controls, Pickier and cryptographic fixture checks pass. Recursive walker source 26a428426 now compiles: native build attempt six succeeded 56/56, peak measured tree footprint 5,697 MB. Owned execution controls, post-change timing measurements and twelve unchanged originals remain queued behind a confirmed-live Home suite; no execution pass credit is assigned. Full feature/suite/all-platform acceptance remains open, with readdir acceptance tracked in #896.

8 days ago

Native readdir validation checkpoint (2026-10-14): walker source 26a428426, corpus/control source a8be1de85, and 837 Git-verified execution/diagnostic artifacts are on main through ecd6216cd. Native build 56/56 and all owned ALPN, fetch-timing, init and recursive traversal controls pass. The final 140-file unchanged filesystem/FileHandle batch and every validator pass: 393 cases, 126 process checks, zero failures, 36 original case skips and three complete-file skips. The earlier twelve-file batch’s one synchronous string-result deadline failure remains recorded; a fresh complete-file recheck passes 262 cases with seven skips, and the later wider batch also passes that file. Original iterations/assertions/deadlines remain unchanged. Timing measurements are diagnostic, vary with host/runtime conditions, and are not attributed wholly to the algorithm; installed Bun is not a rebuilt pinned reference. #896 retains robust stress/pinned-reference acceptance. New #897 tracks observed undefined buffer Dirent names and ignored recursive async name encodings. Full feature/suite/all-platform acceptance remains open.

8 days ago

Native directory encoding checkpoint (2026-10-14): source 68ac671bf through 5e3cdf8c8 and 2,285 Git-verified evidence artifacts are on main through 9f99fd4f8. Native readdir now returns real Buffer Dirent names and plain Buffer arrays, preserves Buffer root/sibling parent identities and root spelling, applies recursive async encodings, and builds encoded parent displays without changing filesystem traversal bytes. Private worker StringImpl copies, GC roots, conversion cleanup, bulk code-unit copies and a bounded ASCII UTF-8 fast path cover ownership and performance. SDK class/global layouts are unchanged. Final build 56/56; Node/Home encoding/concurrency plus traversal controls pass; ABI checks 10/10 with 58 assertions and Pickier pass. The final unchanged 140-file filesystem/FileHandle batch and all validators pass: 393 cases, 126 process checks, zero failures, 36 original case skips and three complete-file skips. Initial compile/control failures, an unreachable-code file abort, diagnostic memory-ceiling failure and later deadline failures remain recorded. Original iterations/assertions/ten-second deadlines are unchanged. #897 stays open for Linux raw-byte, Windows and lifecycle acceptance; APFS rejects the raw invalid-UTF8 fixture with EILSEQ. #898 tracks the independent pure-path UTF-16 gap; #896 retains robust stress/pinned-reference acceptance. The full feature/suite/all-platform objective remains open.

Sign in to comment on this issue.