ReviewOS

reviewos/reviewos.org

Web process memory growth OOM-kills active Git transfers

#8
Closed chris opened this 4 days ago ·
4 days ago

ReviewOS's page-serving process repeatedly reaches its configured 2 GiB cgroup hard limit and is OOM-killed, terminating active smart-HTTP Git transfers. HQ's Mac runner receives curl 56 / OpenSSL unexpected EOF, early EOF and invalid index-pack output before any native acceptance steps.

Observed on deployed source 541c680, with Stacks 0.75.217 / STX 0.2.574 / Bun 1.4.2. Kernel events identify the web service at 15:31 and 15:48 UTC on October 29; its memory peak reaches exactly 2,147,483,648 bytes. The separate API service remains active. Later web restarts continue.

Affected consumer runs:

Both deployed Bun 1.4.2 and local Bun 1.4.4 hold a 50 MiB writer at the pipe when its reader stops, for both Node child_process and native Bun.spawn. That measurement does not support treating the historical stdout buffering defect as the current cause.

Investigate retained data / allocation growth in the web-serving path and repair the responsible first-party source package. Keep the hard memory bound and normal transport; protocol overrides, retry loops, larger limits and scheduled garbage collection are not acceptance.

Acceptance: representative sustained page/code browsing and fresh normal runner checkouts remain within the configured bound, with no OOM restart, truncated Git transfer or cross-account response/cache leak. Preserve cancellation and streaming behavior. Any temporary diagnostics must be removed after verification.

Related: https://github.com/stacksjs/hq.training/issues/346, https://github.com/stacksjs/hq.training/issues/318, https://github.com/stacksjs/hq.training/issues/366, https://github.com/stacksjs/hq.training/issues/368.

4 days ago

The bounded highlighted-code cache fix (08b9f75) passed the normal full ReviewOS CI, including MySQL and PostgreSQL, and deployed successfully: https://github.com/ReviewOS/reviewos.org/actions/runs/40910195796.

A fresh ordinary shallow clone through the supported /git/reviewos/reviewos.org.git mount completed at 08b9f75 and passed git fsck --no-reflogs, without a transport override or retries. Runner source checkouts already use this mount. The separate public marketing clone-link error is tracked at /reviewos/reviewos.org/issues/9.

Memory is still high and the new web instance has restarted, so this issue is not resolved. The aggregate profile points to overlapping STX renders; approximately 175 incoming web connections coincided with growing template/script contexts. The shared framework fix is implemented and tagged at /stacks/stx/issues/2136, pending normal publication and consumer deployment. Real HTTP regressions verify capacity, queued expiry/cancellation, disconnect accounting and request identity on Bun 1.4.2.

All temporary production profiling/preload configuration and its files have been removed after preserving aggregate metadata. The 2 GiB memory limit is unchanged; no periodic GC, alternate HTTP protocol, forced restart or retry workaround remains.

4 days ago

Published STX 0.2.746 is now consumed by ReviewOS at https://github.com/ReviewOS/reviewos.org/commit/8b05734e8a8570df3e8fe3105131bb5442b89b9a and HQ at https://github.com/stacksjs/hq.training/commit/c2d6b80b28ddfd725a6c478c53a2cf12dcb69285. No dependency patch or source link is used.

ReviewOS validation on pinned Bun 1.4.2: lint/types clean; 107 focused cache/highlight/Git/URL checks pass. The installed normal serve entry, against a framework-owned disposable PostgreSQL database with real migrations, rendered the public homepage and canonical clone link; eight concurrent public requests returned 200. Normal full ReviewOS CI has passed both database suites and is deploying: https://github.com/ReviewOS/reviewos.org/actions/runs/40919521264.

Production memory/Git acceptance remains open at /reviewos/reviewos.org/issues/8. Memory and runner admission guards remain unchanged, and all temporary production profiling is removed.

4 days ago

Delivered through the shared source and normal published dependencies:

  • SSR admission, queue capacity/deadline, cancellation and recovery: https://github.com/stacksjs/stx/commit/8aaba3ddf31f112b292af1b662f2eed62eb90d12, published as STX 0.2.746.
  • ReviewOS consumes the complete published cohort and its byte-bounded highlighting cache: https://github.com/ReviewOS/reviewos.org/commit/8b05734e8a8570df3e8fe3105131bb5442b89b9a. The normal source CI and production deployment passed: https://github.com/ReviewOS/reviewos.org/actions/runs/40919521264.
  • Two rounds of eight concurrent public homepage/docs/explore/repository requests passed HTTP 200. Four concurrent real source-file pages also passed HTTP 200, including highlighted Git and highlighting-pool source.
  • Fresh ordinary shallow clones from the canonical Git endpoint passed git fsck --no-reflogs before and after an external managed service restart. Both resolve to the expected deployed source. No transport overrides or retry loops were used.
  • The first process held near 959 MiB through its observation interval before an external managed stop/restart at 17:43 UTC; that was not an OOM and was not initiated by this work. At 17:55:54 UTC the replacement process is active, NRestarts=0, current memory 795,762,688 bytes and peak 1,111,916,544 bytes, below the unchanged 2,147,483,648-byte limit. Neither observation interval recorded an OOM. This is bounded acceptance evidence, not an indefinite uptime guarantee.
  • Temporary diagnostic preload files and service overrides were removed. No scheduled collection, memory-limit increase or forced restart remains.

The application passed 107 focused regressions, full types and strict lint before deployment. The roadmap acceptance and deployment references are recorded at https://github.com/ReviewOS/reviewos.org/commit/92dc4a23; its 10 roadmap checks and full strict lint pass.

The independent LSU UIKit/voice acceptance remains open at /stacks/hq.training/issues/318. Its Mac jobs are currently waiting on disk admission, not a claimed native pass.

Sign in to comment on this issue.