ReviewOS

stacks/ts-cloud

Co-hosting has no port allocation: two apps on one server collide on the template's default ports

#168
Closed glennmichael123 opened this 2 months ago ·
2 months ago

Summary

Attaching a second app to an existing server (#167, operation 1) collides on ports immediately, because every app generated from the Stacks cloud template binds the same two loopback ports. Nothing allocates ports per site on a shared host, and nothing validates the collision across projects, so the failure surfaces as a service that will not start rather than as a plan-time error.

Evidence

Two independent apps, both untouched from the template:

loghq/config/cloud.ts     port: 3022   (app)   port: 3023   (api)
bughq/config/cloud.ts     port: 3022   (app)   port: 3023   (api)

Identical. These are the two apps #167 names as the ones to consolidate onto the statushq box, and a third app from the same template will be the same again. The first attach works; the second cannot bind.

validateDeploymentConfig does build a portOwners map and raise deploymentCoexistenceError, but it iterates config.sites for one config. An attached app is a separate repo with its own config, so the port it wants is invisible to the owner's validation and vice versa.

What's missing

Port allocation is part of per-site isolation. #167 lists isolation as "user, directory, service unit, log stream" and stops short of the one piece of shared state that is guaranteed to conflict. Options, roughly in order of preference:

  1. Allocate at attach time. The host owns a port range; attaching a site claims the next free pair and records it in the host's inventory. The attached app stops declaring absolute ports at all.
  2. Unix sockets instead of TCP. rpx routes to a per-site socket, which removes the shared namespace rather than partitioning it.
  3. Detect and refuse. At minimum, attach --dry-run reads the host's existing site ports and fails the plan with both owners named, instead of discovering it at systemctl start.

Any of these is fine. Silently binding over another app's port is not, and neither is asking every app author to hand-pick a globally unique port.

Acceptance criteria

  • Attaching a second app from the default template to an occupied host succeeds without either app editing a port by hand
  • A port conflict is reported at plan time, naming both sites, not at service start
  • --dry-run shows the ports each site will occupy after the operation
  • Port assignments are recorded where the next attach can see them, so the third app is allocated correctly
  • #167 — server consolidation (this blocks its first operation)
  • #169 — credential blast radius on a shared project
  • stacksjs/stacks#2342 — the CLI surface

Found while planning a loghq deploy onto the statushq box, against ts-cloud 0.7.114.

9 days ago

Correcting this issue's own premise, from implementing it in #170. Two things I got wrong when filing.

1. The failure mode is worse than described

I wrote that "the first attach works; the second cannot bind" and that it "surfaces as a service that will not start". It does not. assertPortsAreFree in stacks already documents what actually happens:

Two processes CAN bind the same port here: ts-cloud's units do not set exclusive binding, so the kernel load-balances between them instead of failing. Nothing errors, both services look healthy, and each domain serves the other tenant's site on roughly half its requests.

With a recorded incident: predicthq.org had been on 3070 for a day and a half, and after another project's deploy it started answering with that project's storefront.

So this is not a noisy startup failure that an operator notices immediately. It is silent traffic mixing between two unrelated apps, which makes it materially more serious than filed, and makes deciding ports ahead of the deploy more valuable rather than less.

2. Detection already exists, in stacks

I said "nothing validates the collision across projects". That is wrong for the deploy path. assertPortsAreFree in storage/framework/core/buddy/src/commands/deploy.ts already:

  • reads the box's live listeners over SSH (ss -lntpH "sport = :$p")
  • resolves the owning systemd unit, so it can name the culprit
  • skips ports held by this project's own units, so a redeploy is not a false positive
  • stops the deploy, naming each port and its holder

Live listeners are better evidence than declared config, and that check should stay as the last line of defence.

What it cannot do is avoid the clash. Its remedy is to exit and tell the operator to pick free ports in config/cloud.ts by hand, which is exactly the acceptance criterion here that asks for the opposite:

  • Attaching a second app from the default template to an occupied host succeeds without either app editing a port by hand

What #170 therefore does

Allocation and plan-time reporting, not a second detector. It reads the rpx registry fragments the box already keeps (/etc/rpx/sites.d/<slug>.json), where every server-app route records its upstream as host:port, so the host was already its own port registry and simply never read. From that it can name a conflict at plan time and, more usefully, hand the attaching app a free port so nobody hand-picks one.

Acceptance criteria 1, 2 and 4 are covered there. Criterion 3, --dry-run showing the ports, is left for the attach/CLI plan surface in #167 and stacksjs/stacks#2342, because the only current caller of validateDeploymentConfig runs before any driver exists and so has no host connection to read fragments with. I would rather leave a tested seam than add a probe that fails open and silently passes the collision through, which is the bug this issue is about.

Sign in to comment on this issue.