ReviewOS
Reload

also looking at this

mail-os/mail

docs: the production server is reached over SSH, not SSM

#11
Closed glennmichael123 wants to merge docs/ssh-access into main
1 file +75 -46

Review threads live on the whole diff, not on one commit, so none are shown here - a thread's line means something in the branch's final form, and painting it into an intermediate step would put it on code it is not about.

CLAUDE.mdmodified+75-46
@@ -89,64 +89,93 @@ This project uses Zig 0.16.0-dev which has breaking changes from 0.15:
8989
9090## Production Server
9191
92- **Instance**: `i-0e365c6bd31da4678` (EC2, Amazon Linux 2023, us-east-1)
93- **Domain**: `mail.stacksjs.com`
94- **Binary**: `/opt/mail/mail-server` (renamed from `mail` to avoid conflict with `mail/` mailbox dir)
95- **Mailboxes**: `/opt/mail/mail/{username}/new/` (Maildir format)
96- **Database**: `/opt/mail/smtp.db` (SQLite)
92Reached over SSH. There is no SSM agent on this host, and no AWS instance behind
93it — an earlier version of this file described an EC2 box in us-east-1 deployed
94via `aws ssm send-command`, which no longer exists.
95
96- **Host**: `178.105.248.188` — Hetzner, Ubuntu 24.04 LTS, x86_64
97- **Hostname**: reports as `statushq-production-app`; the box is shared, so do
98 not assume a unit belongs to mail just because it is running here
99- **Domain**: `mail.stacksjs.com` (A record points at the IP above)
100- **Access**: `ssh -i ~/.ssh/id_ed25519 root@178.105.248.188`
101- **Binary**: `/opt/mail/mail-server`
102- **Service**: `mail.service` — "Mail Server (Zig)", systemd, runs as the
103 `mail-server` user with `CAP_NET_BIND_SERVICE`
97104- **Config**: `/etc/mail/mail.env`
105- **Delivery**: `SMTP_DELIVERY_METHOD=direct` — this host sends its own mail. It
106 is not relaying through SES.
98107- **TLS**: Let's Encrypt at `/etc/letsencrypt/live/mail.stacksjs.com/`
99- **Service**: `mail.service` (systemd, runs as `mail-server` user with `CAP_NET_BIND_SERVICE`)
100- **Delivery**: Amazon SES (`SMTP_DELIVERY_METHOD=ses`)
101- **Ports**: 25 (SMTP), 465 (SMTPS), 587 (Submission), 143 (IMAP), 993 (IMAPS)
108- **Ports**: 25 (SMTP), 465 (SMTPS), 587 (Submission), 143 (IMAP), 993 (IMAPS).
109 ufw allows all five from anywhere, plus 22/80/443.
102110
103## Deployment via AWS SSM
111### State lives under /var/lib, not /opt
104112
105Deploy new builds without SSH. The server has SSM agent running and an IAM role with SSM permissions.
113Everything in `/opt/mail` that holds data is a symlink into
114`/var/lib/mail-storage/data/`. Follow the link before reasoning about a path,
115and back up the real file rather than the symlink:
116
117| via /opt/mail | real path |
118|---|---|
119| `smtp.db` | `/var/lib/mail-storage/data/smtp.db` |
120| `mail/{username}/` | `/var/lib/mail-storage/data/mail/{username}/` |
121| `dkim/` | `/var/lib/mail-storage/data/dkim/` |
122| `forwards.json` | `/var/lib/mail-storage/data/forwards.json` |
123| `backups/` | `/var/lib/mail-storage/data/backups/` |
124
125Maildirs are named after the account's `username` column, which is **not always
126the email address** — 12 of the current accounts are stored bare (`cloud`,
127`zoltan`, `noreply`) and the rest use the full address. A bare-username account
128will not authenticate with its email address.
129
130`mail.log` sits beside them and is not rotated; it was 345MB as of 2026-10-05,
131so `tail -c` a slice rather than reading it whole.
132
133## Deployment over SSH
134
135Cross-compile locally, ship the binary, restart the unit. Keep the old binary —
136the convention on the box is `mail-server.bak-<epoch>`, and there are a dozen of
137them from previous rollbacks.
106138
107139```bash
108# 1. Cross-compile for Linux
140# 1. Build for the server
109141cd packages/zig
110zig build -Dtarget=x86_64-linux
111
112# 2. Upload binary to S3
113aws s3 cp zig-out/bin/mail s3://stacks-production-s3-email/deploy/mail-server-new
114
115# 3. Deploy via SSM (stop service, swap binary, restart)
116aws ssm send-command \
117 --instance-ids i-0e365c6bd31da4678 \
118 --document-name "AWS-RunShellScript" \
119 --parameters 'commands=[
120 "aws s3 cp s3://stacks-production-s3-email/deploy/mail-server-new /tmp/mail-server-new",
121 "chmod +x /tmp/mail-server-new",
122 "systemctl stop mail",
123 "cp /opt/mail/mail-server /opt/mail/mail-server.bak",
124 "mv /tmp/mail-server-new /opt/mail/mail-server",
125 "chown mail-server:mail-server /opt/mail/mail-server",
126 "systemctl start mail",
127 "sleep 2",
128 "systemctl is-active mail"
129 ]'
130
131# 4. Check result
132aws ssm get-command-invocation \
133 --command-id <COMMAND_ID> \
134 --instance-id i-0e365c6bd31da4678
135
136# View logs
137aws ssm send-command \
138 --instance-ids i-0e365c6bd31da4678 \
139 --document-name "AWS-RunShellScript" \
140 --parameters 'commands=["journalctl -u mail --no-pager -n 100"]'
142zig build -Doptimize=ReleaseFast -Dtarget=x86_64-linux
143
144# 2. Ship it beside the running one
145scp -i ~/.ssh/id_ed25519 zig-out/bin/mail root@178.105.248.188:/tmp/mail-server-new
146
147# 3. Swap and restart
148ssh -i ~/.ssh/id_ed25519 root@178.105.248.188 '
149 cp /opt/mail/mail-server /opt/mail/mail-server.bak-$(date +%s)
150 install -o mail-server -g mail-server -m 755 /tmp/mail-server-new /opt/mail/mail-server
151 systemctl restart mail.service
152 systemctl is-active mail.service
153'
154
155# 4. Confirm it came back on all five ports
156ssh -i ~/.ssh/id_ed25519 root@178.105.248.188 \
157 "ss -ltn | grep -cE ':(25|143|465|587|993)\\b'" # expect 5
158
159# Logs
160ssh -i ~/.ssh/id_ed25519 root@178.105.248.188 \
161 "journalctl -u mail.service --no-pager -n 100"
141162```
142163
143The old host key may need updating after instance rebuilds:
144```bash
145ssh-keygen -R mail.stacksjs.com
146```
164No `setcap` step. The binary carries no file capabilities (`getcap` on it is
165empty) — port 25 works because the unit declares
166`AmbientCapabilities=CAP_NET_BIND_SERVICE`, which systemd grants at launch. Keep
167the `install` ownership as `mail-server:mail-server 755`, matching what is there.
168
169If the host key changes after a rebuild: `ssh-keygen -R mail.stacksjs.com`
147170
148171## Cloud Infrastructure (packages/cloud)
149172
173> This describes `packages/cloud/cloud.config.ts` as written, not what is
174> serving mail today. Production currently runs on the Hetzner host above,
175> with `SMTP_DELIVERY_METHOD=direct` and DNS at Porkbun — no EC2, no SES
176> relay, no Route 53. Treat this section as the AWS path the repo still
177> supports, and verify against the live box before acting on it.
178
150179Defined in `cloud.config.ts` using ts-cloud (CloudFormation wrapper):
151180
152181- **EC2**: t3 instance with security groups for mail ports