also looking at this
feat(antispam): harden inbound spam detection
#3
3 files
+670
-90
| @@ -0,0 +1,193 @@ | ||
| 1 | # Mail Server | |
| 2 | ||
| 3 | A performant, self-hosted mail server written in Zig. Supports SMTP, IMAP, POP3, CalDAV/CardDAV, and more. | |
| 4 | ||
| 5 | ## Project Structure | |
| 6 | ||
| 7 | ``` | |
| 8 | mail/ | |
| 9 | ├── packages/ | |
| 10 | │ ├── zig/ # Core mail server (Zig 0.16.0-dev) | |
| 11 | │ │ ├── src/ | |
| 12 | │ │ │ ├── main.zig # Server entry point | |
| 13 | │ │ │ ├── mail_cli.zig # CLI entry point (zig-cli) | |
| 14 | │ │ │ ├── core/ # Core: config, logging, protocol, TLS, sockets | |
| 15 | │ │ │ ├── protocol/ # IMAP, POP3, CalDAV, ActiveSync, etc. | |
| 16 | │ │ │ ├── auth/ # Auth, password hashing (Argon2id), CSRF | |
| 17 | │ │ │ ├── storage/ # SQLite database layer | |
| 18 | │ │ │ ├── delivery/ # Queue, bounce handling, TLS-RPT | |
| 19 | │ │ │ ├── antispam/ # DKIM, SPF, DMARC, ARC, DNSBL | |
| 20 | │ │ │ ├── observability/ # Logging, metrics, tracing, Discord alerts | |
| 21 | │ │ │ ├── features/ # Sieve, quotas, templates, autoresponder | |
| 22 | │ │ │ ├── infrastructure/ # Clustering, io_uring, connection pooling | |
| 23 | │ │ │ └── api/ # REST API, health checks | |
| 24 | │ │ ├── build.zig | |
| 25 | │ │ └── pantry/ # Zig dependencies (zig-tls, zig-cli) | |
| 26 | │ ├── cloud/ # AWS infrastructure (ts-cloud / CloudFormation) | |
| 27 | │ │ └── cloud.config.ts # EC2, SES, Route53, IAM config | |
| 28 | │ └── ts/ # TypeScript SDK (ts-mail) | |
| 29 | ├── pantry.jsonc # Monorepo config (workspaces, scripts) | |
| 30 | └── docs/ # Architecture, security, protocol docs | |
| 31 | ``` | |
| 32 | ||
| 33 | ## Build & Development | |
| 34 | ||
| 35 | ```bash | |
| 36 | # Package manager is `pantry` (like npm/bun) | |
| 37 | pantry run build # ReleaseFast build | |
| 38 | pantry run build:debug # Debug build | |
| 39 | pantry run dev # Build + run server locally | |
| 40 | pantry run test # Run all tests | |
| 41 | pantry run fmt # Format Zig source | |
| 42 | ||
| 43 | # Or directly from packages/zig: | |
| 44 | cd packages/zig | |
| 45 | zig build # Debug build | |
| 46 | zig build -Doptimize=ReleaseFast # Release build | |
| 47 | zig build -Dtarget=x86_64-linux # Cross-compile for Linux | |
| 48 | zig build test # Run tests | |
| 49 | zig build run -- serve # Run server | |
| 50 | ``` | |
| 51 | ||
| 52 | ## Zig 0.16 Specifics | |
| 53 | ||
| 54 | This project uses Zig 0.16.0-dev which has breaking changes from 0.15: | |
| 55 | ||
| 56 | - `std.ArrayList` is unmanaged: init with `.empty`, pass allocator to every method (`append(allocator, ...)`, `deinit(allocator)`) | |
| 57 | - `std.time.sleep` removed: use `time_compat.sleep()` (our wrapper using `std.c.nanosleep`) | |
| 58 | - `std.fs.cwd()` removed: use `fs_compat` module for file operations | |
| 59 | - `std.process.Child.run` removed: use `extern "c" fn system()` or C file I/O | |
| 60 | - `std.c.stat/remove/system` removed: declare `extern "c"` functions directly | |
| 61 | - Argon2 password hashing uses `p=1` (single-threaded) to avoid async I/O requirements in tests | |
| 62 | - Custom compat layers: `src/core/io_compat.zig`, `src/core/fs_compat.zig`, `src/core/time_compat.zig`, `src/core/socket_compat.zig` | |
| 63 | ||
| 64 | ## Production Server | |
| 65 | ||
| 66 | - **Instance**: `i-0e365c6bd31da4678` (EC2, Amazon Linux 2023, us-east-1) | |
| 67 | - **Domain**: `mail.stacksjs.com` | |
| 68 | - **Binary**: `/opt/mail/mail-server` (renamed from `mail` to avoid conflict with `mail/` mailbox dir) | |
| 69 | - **Mailboxes**: `/opt/mail/mail/{username}/new/` (Maildir format) | |
| 70 | - **Database**: `/opt/mail/smtp.db` (SQLite) | |
| 71 | - **Config**: `/etc/mail/mail.env` | |
| 72 | - **TLS**: Let's Encrypt at `/etc/letsencrypt/live/mail.stacksjs.com/` | |
| 73 | - **Service**: `mail.service` (systemd, runs as `mail-server` user with `CAP_NET_BIND_SERVICE`) | |
| 74 | - **Delivery**: Amazon SES (`SMTP_DELIVERY_METHOD=ses`) | |
| 75 | - **Ports**: 25 (SMTP), 465 (SMTPS), 587 (Submission), 143 (IMAP), 993 (IMAPS) | |
| 76 | ||
| 77 | ## Deployment via AWS SSM | |
| 78 | ||
| 79 | Deploy new builds without SSH. The server has SSM agent running and an IAM role with SSM permissions. | |
| 80 | ||
| 81 | ```bash | |
| 82 | # 1. Cross-compile for Linux | |
| 83 | cd packages/zig | |
| 84 | zig build -Dtarget=x86_64-linux | |
| 85 | ||
| 86 | # 2. Upload binary to S3 | |
| 87 | aws s3 cp zig-out/bin/mail s3://stacks-production-s3-email/deploy/mail-server-new | |
| 88 | ||
| 89 | # 3. Deploy via SSM (stop service, swap binary, restart) | |
| 90 | aws ssm send-command \ | |
| 91 | --instance-ids i-0e365c6bd31da4678 \ | |
| 92 | --document-name "AWS-RunShellScript" \ | |
| 93 | --parameters 'commands=[ | |
| 94 | "aws s3 cp s3://stacks-production-s3-email/deploy/mail-server-new /tmp/mail-server-new", | |
| 95 | "chmod +x /tmp/mail-server-new", | |
| 96 | "systemctl stop mail", | |
| 97 | "cp /opt/mail/mail-server /opt/mail/mail-server.bak", | |
| 98 | "mv /tmp/mail-server-new /opt/mail/mail-server", | |
| 99 | "chown mail-server:mail-server /opt/mail/mail-server", | |
| 100 | "systemctl start mail", | |
| 101 | "sleep 2", | |
| 102 | "systemctl is-active mail" | |
| 103 | ]' | |
| 104 | ||
| 105 | # 4. Check result | |
| 106 | aws ssm get-command-invocation \ | |
| 107 | --command-id <COMMAND_ID> \ | |
| 108 | --instance-id i-0e365c6bd31da4678 | |
| 109 | ||
| 110 | # View logs | |
| 111 | aws ssm send-command \ | |
| 112 | --instance-ids i-0e365c6bd31da4678 \ | |
| 113 | --document-name "AWS-RunShellScript" \ | |
| 114 | --parameters 'commands=["journalctl -u mail --no-pager -n 100"]' | |
| 115 | ``` | |
| 116 | ||
| 117 | The old host key may need updating after instance rebuilds: | |
| 118 | ```bash | |
| 119 | ssh-keygen -R mail.stacksjs.com | |
| 120 | ``` | |
| 121 | ||
| 122 | ## Cloud Infrastructure (packages/cloud) | |
| 123 | ||
| 124 | Defined in `cloud.config.ts` using ts-cloud (CloudFormation wrapper): | |
| 125 | ||
| 126 | - **EC2**: t3 instance with security groups for mail ports | |
| 127 | - **SES**: Domain verification, DKIM signing | |
| 128 | - **Route 53**: MX, A, SPF, DMARC, DKIM DNS records | |
| 129 | - **IAM**: Role with SES, S3, Route53, SSM permissions | |
| 130 | - **S3**: `stacks-production-s3-email` for deployments and backups | |
| 131 | - **User data script** handles: Zig install, build from git, Let's Encrypt, fail2ban, systemd service, log rotation, certbot renewal | |
| 132 | ||
| 133 | Deploy infrastructure: | |
| 134 | ```bash | |
| 135 | pantry run deploy # Deploy CloudFormation stack | |
| 136 | pantry run cloud:status # Check stack status | |
| 137 | pantry run cloud:diff # Preview changes | |
| 138 | ``` | |
| 139 | ||
| 140 | ## Discord Health Monitoring | |
| 141 | ||
| 142 | The server includes a background health monitor (`src/observability/discord.zig`) that sends alerts to Discord via webhook. Checks include: active connections, database health, TLS cert expiry, disk space, and heartbeat. | |
| 143 | ||
| 144 | Configure via environment variable: | |
| 145 | ``` | |
| 146 | DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/... | |
| 147 | ``` | |
| 148 | ||
| 149 | ## Key Architecture Decisions | |
| 150 | ||
| 151 | - **Maildir format**: Messages stored as individual files with `:2,FLAGS` suffix (S=Seen, R=Answered, F=Flagged, D=Draft, T=Deleted) | |
| 152 | - **IMAP flag persistence**: Flags are persisted by renaming maildir files. `BODY[]` (non-PEEK) auto-sets `\Seen` per RFC 3501 | |
| 153 | - **UID STORE**: Handles comma-separated UID sets (e.g., `83,85,88,90`) by converting UIDs to sequence numbers via `uidSetToSeqSet()` | |
| 154 | - **Password hashing**: Argon2id with 64MB memory, 3 iterations, p=1 (format: `$argon2id$v=19$m=65536,t=3,p=1$<salt>$<hash>`) | |
| 155 | - **SQLite**: Primary database for user accounts, UID mappings, sessions | |
| 156 | - **TLS**: Custom TLS 1.3 implementation via zig-tls dependency | |
| 157 | ||
| 158 | ## Testing | |
| 159 | ||
| 160 | ```bash | |
| 161 | cd packages/zig && zig build test | |
| 162 | ``` | |
| 163 | ||
| 164 | Tests are split across multiple binaries (main, auth, imap, protocol, config, connection_wrapper, etc.). The logging tests produce stderr output which Zig's test runner reports as warnings — this is expected behavior, not a failure. | |
| 165 | ||
| 166 | Test binaries link against `sqlite3` system library. The argon2 password tests use `p=1` to avoid needing async I/O vtable (which is uninitialized in the test runner). | |
| 167 | ||
| 168 | ## Common Tasks | |
| 169 | ||
| 170 | - **Add a user**: `mail-server user add <email> <password>` (on server) | |
| 171 | - **Check service**: `systemctl status mail` (on server via SSM) | |
| 172 | - **View IMAP logs**: `journalctl -u mail | grep IMAP` | |
| 173 | - **Backup**: `mail-server backup create` → S3 bucket `stacks-production-s3-backups` | |
| 174 | ||
| 175 | --- | |
| 176 | ||
| 177 | ## Linting | |
| 178 | ||
| 179 | - Use **pickier** for linting — never use eslint directly | |
| 180 | - Run `bunx --bun pickier .` to lint, `bunx --bun pickier . --fix` to auto-fix | |
| 181 | - When fixing unused variable warnings, prefer `// eslint-disable-next-line` comments over prefixing with `_` | |
| 182 | ||
| 183 | ## Frontend | |
| 184 | ||
| 185 | - Use **stx** for templating — never write vanilla JS (`var`, `document.*`, `window.*`) in stx templates | |
| 186 | - Use **crosswind** as the default CSS framework which enables standard Tailwind-like utility classes | |
| 187 | - stx `<script>` tags should only contain stx-compatible code (signals, composables, directives) | |
| 188 | ||
| 189 | ## Dependencies | |
| 190 | ||
| 191 | - **buddy-bot** handles dependency updates — not renovatebot | |
| 192 | - **better-dx** provides shared dev tooling as peer dependencies — do not install its peers (e.g., `typescript`, `pickier`, `bun-plugin-dtsx`) separately if `better-dx` is already in `package.json` | |
| 193 | - If `better-dx` is in `package.json`, ensure `bunfig.toml` includes `linker = "hoisted"` | |