also looking at this
feat(antispam): harden inbound spam detection
#3
3 files
+670
-90
| @@ -0,0 +1,193 @@ | ||
| 1 | # Mail Server | |
| 2 | ||
| 3 | A performant, self-hosted mail server written in Zig. Supports SMTP, IMAP, POP3, CalDAV/CardDAV, and more. | |
| 4 | ||
| 5 | ## Project Structure | |
| 6 | ||
| 7 | ``` | |
| 8 | mail/ | |
| 9 | ├── packages/ | |
| 10 | │ ├── zig/ # Core mail server (Zig 0.16.0-dev) | |
| 11 | │ │ ├── src/ | |
| 12 | │ │ │ ├── main.zig # Server entry point | |
| 13 | │ │ │ ├── mail_cli.zig # CLI entry point (zig-cli) | |
| 14 | │ │ │ ├── core/ # Core: config, logging, protocol, TLS, sockets | |
| 15 | │ │ │ ├── protocol/ # IMAP, POP3, CalDAV, ActiveSync, etc. | |
| 16 | │ │ │ ├── auth/ # Auth, password hashing (Argon2id), CSRF | |
| 17 | │ │ │ ├── storage/ # SQLite database layer | |
| 18 | │ │ │ ├── delivery/ # Queue, bounce handling, TLS-RPT | |
| 19 | │ │ │ ├── antispam/ # DKIM, SPF, DMARC, ARC, DNSBL | |
| 20 | │ │ │ ├── observability/ # Logging, metrics, tracing, Discord alerts | |
| 21 | │ │ │ ├── features/ # Sieve, quotas, templates, autoresponder | |
| 22 | │ │ │ ├── infrastructure/ # Clustering, io_uring, connection pooling | |
| 23 | │ │ │ └── api/ # REST API, health checks | |
| 24 | │ │ ├── build.zig | |
| 25 | │ │ └── pantry/ # Zig dependencies (zig-tls, zig-cli) | |
| 26 | │ ├── cloud/ # AWS infrastructure (ts-cloud / CloudFormation) | |
| 27 | │ │ └── cloud.config.ts # EC2, SES, Route53, IAM config | |
| 28 | │ └── ts/ # TypeScript SDK (ts-mail) | |
| 29 | ├── pantry.jsonc # Monorepo config (workspaces, scripts) | |
| 30 | └── docs/ # Architecture, security, protocol docs | |
| 31 | ``` | |
| 32 | ||
| 33 | ## Build & Development | |
| 34 | ||
| 35 | ```bash | |
| 36 | # Package manager is `pantry` (like npm/bun) | |
| 37 | pantry run build # ReleaseFast build | |
| 38 | pantry run build:debug # Debug build | |
| 39 | pantry run dev # Build + run server locally | |
| 40 | pantry run test # Run all tests | |
| 41 | pantry run fmt # Format Zig source | |
| 42 | ||
| 43 | # Or directly from packages/zig: | |
| 44 | cd packages/zig | |
| 45 | zig build # Debug build | |
| 46 | zig build -Doptimize=ReleaseFast # Release build | |
| 47 | zig build -Dtarget=x86_64-linux # Cross-compile for Linux | |
| 48 | zig build test # Run tests | |
| 49 | zig build run -- serve # Run server | |
| 50 | ``` | |
| 51 | ||
| 52 | ## Zig 0.16 Specifics | |
| 53 | ||
| 54 | This project uses Zig 0.16.0-dev which has breaking changes from 0.15: | |
| 55 | ||
| 56 | - `std.ArrayList` is unmanaged: init with `.empty`, pass allocator to every method (`append(allocator, ...)`, `deinit(allocator)`) | |
| 57 | - `std.time.sleep` removed: use `time_compat.sleep()` (our wrapper using `std.c.nanosleep`) | |
| 58 | - `std.fs.cwd()` removed: use `fs_compat` module for file operations | |
| 59 | - `std.process.Child.run` removed: use `extern "c" fn system()` or C file I/O | |
| 60 | - `std.c.stat/remove/system` removed: declare `extern "c"` functions directly | |
| 61 | - Argon2 password hashing uses `p=1` (single-threaded) to avoid async I/O requirements in tests | |
| 62 | - Custom compat layers: `src/core/io_compat.zig`, `src/core/fs_compat.zig`, `src/core/time_compat.zig`, `src/core/socket_compat.zig` | |
| 63 | ||
| 64 | ## Production Server | |
| 65 | ||
| 66 | - **Instance**: `i-0e365c6bd31da4678` (EC2, Amazon Linux 2023, us-east-1) | |
| 67 | - **Domain**: `mail.stacksjs.com` | |
| 68 | - **Binary**: `/opt/mail/mail-server` (renamed from `mail` to avoid conflict with `mail/` mailbox dir) | |
| 69 | - **Mailboxes**: `/opt/mail/mail/{username}/new/` (Maildir format) | |
| 70 | - **Database**: `/opt/mail/smtp.db` (SQLite) | |
| 71 | - **Config**: `/etc/mail/mail.env` | |
| 72 | - **TLS**: Let's Encrypt at `/etc/letsencrypt/live/mail.stacksjs.com/` | |
| 73 | - **Service**: `mail.service` (systemd, runs as `mail-server` user with `CAP_NET_BIND_SERVICE`) | |
| 74 | - **Delivery**: Amazon SES (`SMTP_DELIVERY_METHOD=ses`) | |
| 75 | - **Ports**: 25 (SMTP), 465 (SMTPS), 587 (Submission), 143 (IMAP), 993 (IMAPS) | |
| 76 | ||
| 77 | ## Deployment via AWS SSM | |
| 78 | ||
| 79 | Deploy new builds without SSH. The server has SSM agent running and an IAM role with SSM permissions. | |
| 80 | ||
| 81 | ```bash | |
| 82 | # 1. Cross-compile for Linux | |
| 83 | cd packages/zig | |
| 84 | zig build -Dtarget=x86_64-linux | |
| 85 | ||
| 86 | # 2. Upload binary to S3 | |
| 87 | aws s3 cp zig-out/bin/mail s3://stacks-production-s3-email/deploy/mail-server-new | |
| 88 | ||
| 89 | # 3. Deploy via SSM (stop service, swap binary, restart) | |
| 90 | aws ssm send-command \ | |
| 91 | --instance-ids i-0e365c6bd31da4678 \ | |
| 92 | --document-name "AWS-RunShellScript" \ | |
| 93 | --parameters 'commands=[ | |
| 94 | "aws s3 cp s3://stacks-production-s3-email/deploy/mail-server-new /tmp/mail-server-new", | |
| 95 | "chmod +x /tmp/mail-server-new", | |
| 96 | "systemctl stop mail", | |
| 97 | "cp /opt/mail/mail-server /opt/mail/mail-server.bak", | |
| 98 | "mv /tmp/mail-server-new /opt/mail/mail-server", | |
| 99 | "chown mail-server:mail-server /opt/mail/mail-server", | |
| 100 | "systemctl start mail", | |
| 101 | "sleep 2", | |
| 102 | "systemctl is-active mail" | |
| 103 | ]' | |
| 104 | ||
| 105 | # 4. Check result | |
| 106 | aws ssm get-command-invocation \ | |
| 107 | --command-id <COMMAND_ID> \ | |
| 108 | --instance-id i-0e365c6bd31da4678 | |
| 109 | ||
| 110 | # View logs | |
| 111 | aws ssm send-command \ | |
| 112 | --instance-ids i-0e365c6bd31da4678 \ | |
| 113 | --document-name "AWS-RunShellScript" \ | |
| 114 | --parameters 'commands=["journalctl -u mail --no-pager -n 100"]' | |
| 115 | ``` | |
| 116 | ||
| 117 | The old host key may need updating after instance rebuilds: | |
| 118 | ```bash | |
| 119 | ssh-keygen -R mail.stacksjs.com | |
| 120 | ``` | |
| 121 | ||
| 122 | ## Cloud Infrastructure (packages/cloud) | |
| 123 | ||
| 124 | Defined in `cloud.config.ts` using ts-cloud (CloudFormation wrapper): | |
| 125 | ||
| 126 | - **EC2**: t3 instance with security groups for mail ports | |
| 127 | - **SES**: Domain verification, DKIM signing | |
| 128 | - **Route 53**: MX, A, SPF, DMARC, DKIM DNS records | |
| 129 | - **IAM**: Role with SES, S3, Route53, SSM permissions | |
| 130 | - **S3**: `stacks-production-s3-email` for deployments and backups | |
| 131 | - **User data script** handles: Zig install, build from git, Let's Encrypt, fail2ban, systemd service, log rotation, certbot renewal | |
| 132 | ||
| 133 | Deploy infrastructure: | |
| 134 | ```bash | |
| 135 | pantry run deploy # Deploy CloudFormation stack | |
| 136 | pantry run cloud:status # Check stack status | |
| 137 | pantry run cloud:diff # Preview changes | |
| 138 | ``` | |
| 139 | ||
| 140 | ## Discord Health Monitoring | |
| 141 | ||
| 142 | The server includes a background health monitor (`src/observability/discord.zig`) that sends alerts to Discord via webhook. Checks include: active connections, database health, TLS cert expiry, disk space, and heartbeat. | |
| 143 | ||
| 144 | Configure via environment variable: | |
| 145 | ``` | |
| 146 | DISCORD_WEBHOOK_URL=https://discord.com/api/webhooks/... | |
| 147 | ``` | |
| 148 | ||
| 149 | ## Key Architecture Decisions | |
| 150 | ||
| 151 | - **Maildir format**: Messages stored as individual files with `:2,FLAGS` suffix (S=Seen, R=Answered, F=Flagged, D=Draft, T=Deleted) | |
| 152 | - **IMAP flag persistence**: Flags are persisted by renaming maildir files. `BODY[]` (non-PEEK) auto-sets `\Seen` per RFC 3501 | |
| 153 | - **UID STORE**: Handles comma-separated UID sets (e.g., `83,85,88,90`) by converting UIDs to sequence numbers via `uidSetToSeqSet()` | |
| 154 | - **Password hashing**: Argon2id with 64MB memory, 3 iterations, p=1 (format: `$argon2id$v=19$m=65536,t=3,p=1$<salt>$<hash>`) | |
| 155 | - **SQLite**: Primary database for user accounts, UID mappings, sessions | |
| 156 | - **TLS**: Custom TLS 1.3 implementation via zig-tls dependency | |
| 157 | ||
| 158 | ## Testing | |
| 159 | ||
| 160 | ```bash | |
| 161 | cd packages/zig && zig build test | |
| 162 | ``` | |
| 163 | ||
| 164 | Tests are split across multiple binaries (main, auth, imap, protocol, config, connection_wrapper, etc.). The logging tests produce stderr output which Zig's test runner reports as warnings — this is expected behavior, not a failure. | |
| 165 | ||
| 166 | Test binaries link against `sqlite3` system library. The argon2 password tests use `p=1` to avoid needing async I/O vtable (which is uninitialized in the test runner). | |
| 167 | ||
| 168 | ## Common Tasks | |
| 169 | ||
| 170 | - **Add a user**: `mail-server user add <email> <password>` (on server) | |
| 171 | - **Check service**: `systemctl status mail` (on server via SSM) | |
| 172 | - **View IMAP logs**: `journalctl -u mail | grep IMAP` | |
| 173 | - **Backup**: `mail-server backup create` → S3 bucket `stacks-production-s3-backups` | |
| 174 | ||
| 175 | --- | |
| 176 | ||
| 177 | ## Linting | |
| 178 | ||
| 179 | - Use **pickier** for linting — never use eslint directly | |
| 180 | - Run `bunx --bun pickier .` to lint, `bunx --bun pickier . --fix` to auto-fix | |
| 181 | - When fixing unused variable warnings, prefer `// eslint-disable-next-line` comments over prefixing with `_` | |
| 182 | ||
| 183 | ## Frontend | |
| 184 | ||
| 185 | - Use **stx** for templating — never write vanilla JS (`var`, `document.*`, `window.*`) in stx templates | |
| 186 | - Use **crosswind** as the default CSS framework which enables standard Tailwind-like utility classes | |
| 187 | - stx `<script>` tags should only contain stx-compatible code (signals, composables, directives) | |
| 188 | ||
| 189 | ## Dependencies | |
| 190 | ||
| 191 | - **buddy-bot** handles dependency updates — not renovatebot | |
| 192 | - **better-dx** provides shared dev tooling as peer dependencies — do not install its peers (e.g., `typescript`, `pickier`, `bun-plugin-dtsx`) separately if `better-dx` is already in `package.json` | |
| 193 | - If `better-dx` is in `package.json`, ensure `bunfig.toml` includes `linker = "hoisted"` | |
| @@ -108,30 +108,46 @@ const url_shorteners = [_][]const u8{ | ||
| 108 | 108 | /// Attachment name suffixes that are almost always malicious in email. |
| 109 | 109 | const dangerous_exts = [_][]const u8{ |
| 110 | 110 | ".exe", ".scr", ".pif", ".bat", ".cmd", ".com", ".vbs", |
| 111 | ".js", ".jar", ".cpl", ".lnk", ".iso", ".hta", | |
| 111 | ".js", ".jar", ".cpl", ".lnk", ".iso", ".hta", ".msi", | |
| 112 | ".reg", ".ps1", ".wsf", ".chm", ".img", | |
| 112 | 113 | }; |
| 113 | 114 | |
| 115 | fn hasDangerousSuffix(value: []const u8) bool { | |
| 116 | for (dangerous_exts) |ext| { | |
| 117 | if (value.len >= ext.len and | |
| 118 | std.ascii.eqlIgnoreCase(value[value.len - ext.len ..], ext)) return true; | |
| 119 | ||
| 120 | // RFC 2231 filenames commonly percent-encode the dot, e.g. | |
| 121 | // filename*=UTF-8''invoice%2Eexe. | |
| 122 | if (value.len >= ext.len + 2 and value[value.len - ext.len - 2] == '%') { | |
| 123 | const hi = hexVal(value[value.len - ext.len - 1]) orelse continue; | |
| 124 | const lo = hexVal(value[value.len - ext.len]) orelse continue; | |
| 125 | if ((hi << 4) | lo == '.' and | |
| 126 | std.ascii.eqlIgnoreCase(value[value.len - ext.len + 1 ..], ext[1..])) return true; | |
| 127 | } | |
| 128 | } | |
| 129 | return false; | |
| 130 | } | |
| 131 | ||
| 114 | 132 | /// True if a MIME `name=` / `filename=` parameter declares a file whose |
| 115 | 133 | /// extension is in `dangerous_exts`. Only the parameter *value* is examined, |
| 116 | 134 | /// so link URLs in the body (e.g. `href="https://example.com"`) cannot |
| 117 | 135 | /// false-match a bare extension like ".com". Matching `name=` also covers |
| 118 | 136 | /// `filename=` since that string ends in `name=`. |
| 119 | 137 | fn hasDangerousAttachment(scan: []const u8) bool { |
| 120 | var pos: usize = 0; | |
| 121 | while (std.ascii.indexOfIgnoreCasePos(scan, pos, "name=")) |at| { | |
| 122 | var v = at + "name=".len; | |
| 123 | pos = v; // advance past this match for the next iteration | |
| 124 | if (v < scan.len and (scan[v] == '"' or scan[v] == '\'')) v += 1; | |
| 125 | var end = v; | |
| 126 | while (end < scan.len) : (end += 1) { | |
| 127 | const c = scan[end]; | |
| 128 | if (c == '"' or c == '\'' or c == ';' or c == '\r' or c == '\n' or c == ' ') break; | |
| 129 | } | |
| 130 | const value = scan[v..end]; | |
| 131 | for (dangerous_exts) |ext| { | |
| 132 | if (value.len >= ext.len and | |
| 133 | std.ascii.eqlIgnoreCase(value[value.len - ext.len ..], ext)) | |
| 134 | return true; | |
| 138 | const markers = [_][]const u8{ "name=", "name*=" }; | |
| 139 | for (markers) |marker| { | |
| 140 | var pos: usize = 0; | |
| 141 | while (std.ascii.indexOfIgnoreCasePos(scan, pos, marker)) |at| { | |
| 142 | var v = at + marker.len; | |
| 143 | pos = v; | |
| 144 | if (v < scan.len and (scan[v] == '"' or scan[v] == '\'')) v += 1; | |
| 145 | var end = v; | |
| 146 | while (end < scan.len) : (end += 1) { | |
| 147 | const c = scan[end]; | |
| 148 | if (c == '"' or c == ';' or c == '\r' or c == '\n' or c == ' ') break; | |
| 149 | } | |
| 150 | if (hasDangerousSuffix(scan[v..end])) return true; | |
| 135 | 151 | } |
| 136 | 152 | } |
| 137 | 153 | return false; |
| @@ -143,21 +159,24 @@ fn hasDangerousAttachment(scan: []const u8) bool { | ||
| 143 | 159 | /// mail doesn't score; a genuine pitch stacks several. Individually low-weight |
| 144 | 160 | /// and capped, like `spam_phrases`. |
| 145 | 161 | const seo_phrases = [_][]const u8{ |
| 146 | "search engine optimization", "seo service", "seo services", "seo package", | |
| 147 | "seo expert", "seo company", "seo agency", "1st page of google", | |
| 148 | "first page of google", "top of google", "page of google", "google ranking", | |
| 149 | "search ranking", "increase your ranking", "improve your ranking", "rank higher", | |
| 150 | "higher ranking", "website traffic", "organic traffic", "increase traffic", | |
| 151 | "website design", "web design service", "website development", "web development", | |
| 152 | "web developer", "website redesign", "redesign of your", "redesign your website", | |
| 153 | "revamp your website", "existing website", "build your website", "backlink", | |
| 154 | "link building", "guest post", "domain authority", "digital marketing", | |
| 155 | "lead generation", "se0", | |
| 162 | "search engine optimization", "seo service", "seo services", "seo package", | |
| 163 | "seo expert", "seo company", "seo agency", "1st page of google", | |
| 164 | "first page of google", "top of google", "page of google", "google ranking", | |
| 165 | "search ranking", "increase your ranking", "improve your ranking", "rank higher", | |
| 166 | "higher ranking", "website traffic", "organic traffic", "increase traffic", | |
| 167 | "website visibility", "visibility on google", "optimized for search engines", "improving rankings", | |
| 168 | "website ranking", "errors at your website", "website errors", "fix errors", | |
| 169 | "screenshots of these errors", "website design", "web design service", "website development", | |
| 170 | "web development", "web developer", "website redesign", "redesign of your", | |
| 171 | "redesign your website", "revamp your website", "existing website", "build your website", | |
| 172 | "backlink", "link building", "guest post", "domain authority", | |
| 173 | "digital marketing", "lead generation", "se0", | |
| 156 | 174 | // App-development pitches, the sibling of the SEO pitch ("interested in |
| 157 | 175 | // building a mobile app?"). Same bigram discipline: no bare "app". |
| 158 | "mobile app", "app development", | |
| 159 | "app developer", "application development", "build an app", "building an app", | |
| 160 | "develop an app", "ios and android", "android and ios", "app idea", | |
| 176 | "mobile app", | |
| 177 | "app development", "app developer", "application development", "build an app", | |
| 178 | "building an app", "develop an app", "ios and android", "android and ios", | |
| 179 | "app idea", | |
| 161 | 180 | }; |
| 162 | 181 | |
| 163 | 182 | /// Cold-outreach scaffolding: the opener/closer boilerplate every unsolicited |
| @@ -200,6 +219,27 @@ const spoofed_brands = [_][]const u8{ | ||
| 200 | 219 | "binance", "fedex", "norton", "mcafee", "docusign", |
| 201 | 220 | }; |
| 202 | 221 | |
| 222 | /// Parcel-fee phishing impersonates a carrier while sending from an unrelated, | |
| 223 | /// but often perfectly SPF/DKIM/DMARC-authenticated, compromised domain. It is | |
| 224 | /// important that these are conjunctions: ordinary shipment notices mention a | |
| 225 | /// carrier, and ordinary invoices mention payment, but the scam stacks several | |
| 226 | /// urgent fee/payment prompts while claiming a carrier identity. | |
| 227 | const carriers = [_]struct { brand: []const u8, domain: []const u8 }{ | |
| 228 | .{ .brand = "dhl", .domain = "dhl.com" }, | |
| 229 | .{ .brand = "fedex", .domain = "fedex.com" }, | |
| 230 | .{ .brand = "ups", .domain = "ups.com" }, | |
| 231 | .{ .brand = "usps", .domain = "usps.com" }, | |
| 232 | }; | |
| 233 | const parcel_phrases = [_][]const u8{ | |
| 234 | "shipment held", "shipment release", "customs documentation", | |
| 235 | "customs fee", "final delivery", "local facility", | |
| 236 | }; | |
| 237 | const payment_phrases = [_][]const u8{ | |
| 238 | "fee required", "fee must be paid", "amount due", | |
| 239 | "awaiting payment", "make payment", "payment information", | |
| 240 | "complete the process", "release fee", | |
| 241 | }; | |
| 242 | ||
| 203 | 243 | fn hexVal(c: u8) ?u8 { |
| 204 | 244 | return switch (c) { |
| 205 | 245 | '0'...'9' => c - '0', |
| @@ -255,6 +295,9 @@ fn matchEntity(s: []const u8) ?struct { ch: u8, len: usize } { | ||
| 255 | 295 | .{ .name = ">", .ch = '>' }, |
| 256 | 296 | .{ .name = """, .ch = '"' }, |
| 257 | 297 | .{ .name = "'", .ch = '\'' }, |
| 298 | .{ .name = " ", .ch = ' ' }, | |
| 299 | .{ .name = " ", .ch = ' ' }, | |
| 300 | .{ .name = " ", .ch = ' ' }, | |
| 258 | 301 | }; |
| 259 | 302 | for (table) |e| { |
| 260 | 303 | if (std.ascii.startsWithIgnoreCase(s, e.name)) return .{ .ch = e.ch, .len = e.name.len }; |
| @@ -273,6 +316,18 @@ fn stripHtmlText(in: []const u8, out: []u8) []const u8 { | ||
| 273 | 316 | var pending_space = false; |
| 274 | 317 | while (i < in.len and o < out.len) { |
| 275 | 318 | var c = in[i]; |
| 319 | // Zero-width Unicode characters and their common HTML encodings are | |
| 320 | // invisible in clients but otherwise split phrases such as "payment". | |
| 321 | const zero_width = [_][]const u8{ "\xE2\x80\x8B", "\xE2\x80\x8C", "\xE2\x80\x8D", "​", "​", "​" }; | |
| 322 | var skipped_zero_width = false; | |
| 323 | for (zero_width) |zw| { | |
| 324 | if (std.mem.startsWith(u8, in[i..], zw)) { | |
| 325 | i += zw.len; | |
| 326 | skipped_zero_width = true; | |
| 327 | break; | |
| 328 | } | |
| 329 | } | |
| 330 | if (skipped_zero_width) continue; | |
| 276 | 331 | if (c == '<') { |
| 277 | 332 | if (std.ascii.startsWithIgnoreCase(in[i..], "<style")) { |
| 278 | 333 | if (std.ascii.indexOfIgnoreCase(in[i..], "</style")) |rel| { |
| @@ -320,15 +375,16 @@ fn stripHtmlText(in: []const u8, out: []u8) []const u8 { | ||
| 320 | 375 | /// window and decode its payload (bounded by `out`). Wrapping the pitch in a |
| 321 | 376 | /// base64 text/html part is the oldest filter-evasion trick there is; without |
| 322 | 377 | /// this the phrase scan only ever sees boundary noise. |
| 323 | fn decodeBase64Part(window: []const u8, out: []u8) []const u8 { | |
| 378 | fn decodeBase64Parts(window: []const u8, out: []u8) []const u8 { | |
| 324 | 379 | const marker = "content-transfer-encoding:"; |
| 325 | 380 | var search: usize = 0; |
| 381 | var out_len: usize = 0; | |
| 326 | 382 | while (search < window.len) { |
| 327 | const rel = std.ascii.indexOfIgnoreCase(window[search..], marker) orelse return out[0..0]; | |
| 383 | const rel = std.ascii.indexOfIgnoreCase(window[search..], marker) orelse break; | |
| 328 | 384 | const vstart = search + rel + marker.len; |
| 329 | 385 | const line_end = std.mem.indexOfScalarPos(u8, window, vstart, '\n') orelse window.len; |
| 330 | 386 | if (std.ascii.indexOfIgnoreCase(window[vstart..line_end], "base64") == null) { |
| 331 | search = line_end; | |
| 387 | search = @min(line_end + 1, window.len); | |
| 332 | 388 | continue; |
| 333 | 389 | } |
| 334 | 390 | // Skip the remaining part headers up to the blank line. |
| @@ -357,14 +413,19 @@ fn decodeBase64Part(window: []const u8, out: []u8) []const u8 { | ||
| 357 | 413 | n += 1; |
| 358 | 414 | } |
| 359 | 415 | } |
| 416 | search = @min(p + 1, window.len); | |
| 360 | 417 | n -= n % 4; // drop a trailing partial quantum (window cut mid-line) |
| 361 | if (n == 0) return out[0..0]; | |
| 362 | const dec_len = std.base64.standard.Decoder.calcSizeForSlice(collected[0..n]) catch return out[0..0]; | |
| 363 | if (dec_len > out.len) return out[0..0]; | |
| 364 | std.base64.standard.Decoder.decode(out[0..dec_len], collected[0..n]) catch return out[0..0]; | |
| 365 | return out[0..dec_len]; | |
| 418 | if (n == 0) continue; | |
| 419 | const dec_len = std.base64.standard.Decoder.calcSizeForSlice(collected[0..n]) catch continue; | |
| 420 | if (dec_len > out.len - out_len) break; | |
| 421 | std.base64.standard.Decoder.decode(out[out_len .. out_len + dec_len], collected[0..n]) catch continue; | |
| 422 | out_len += dec_len; | |
| 423 | if (out_len < out.len) { | |
| 424 | out[out_len] = '\n'; | |
| 425 | out_len += 1; | |
| 426 | } | |
| 366 | 427 | } |
| 367 | return out[0..0]; | |
| 428 | return out[0..out_len]; | |
| 368 | 429 | } |
| 369 | 430 | |
| 370 | 431 | /// Decode RFC 2047 encoded-words ("=?utf-8?B?...?=" / "=?utf-8?Q?...?=") so |
| @@ -496,9 +557,15 @@ pub fn evaluate(signals: Signals, headers: []const u8, body: []const u8, thresho | ||
| 496 | 557 | // === 3. Message structure === |
| 497 | 558 | // Subject heuristics run on the RFC 2047-decoded form — encoded-word |
| 498 | 559 | // subjects otherwise sail past every subject rule. |
| 499 | var subj_buf: [256]u8 = undefined; | |
| 500 | const subject_raw = headerValue(headers, "Subject"); | |
| 560 | var subj_raw_buf: [1024]u8 = undefined; | |
| 561 | var subj_buf: [1024]u8 = undefined; | |
| 562 | const subject_raw = headerValueUnfolded(headers, "Subject", &subj_raw_buf); | |
| 501 | 563 | const subject: ?[]const u8 = if (subject_raw) |s| decodeEncodedWords(s, &subj_buf) else null; |
| 564 | var from_raw_buf: [1024]u8 = undefined; | |
| 565 | var from_decoded_buf: [1024]u8 = undefined; | |
| 566 | const from_raw = headerValueUnfolded(headers, "From", &from_raw_buf); | |
| 567 | const from_decoded: ?[]const u8 = if (from_raw) |s| decodeEncodedWords(s, &from_decoded_buf) else null; | |
| 568 | const from_domain = if (from_raw) |s| domainOfHeaderValue(s) else null; | |
| 502 | 569 | if (subject == null or std.mem.trim(u8, subject.?, " \t").len == 0) { |
| 503 | 570 | score += 0.8; |
| 504 | 571 | tl.add("NO_SUBJECT"); |
| @@ -555,14 +622,13 @@ pub fn evaluate(signals: Signals, headers: []const u8, body: []const u8, thresho | ||
| 555 | 622 | // claims a consumer brand the sending domain has nothing to do with |
| 556 | 623 | // (e.g. `"Netflix.com" <l1xwv4u9@aqg.io>`). Independent of DMARC — the |
| 557 | 624 | // display name is never authenticated even when the domain's DMARC passes. |
| 558 | if (headerValue(headers, "From")) |from_hdr| { | |
| 625 | if (from_decoded) |from_hdr| { | |
| 559 | 626 | const disp_end = std.mem.indexOfScalar(u8, from_hdr, '<') orelse from_hdr.len; |
| 560 | 627 | const display = from_hdr[0..disp_end]; |
| 561 | const from_dom = headerDomain(headers, "From"); | |
| 562 | 628 | for (spoofed_brands) |brand| { |
| 563 | 629 | if (std.ascii.indexOfIgnoreCase(display, brand) != null) { |
| 564 | const dom_has_brand = if (from_dom) |fd| | |
| 565 | std.ascii.indexOfIgnoreCase(fd, brand) != null | |
| 630 | const dom_has_brand = if (from_domain) |fd| | |
| 631 | domainHasLabel(fd, brand) | |
| 566 | 632 | else |
| 567 | 633 | false; |
| 568 | 634 | if (!dom_has_brand) { |
| @@ -576,10 +642,11 @@ pub fn evaluate(signals: Signals, headers: []const u8, body: []const u8, thresho | ||
| 576 | 642 | |
| 577 | 643 | // From / Reply-To domain mismatch (only when DMARC didn't already pass). |
| 578 | 644 | if (signals.dmarc != .pass) { |
| 579 | const from_dom = headerDomain(headers, "From"); | |
| 580 | const reply_dom = headerDomain(headers, "Reply-To"); | |
| 581 | if (from_dom != null and reply_dom != null and | |
| 582 | !std.ascii.eqlIgnoreCase(from_dom.?, reply_dom.?)) | |
| 645 | var reply_raw_buf: [1024]u8 = undefined; | |
| 646 | const reply_raw = headerValueUnfolded(headers, "Reply-To", &reply_raw_buf); | |
| 647 | const reply_domain = if (reply_raw) |s| domainOfHeaderValue(s) else null; | |
| 648 | if (from_domain != null and reply_domain != null and | |
| 649 | !std.ascii.eqlIgnoreCase(from_domain.?, reply_domain.?)) | |
| 583 | 650 | { |
| 584 | 651 | score += 0.7; |
| 585 | 652 | tl.add("REPLYTO_MISMATCH"); |
| @@ -587,7 +654,9 @@ pub fn evaluate(signals: Signals, headers: []const u8, body: []const u8, thresho | ||
| 587 | 654 | } |
| 588 | 655 | |
| 589 | 656 | // Phrase + structural body heuristics (scan subject + start of body). |
| 590 | const scan_body = body[0..@min(body.len, 8192)]; | |
| 657 | // A larger bounded window prevents trivial padding (large comments/style | |
| 658 | // blocks before the pitch) from pushing content beyond the old 8 KiB scan. | |
| 659 | const scan_body = body[0..@min(body.len, 16384)]; | |
| 591 | 660 | |
| 592 | 661 | // Phrase scans run on a NORMALIZED view of the body, not the raw bytes: |
| 593 | 662 | // quoted-printable is decoded (soft breaks rejoin split phrases), HTML |
| @@ -595,13 +664,13 @@ pub fn evaluate(signals: Signals, headers: []const u8, body: []const u8, thresho | ||
| 595 | 664 | // base64-encoded MIME part is decoded and appended. Raw-byte scans stay |
| 596 | 665 | // raw where the markup itself is the signal (URLs, hidden-text CSS, |
| 597 | 666 | // attachment filenames). |
| 598 | var qp_buf: [8192]u8 = undefined; | |
| 599 | var text_buf: [8192]u8 = undefined; | |
| 600 | var b64_buf: [6144]u8 = undefined; | |
| 601 | var b64_text_buf: [6144]u8 = undefined; | |
| 667 | var qp_buf: [16384]u8 = undefined; | |
| 668 | var text_buf: [16384]u8 = undefined; | |
| 669 | var b64_buf: [12288]u8 = undefined; | |
| 670 | var b64_text_buf: [12288]u8 = undefined; | |
| 602 | 671 | const scan_qp = decodeQuotedPrintable(scan_body, &qp_buf); |
| 603 | 672 | const scan_text = stripHtmlText(scan_qp, &text_buf); |
| 604 | const b64_part = decodeBase64Part(scan_body, &b64_buf); | |
| 673 | const b64_part = decodeBase64Parts(scan_body, &b64_buf); | |
| 605 | 674 | const b64_text = if (b64_part.len > 0) stripHtmlText(b64_part, &b64_text_buf) else b64_part; |
| 606 | 675 | |
| 607 | 676 | const Scan = struct { |
| @@ -618,6 +687,37 @@ pub fn evaluate(signals: Signals, headers: []const u8, body: []const u8, thresho | ||
| 618 | 687 | }; |
| 619 | 688 | const scan = Scan{ .subject = subject, .text = scan_text, .b64 = b64_text }; |
| 620 | 689 | |
| 690 | // Authenticated-domain parcel phishing. DMARC proves only that the sender | |
| 691 | // controls (or compromised) its own domain; it does not make a claim to be | |
| 692 | // DHL/FedEx/UPS/USPS legitimate. Require a carrier/domain mismatch plus a | |
| 693 | // parcel phrase and at least two payment prompts to keep real tracking and | |
| 694 | // invoice messages out of Junk. | |
| 695 | var claimed_carrier: ?@TypeOf(carriers[0]) = null; | |
| 696 | for (carriers) |carrier| { | |
| 697 | if (scanHitWord(scan, carrier.brand)) { | |
| 698 | claimed_carrier = carrier; | |
| 699 | break; | |
| 700 | } | |
| 701 | } | |
| 702 | var parcel_hits: usize = 0; | |
| 703 | for (parcel_phrases) |phrase| { | |
| 704 | if (scan.hit(phrase)) parcel_hits += 1; | |
| 705 | } | |
| 706 | var payment_hits: usize = 0; | |
| 707 | for (payment_phrases) |phrase| { | |
| 708 | if (scan.hit(phrase)) payment_hits += 1; | |
| 709 | } | |
| 710 | if (claimed_carrier) |carrier| { | |
| 711 | const from_matches = if (from_domain) |fd| | |
| 712 | domainIsOrSubdomain(fd, carrier.domain) | |
| 713 | else | |
| 714 | false; | |
| 715 | if (!from_matches and parcel_hits > 0 and payment_hits >= 2) { | |
| 716 | score += 6.0; | |
| 717 | tl.add("CARRIER_PAYMENT_PHISH"); | |
| 718 | } | |
| 719 | } | |
| 720 | ||
| 621 | 721 | var phrase_hits: f32 = 0; |
| 622 | 722 | for (spam_phrases) |phrase| { |
| 623 | 723 | if (scan.hit(phrase)) phrase_hits += 0.6; |
| @@ -673,7 +773,7 @@ pub fn evaluate(signals: Signals, headers: []const u8, body: []const u8, thresho | ||
| 673 | 773 | // scaffolding alone needs 2+ distinct phrases before it counts as |
| 674 | 774 | // solicitation content here. |
| 675 | 775 | if (seo_hits > 0 or harvest_hit or outreach_count >= 2) { |
| 676 | if (headerDomain(headers, "From")) |fd| { | |
| 776 | if (from_domain) |fd| { | |
| 677 | 777 | for (free_providers) |fp| { |
| 678 | 778 | if (std.ascii.eqlIgnoreCase(fd, fp)) { |
| 679 | 779 | score += 2.0; |
| @@ -687,7 +787,10 @@ pub fn evaluate(signals: Signals, headers: []const u8, body: []const u8, thresho | ||
| 687 | 787 | // Link farm: a wall of URLs is a strong bulk-mail signal. Scanned on the |
| 688 | 788 | // QP-decoded view so a soft line break can't split "https://" or a |
| 689 | 789 | // shortener host. |
| 690 | const url_count = countOccurrences(scan_qp, "http://") + countOccurrences(scan_qp, "https://"); | |
| 790 | const url_count = countOccurrencesIgnoreCase(scan_qp, "http://") + | |
| 791 | countOccurrencesIgnoreCase(scan_qp, "https://") + | |
| 792 | countOccurrencesIgnoreCase(b64_part, "http://") + | |
| 793 | countOccurrencesIgnoreCase(b64_part, "https://"); | |
| 691 | 794 | if (url_count >= 20) { |
| 692 | 795 | score += 1.5; |
| 693 | 796 | tl.add("MANY_URLS"); |
formatting only · 12 hidden lines
| @@ -710,9 +813,9 @@ pub fn evaluate(signals: Signals, headers: []const u8, body: []const u8, thresho formatting only | ||
| 710 | 813 | // QP-decoded view: markup intact (not tag-stripped) but soft breaks unsplit. |
| 711 | 814 | if (signals.dmarc != .pass) { |
| 712 | 815 | const hidden_markers = [_][]const u8{ |
| 713 | "display:none", "display: none", | |
| 714 | "visibility:hidden", "visibility: hidden", | |
| 715 | "font-size:0", "font-size: 0", | |
| 816 | "display:none", "display: none", | |
| 817 | "visibility:hidden", "visibility: hidden", | |
| 818 | "font-size:0", "font-size: 0", | |
| 716 | 819 | }; |
| 717 | 820 | var hidden_hits: usize = 0; |
| 718 | 821 | for (hidden_markers) |m| hidden_hits += countOccurrencesIgnoreCase(scan_qp, m); |
| @@ -760,6 +863,40 @@ pub fn headerValue(headers: []const u8, name: []const u8) ?[]const u8 { | ||
| 760 | 863 | return null; |
| 761 | 864 | } |
| 762 | 865 | |
| 866 | /// Return a header value with RFC 5322 continuation lines unfolded into a | |
| 867 | /// caller-owned buffer. This closes a common evasion where a spam phrase is | |
| 868 | /// split immediately after `Subject:` onto a whitespace-prefixed line. | |
| 869 | fn headerValueUnfolded(headers: []const u8, name: []const u8, out: []u8) ?[]const u8 { | |
| 870 | var it = std.mem.splitScalar(u8, headers, '\n'); | |
| 871 | var found = false; | |
| 872 | var len: usize = 0; | |
| 873 | while (it.next()) |raw_line| { | |
| 874 | const line = std.mem.trimEnd(u8, raw_line, "\r"); | |
| 875 | if (!found) { | |
| 876 | if (line.len < name.len + 1 or | |
| 877 | !std.ascii.eqlIgnoreCase(line[0..name.len], name) or | |
| 878 | line[name.len] != ':') continue; | |
| 879 | const value = std.mem.trim(u8, line[name.len + 1 ..], " \t"); | |
| 880 | const n = @min(value.len, out.len); | |
| 881 | @memcpy(out[0..n], value[0..n]); | |
| 882 | len = n; | |
| 883 | found = true; | |
| 884 | continue; | |
| 885 | } | |
| 886 | if (line.len == 0 or (line[0] != ' ' and line[0] != '\t')) break; | |
| 887 | const continuation = std.mem.trim(u8, line, " \t"); | |
| 888 | if (continuation.len == 0 or len >= out.len) continue; | |
| 889 | if (len > 0) { | |
| 890 | out[len] = ' '; | |
| 891 | len += 1; | |
| 892 | } | |
| 893 | const n = @min(continuation.len, out.len - len); | |
| 894 | @memcpy(out[len .. len + n], continuation[0..n]); | |
| 895 | len += n; | |
| 896 | } | |
| 897 | return if (found) out[0..len] else null; | |
| 898 | } | |
| 899 | ||
| 763 | 900 | pub fn headerPresent(headers: []const u8, name: []const u8) bool { |
| 764 | 901 | return headerValue(headers, name) != null; |
| 765 | 902 | } |
| @@ -767,6 +904,10 @@ pub fn headerPresent(headers: []const u8, name: []const u8) bool { | ||
| 767 | 904 | /// Domain portion of an address-bearing header (e.g. From, Reply-To). |
| 768 | 905 | fn headerDomain(headers: []const u8, name: []const u8) ?[]const u8 { |
| 769 | 906 | const v = headerValue(headers, name) orelse return null; |
| 907 | return domainOfHeaderValue(v); | |
| 908 | } | |
| 909 | ||
| 910 | fn domainOfHeaderValue(v: []const u8) ?[]const u8 { | |
| 770 | 911 | const at = std.mem.lastIndexOfScalar(u8, v, '@') orelse return null; |
| 771 | 912 | var d = v[at + 1 ..]; |
| 772 | 913 | var end: usize = 0; |
| @@ -779,6 +920,48 @@ fn headerDomain(headers: []const u8, name: []const u8) ?[]const u8 { | ||
| 779 | 920 | return if (d.len == 0) null else d; |
| 780 | 921 | } |
| 781 | 922 | |
| 923 | fn domainIsOrSubdomain(domain: []const u8, expected: []const u8) bool { | |
| 924 | if (std.ascii.eqlIgnoreCase(domain, expected)) return true; | |
| 925 | return domain.len > expected.len and | |
| 926 | domain[domain.len - expected.len - 1] == '.' and | |
| 927 | std.ascii.eqlIgnoreCase(domain[domain.len - expected.len ..], expected); | |
| 928 | } | |
| 929 | ||
| 930 | /// Match a complete DNS label, not an arbitrary substring. Without label | |
| 931 | /// boundaries a sender like `paypal-security.example` could evade display-name | |
| 932 | /// spoof detection merely by putting the impersonated brand in its domain. | |
| 933 | fn domainHasLabel(domain: []const u8, label: []const u8) bool { | |
| 934 | var it = std.mem.splitScalar(u8, domain, '.'); | |
| 935 | while (it.next()) |part| { | |
| 936 | if (std.ascii.eqlIgnoreCase(part, label)) return true; | |
| 937 | } | |
| 938 | return false; | |
| 939 | } | |
| 940 | ||
| 941 | fn isWordByte(c: u8) bool { | |
| 942 | return std.ascii.isAlphanumeric(c) or c == '_'; | |
| 943 | } | |
| 944 | ||
| 945 | fn containsWordIgnoreCase(haystack: []const u8, needle: []const u8) bool { | |
| 946 | var pos: usize = 0; | |
| 947 | while (std.ascii.indexOfIgnoreCasePos(haystack, pos, needle)) |at| { | |
| 948 | const before_ok = at == 0 or !isWordByte(haystack[at - 1]); | |
| 949 | const end = at + needle.len; | |
| 950 | const after_ok = end == haystack.len or !isWordByte(haystack[end]); | |
| 951 | if (before_ok and after_ok) return true; | |
| 952 | pos = at + 1; | |
| 953 | } | |
| 954 | return false; | |
| 955 | } | |
| 956 | ||
| 957 | fn scanHitWord(scan: anytype, word: []const u8) bool { | |
| 958 | if (scan.subject) |s| { | |
| 959 | if (containsWordIgnoreCase(s, word)) return true; | |
| 960 | } | |
| 961 | return containsWordIgnoreCase(scan.text, word) or | |
| 962 | (scan.b64.len > 0 and containsWordIgnoreCase(scan.b64, word)); | |
| 963 | } | |
| 964 | ||
| 782 | 965 | fn isMostlyUppercase(s: []const u8) bool { |
| 783 | 966 | var letters: usize = 0; |
| 784 | 967 | var upper: usize = 0; |
| @@ -896,6 +1079,13 @@ test "dangerous attachment flagged" { | ||
| 896 | 1079 | try std.testing.expect(std.mem.indexOf(u8, r.tests(), "DANGEROUS_ATTACHMENT") != null); |
| 897 | 1080 | } |
| 898 | 1081 | |
| 1082 | test "RFC 2231 encoded dangerous attachment is flagged" { | |
| 1083 | const headers = "From: x@y.example\nSubject: invoice\nDate: d\nMessage-ID: <1@y>"; | |
| 1084 | const body = "Content-Disposition: attachment; filename*=UTF-8''invoice%2Eps1\n"; | |
| 1085 | const r = evaluate(.{ .spf = .pass }, headers, body, .{}); | |
| 1086 | try std.testing.expect(std.mem.indexOf(u8, r.tests(), "DANGEROUS_ATTACHMENT") != null); | |
| 1087 | } | |
| 1088 | ||
| 899 | 1089 | test "bare-domain link is not a dangerous attachment" { |
| 900 | 1090 | const headers = "From: Hetzner <support-cloud@hetzner.com>\nSubject: Invoice\nDate: d\nMessage-ID: <1@hetzner.com>"; |
| 901 | 1091 | // A newsletter that links to the sender's homepage — the old body-wide |
| @@ -973,6 +1163,42 @@ test "unsolicited SEO pitch from a free provider lands in Junk" { | ||
| 973 | 1163 | try std.testing.expectEqual(Disposition.junk, r.disposition); |
| 974 | 1164 | } |
| 975 | 1165 | |
| 1166 | test "real-world organic traffic follow-up lands in Junk" { | |
| 1167 | const headers = | |
| 1168 | "From: Ronit Roy <ronitroy17906@outlook.com>\n" ++ | |
| 1169 | "To: info@example.com\n" ++ | |
| 1170 | "Subject: Gentle Reminder.....\n" ++ | |
| 1171 | "Date: Mon, 6 Jul 2026 11:17:07 +0000\n" ++ | |
| 1172 | "Message-ID: <1@outlook.com>"; | |
| 1173 | const body = | |
| 1174 | "Hi, Hope you're doing well. I wanted to follow up and see if you're interested in " ++ | |
| 1175 | "improving your website's visibility on Google. I can share details along with pricing. " ++ | |
| 1176 | "Many great websites struggle because they aren't properly optimized for search engines. " ++ | |
| 1177 | "We fix that - improving rankings, visibility, and conversions through data-driven SEO " ++ | |
| 1178 | "strategies. Would you like to explore how we can optimize your site? I can send our " ++ | |
| 1179 | "proposal and pricing."; | |
| 1180 | const r = evaluate(.{ .spf = .pass, .dkim = .pass, .dmarc = .pass }, headers, body, .{}); | |
| 1181 | try std.testing.expect(std.mem.indexOf(u8, r.tests(), "SEO_SOLICITATION") != null); | |
| 1182 | try std.testing.expect(std.mem.indexOf(u8, r.tests(), "FREEMAIL_OUTREACH") != null); | |
| 1183 | try std.testing.expectEqual(Disposition.junk, r.disposition); | |
| 1184 | } | |
| 1185 | ||
| 1186 | test "real-world website ranking error pitch lands in Junk" { | |
| 1187 | const headers = | |
| 1188 | "From: Jessica <jessica@seopackagesprice.com>\n" ++ | |
| 1189 | "To: info@example.com\n" ++ | |
| 1190 | "Subject: Increase Web Reach\n" ++ | |
| 1191 | "Date: Mon, 6 Jul 2026 14:23:36 +0530\n" ++ | |
| 1192 | "Message-ID: <1@seopackagesprice.com>"; | |
| 1193 | const body = | |
| 1194 | "Hi info@example.com, Greetings, I noticed some errors at your website ranking. " ++ | |
| 1195 | "I can help to fix errors for better Google ranking. " ++ | |
| 1196 | "May I send you screenshots of these errors? Sincerely, Jessica"; | |
| 1197 | const r = evaluate(.{ .spf = .pass, .dkim = .pass, .dmarc = .pass }, headers, body, .{}); | |
| 1198 | try std.testing.expect(std.mem.indexOf(u8, r.tests(), "SEO_SOLICITATION") != null); | |
| 1199 | try std.testing.expectEqual(Disposition.junk, r.disposition); | |
| 1200 | } | |
| 1201 | ||
| 976 | 1202 | test "fabricated reply prefix without thread headers is flagged" { |
| 977 | 1203 | const headers = |
| 978 | 1204 | "From: Lucy <lucy@lexxdigital.com>\n" ++ |
| @@ -1004,6 +1230,23 @@ test "brand name in display with unrelated domain is brand spoofing" { | ||
| 1004 | 1230 | try std.testing.expect(std.mem.indexOf(u8, r.tests(), "BRAND_SPOOF") != null); |
| 1005 | 1231 | } |
| 1006 | 1232 | |
| 1233 | test "brand in lookalike sender domain does not bypass spoof detection" { | |
| 1234 | const headers = | |
| 1235 | "From: PayPal <billing@paypal-security.example>\n" ++ | |
| 1236 | "Subject: Account notice\nDate: d\nMessage-ID: <1@paypal-security.example>"; | |
| 1237 | const r = evaluate(.{ .spf = .pass, .dkim = .pass, .dmarc = .pass }, headers, "review account", .{}); | |
| 1238 | try std.testing.expect(std.mem.indexOf(u8, r.tests(), "BRAND_SPOOF") != null); | |
| 1239 | } | |
| 1240 | ||
| 1241 | test "encoded brand in folded From header is still detected" { | |
| 1242 | const headers = | |
| 1243 | "From: =?UTF-8?B?UGF5UGFs?=\n" ++ | |
| 1244 | " <billing@unrelated.example>\n" ++ | |
| 1245 | "Subject: Account notice\nDate: d\nMessage-ID: <1@unrelated.example>"; | |
| 1246 | const r = evaluate(.{ .spf = .pass, .dkim = .pass, .dmarc = .pass }, headers, "review account", .{}); | |
| 1247 | try std.testing.expect(std.mem.indexOf(u8, r.tests(), "BRAND_SPOOF") != null); | |
| 1248 | } | |
| 1249 | ||
| 1007 | 1250 | test "legit brand mail from its own domain is not brand spoofing" { |
| 1008 | 1251 | const headers = |
| 1009 | 1252 | "From: Netflix <info@mailer.netflix.com>\n" ++ |
| @@ -1014,6 +1257,83 @@ test "legit brand mail from its own domain is not brand spoofing" { | ||
| 1014 | 1257 | try std.testing.expect(std.mem.indexOf(u8, r.tests(), "BRAND_SPOOF") == null); |
| 1015 | 1258 | } |
| 1016 | 1259 | |
| 1260 | test "authenticated parcel payment phishing lands in Junk" { | |
| 1261 | const headers = | |
| 1262 | "From: Customs Department <random@blueoceancorp.com.vn>\n" ++ | |
| 1263 | "To: victim@example.com\n" ++ | |
| 1264 | "Subject: Shipment held - fee required\n" ++ | |
| 1265 | "Date: Thu, 9 Jul 2026 12:38:39 +0700\n" ++ | |
| 1266 | "Message-ID: <1@blueoceancorp.com.vn>"; | |
| 1267 | const body = | |
| 1268 | "DHL Express Shipment Release Information. We received the customs documentation. " ++ | |
| 1269 | "A fee must be paid before final delivery. Total amount due $3.35. " ++ | |
| 1270 | "Status: Awaiting payment at the local facility. Make payment to complete the process."; | |
| 1271 | const r = evaluate(.{ .spf = .pass, .dkim = .pass, .dmarc = .pass }, headers, body, .{}); | |
| 1272 | try std.testing.expect(std.mem.indexOf(u8, r.tests(), "CARRIER_PAYMENT_PHISH") != null); | |
| 1273 | try std.testing.expectEqual(Disposition.junk, r.disposition); | |
| 1274 | } | |
| 1275 | ||
| 1276 | test "legitimate carrier tracking notice is not parcel payment phishing" { | |
| 1277 | const headers = | |
| 1278 | "From: Shop <shipping@trusted-shop.example>\n" ++ | |
| 1279 | "To: customer@example.com\n" ++ | |
| 1280 | "Subject: Your order shipped\nDate: d\nMessage-ID: <1@trusted-shop.example>"; | |
| 1281 | const body = "DHL has your parcel. Track the shipment for the expected final delivery date."; | |
| 1282 | const r = evaluate(.{ .spf = .pass, .dkim = .pass, .dmarc = .pass }, headers, body, .{}); | |
| 1283 | try std.testing.expect(std.mem.indexOf(u8, r.tests(), "CARRIER_PAYMENT_PHISH") == null); | |
| 1284 | try std.testing.expectEqual(Disposition.accept, r.disposition); | |
| 1285 | } | |
| 1286 | ||
| 1287 | test "legitimate carrier payment mail from carrier domain is not impersonation" { | |
| 1288 | const headers = | |
| 1289 | "From: DHL <billing@dhl.com>\nTo: customer@example.com\n" ++ | |
| 1290 | "Subject: Customs fee required\nDate: d\nMessage-ID: <1@dhl.com>"; | |
| 1291 | const body = "DHL shipment release: customs fee required, amount due, make payment."; | |
| 1292 | const r = evaluate(.{ .spf = .pass, .dkim = .pass, .dmarc = .pass }, headers, body, .{}); | |
| 1293 | try std.testing.expect(std.mem.indexOf(u8, r.tests(), "CARRIER_PAYMENT_PHISH") == null); | |
| 1294 | try std.testing.expectEqual(Disposition.accept, r.disposition); | |
| 1295 | } | |
| 1296 | ||
| 1297 | test "UPS is matched as a word and not inside groups" { | |
| 1298 | const headers = | |
| 1299 | "From: Community <hello@community.example>\nTo: a@b\n" ++ | |
| 1300 | "Subject: Local groups\nDate: d\nMessage-ID: <1@community.example>"; | |
| 1301 | const body = "Our community groups discussed shipment release and amount due, then make payment."; | |
| 1302 | const r = evaluate(.{ .spf = .pass, .dkim = .pass, .dmarc = .pass }, headers, body, .{}); | |
| 1303 | try std.testing.expect(std.mem.indexOf(u8, r.tests(), "CARRIER_PAYMENT_PHISH") == null); | |
| 1304 | } | |
| 1305 | ||
| 1306 | test "folded subject is unfolded before scoring" { | |
| 1307 | const headers = | |
| 1308 | "From: x@promo.example\nTo: a@b\n" ++ | |
| 1309 | "Subject: Shipment held -\n\tfee required\n" ++ | |
| 1310 | "Date: d\nMessage-ID: <1@promo.example>"; | |
| 1311 | const body = "DHL shipment release, amount due, awaiting payment at the local facility."; | |
| 1312 | const r = evaluate(.{ .spf = .pass, .dkim = .pass, .dmarc = .pass }, headers, body, .{}); | |
| 1313 | try std.testing.expect(std.mem.indexOf(u8, r.tests(), "CARRIER_PAYMENT_PHISH") != null); | |
| 1314 | } | |
| 1315 | ||
| 1316 | test "content padded beyond eight KiB is still scanned" { | |
| 1317 | const headers = | |
| 1318 | "From: x@promo.example\nTo: a@b\nSubject: hello\nDate: d\nMessage-ID: <1@promo.example>"; | |
| 1319 | var body: std.ArrayList(u8) = .empty; | |
| 1320 | defer body.deinit(std.testing.allocator); | |
| 1321 | try body.appendNTimes(std.testing.allocator, 'x', 9000); | |
| 1322 | try body.appendSlice(std.testing.allocator, " DHL shipment release fee required amount due awaiting payment"); | |
| 1323 | const r = evaluate(.{ .spf = .pass, .dkim = .pass, .dmarc = .pass }, headers, body.items, .{}); | |
| 1324 | try std.testing.expect(std.mem.indexOf(u8, r.tests(), "CARRIER_PAYMENT_PHISH") != null); | |
| 1325 | } | |
| 1326 | ||
| 1327 | test "uppercase URL schemes count toward link farm" { | |
| 1328 | const headers = "From: x@y.example\nSubject: links\nDate: d\nMessage-ID: <1@y>"; | |
| 1329 | const body = "HTTPS://a.test HTTPS://b.test HTTPS://c.test HTTPS://d.test HTTPS://e.test " ++ | |
| 1330 | "HTTPS://f.test HTTPS://g.test HTTPS://h.test HTTPS://i.test HTTPS://j.test " ++ | |
| 1331 | "HTTPS://k.test HTTPS://l.test HTTPS://m.test HTTPS://n.test HTTPS://o.test " ++ | |
| 1332 | "HTTPS://p.test HTTPS://q.test HTTPS://r.test HTTPS://s.test HTTPS://t.test"; | |
| 1333 | const r = evaluate(.{ .spf = .pass }, headers, body, .{}); | |
| 1334 | try std.testing.expect(std.mem.indexOf(u8, r.tests(), "MANY_URLS") != null); | |
| 1335 | } | |
| 1336 | ||
| 1017 | 1337 | test "real-world app-dev cold outreach (QP, outlook freemail) lands in Junk" { |
| 1018 | 1338 | // Reconstruction of a live miss: scored 1.5 (HTML_HIDDEN_TEXT only) |
| 1019 | 1339 | // before the outreach/app-dev rules and QP decoding existed. |
| @@ -1070,6 +1390,27 @@ test "base64-encoded MIME part cannot hide the pitch" { | ||
| 1070 | 1390 | try std.testing.expect(std.mem.indexOf(u8, r.tests(), "OUTREACH_PITCH") != null); |
| 1071 | 1391 | } |
| 1072 | 1392 | |
| 1393 | test "a benign first base64 part cannot hide spam in a later part" { | |
| 1394 | const headers = "From: x@gmail.com\nTo: a@b\nSubject: hello\nDate: d\nMessage-ID: <1@g>"; | |
| 1395 | const body = | |
| 1396 | "--b\r\nContent-Type: text/plain\r\nContent-Transfer-Encoding: base64\r\n\r\n" ++ | |
| 1397 | "SGVsbG8sIHRoaXMgaXMgYSBub3JtYWwgYWx0ZXJuYXRpdmUu\r\n" ++ | |
| 1398 | "--b\r\nContent-Type: text/html\r\nContent-Transfer-Encoding: base64\r\n\r\n" ++ | |
| 1399 | "c2VvIHNlcnZpY2VzIG1vYmlsZSBhcHAgZGV2ZWxvcG1lbnQgcHJpY2UgbGlzdCB3aGF0c2FwcA==\r\n--b--\r\n"; | |
| 1400 | const r = evaluate(.{ .spf = .pass }, headers, body, .{}); | |
| 1401 | try std.testing.expect(std.mem.indexOf(u8, r.tests(), "SEO_SOLICITATION") != null); | |
| 1402 | try std.testing.expect(std.mem.indexOf(u8, r.tests(), "CONTACT_HARVEST") != null); | |
| 1403 | } | |
| 1404 | ||
| 1405 | test "URLs in a base64 MIME part count toward link farm" { | |
| 1406 | const headers = "From: x@y.example\nSubject: links\nDate: d\nMessage-ID: <1@y>"; | |
| 1407 | const body = | |
| 1408 | "--b\r\nContent-Type: text/html\r\nContent-Transfer-Encoding: base64\r\n\r\n" ++ | |
| 1409 | "aHR0cHM6Ly9hLnRlc3QgaHR0cHM6Ly9iLnRlc3QgaHR0cHM6Ly9jLnRlc3QgaHR0cHM6Ly9kLnRlc3QgaHR0cHM6Ly9lLnRlc3QgaHR0cHM6Ly9mLnRlc3QgaHR0cHM6Ly9nLnRlc3QgaHR0cHM6Ly9oLnRlc3QgaHR0cHM6Ly9pLnRlc3QgaHR0cHM6Ly9qLnRlc3QgaHR0cHM6Ly9rLnRlc3QgaHR0cHM6Ly9sLnRlc3QgaHR0cHM6Ly9tLnRlc3QgaHR0cHM6Ly9uLnRlc3QgaHR0cHM6Ly9vLnRlc3QgaHR0cHM6Ly9wLnRlc3QgaHR0cHM6Ly9xLnRlc3QgaHR0cHM6Ly9yLnRlc3QgaHR0cHM6Ly9zLnRlc3QgaHR0cHM6Ly90LnRlc3Q=\r\n--b--\r\n"; | |
| 1410 | const r = evaluate(.{ .spf = .pass }, headers, body, .{}); | |
| 1411 | try std.testing.expect(std.mem.indexOf(u8, r.tests(), "MANY_URLS") != null); | |
| 1412 | } | |
| 1413 | ||
| 1073 | 1414 | test "encoded-word subject is decoded before subject heuristics" { |
| 1074 | 1415 | // =?UTF-8?B?...?= of "WIN A FREE IPHONE RIGHT NOW!!" |
| 1075 | 1416 | const headers = |
| @@ -1089,6 +1430,30 @@ test "html entities cannot split a phrase" { | ||
| 1089 | 1430 | try std.testing.expect(std.mem.indexOf(u8, r.tests(), "OUTREACH_PITCH") != null); |
| 1090 | 1431 | } |
| 1091 | 1432 | |
| 1433 | test "numeric HTML entities cannot split a phrase" { | |
| 1434 | const headers = "From: x@gmail.com\nTo: a@b\nSubject: hi\nDate: d\nMessage-ID: <1@g>"; | |
| 1435 | const body = "we offer mobile app development and seo services"; | |
| 1436 | const r = evaluate(.{ .spf = .pass }, headers, body, .{}); | |
| 1437 | try std.testing.expect(std.mem.indexOf(u8, r.tests(), "SEO_SOLICITATION") != null); | |
| 1438 | } | |
| 1439 | ||
| 1440 | test "zero-width Unicode cannot split a phrase" { | |
| 1441 | const headers = "From: x@gmail.com\nTo: a@b\nSubject: hi\nDate: d\nMessage-ID: <1@g>"; | |
| 1442 | const body = "we offer mobile\xE2\x80\x8B app development and seo ser​vices"; | |
| 1443 | const r = evaluate(.{ .spf = .pass }, headers, body, .{}); | |
| 1444 | try std.testing.expect(std.mem.indexOf(u8, r.tests(), "SEO_SOLICITATION") != null); | |
| 1445 | } | |
| 1446 | ||
| 1447 | test "long subject is not truncated at 512 bytes" { | |
| 1448 | var headers: std.ArrayList(u8) = .empty; | |
| 1449 | defer headers.deinit(std.testing.allocator); | |
| 1450 | try headers.appendSlice(std.testing.allocator, "From: x@y.example\nSubject: "); | |
| 1451 | try headers.appendNTimes(std.testing.allocator, 'x', 600); | |
| 1452 | try headers.appendSlice(std.testing.allocator, " viagra\nDate: d\nMessage-ID: <1@y>"); | |
| 1453 | const r = evaluate(.{ .spf = .pass }, headers.items, "", .{}); | |
| 1454 | try std.testing.expect(std.mem.indexOf(u8, r.tests(), "SPAM_PHRASES") != null); | |
| 1455 | } | |
| 1456 | ||
| 1092 | 1457 | test "undisclosed recipients is flagged" { |
| 1093 | 1458 | const headers = "From: x@y.example\nTo: undisclosed-recipients:;\nSubject: hi\nDate: d\nMessage-ID: <1@y>"; |
| 1094 | 1459 | const r = evaluate(.{ .spf = .pass }, headers, "hello\n", .{}); |
| @@ -1,11 +1,7 @@ | ||
| 1 | 1 | { |
| 2 | 2 | "version": "1.0.0", |
| 3 | "lockfileVersion": 1, | |
| 3 | "lockfileVersion": 2, | |
| 4 | 4 | "workspaces": { |
| 5 | "packages/devtools": { | |
| 6 | "name": "mail-devtools", | |
| 7 | "version": "0.1.0" | |
| 8 | }, | |
| 9 | 5 | "packages/cloud": { |
| 10 | 6 | "name": "mail-cloud", |
| 11 | 7 | "version": "0.1.0", |
| @@ -13,59 +9,85 @@ | ||
| 13 | 9 | "ts-cloud": "link:ts-cloud" |
| 14 | 10 | }, |
| 15 | 11 | "devDependencies": { |
| 16 | "typescript": "^5.9.3", | |
| 17 | "@types/bun": "^1.3.0" | |
| 12 | "@types/bun": "^1.3.0", | |
| 13 | "typescript": "^5.9.3" | |
| 14 | } | |
| 15 | }, | |
| 16 | "packages/devtools": { | |
| 17 | "name": "mail-devtools", | |
| 18 | "version": "0.1.0", | |
| 19 | "dependencies": { | |
| 20 | "bun-plugin-stx": "^0.2.6" | |
| 18 | 21 | } |
| 19 | 22 | }, |
| 20 | 23 | "packages/ts": { |
| 21 | 24 | "name": "ts-mail", |
| 22 | "version": "0.28.0" | |
| 25 | "version": "0.1.0" | |
| 26 | }, | |
| 27 | "packages/webmail": { | |
| 28 | "name": "mail-webmail", | |
| 29 | "version": "0.0.1", | |
| 30 | "dependencies": { | |
| 31 | "bun-plugin-stx": "^0.2.6" | |
| 32 | } | |
| 23 | 33 | } |
| 24 | 34 | }, |
| 25 | 35 | "packages": { |
| 26 | "github:zig-utils/zig-bump@main": { | |
| 27 | "name": "github:zig-utils/zig-bump", | |
| 28 | "version": "main", | |
| 29 | "source": "github" | |
| 36 | "@types/bun@^1.3.0": { | |
| 37 | "name": "@types/bun", | |
| 38 | "version": "^1.3.0", | |
| 39 | "source": "npm", | |
| 40 | "dependencies": { | |
| 41 | "bun-types": "1.3.10" | |
| 42 | } | |
| 30 | 43 | }, |
| 31 | "typescript@^5.9.3": { | |
| 32 | "name": "typescript", | |
| 33 | "version": "^5.9.3", | |
| 44 | "bun-plugin-stx@^0.2.6": { | |
| 45 | "name": "bun-plugin-stx", | |
| 46 | "version": "^0.2.6", | |
| 34 | 47 | "source": "npm", |
| 48 | "dependencies": { | |
| 49 | "@stacksjs/stx": "0.2.72", | |
| 50 | "stx-router": "0.2.72" | |
| 51 | }, | |
| 35 | 52 | "bin": { |
| 36 | "tsc": "./bin/tsc", | |
| 37 | "tsserver": "./bin/tsserver" | |
| 53 | "serve": "./dist/serve.js", | |
| 54 | "stx-serve": "./dist/serve.js" | |
| 38 | 55 | } |
| 39 | 56 | }, |
| 57 | "github:zig-utils/zig-bump@main": { | |
| 58 | "name": "github:zig-utils/zig-bump", | |
| 59 | "version": "main", | |
| 60 | "source": "github" | |
| 61 | }, | |
| 40 | 62 | "github:zig-utils/zig-cli@main": { |
| 41 | 63 | "name": "github:zig-utils/zig-cli", |
| 42 | 64 | "version": "main", |
| 43 | 65 | "source": "github" |
| 44 | 66 | }, |
| 45 | "@types/bun@1.3.10": { | |
| 46 | "name": "@types/bun", | |
| 47 | "version": "1.3.10", | |
| 48 | "source": "npm", | |
| 49 | "resolved": "https://registry.npmjs.org/@types/bun/-/bun-1.3.10.tgz", | |
| 50 | "integrity": "sha512-0+rlrUrOrTSskibryHbvQkDOWRJwJZqZlxrUs1u4oOoTln8+WIXBPmAuCF35SWB2z4Zl3E84Nl/D0P7803nigQ==", | |
| 51 | "dependencies": { | |
| 52 | "bun-types": "1.3.10" | |
| 53 | } | |
| 67 | "github:zig-utils/zig-search-engine@main": { | |
| 68 | "name": "github:zig-utils/zig-search-engine", | |
| 69 | "version": "main", | |
| 70 | "source": "github" | |
| 54 | 71 | }, |
| 55 | 72 | "github:zig-utils/zig-tls@main": { |
| 56 | 73 | "name": "github:zig-utils/zig-tls", |
| 57 | 74 | "version": "main", |
| 58 | 75 | "source": "github" |
| 59 | 76 | }, |
| 60 | "ziglang.org@0.16.0-dev": { | |
| 61 | "name": "ziglang.org", | |
| 62 | "version": "0.16.0-dev", | |
| 63 | "source": "pantry" | |
| 64 | }, | |
| 65 | 77 | "ts-cloud@link:ts-cloud": { |
| 66 | 78 | "name": "ts-cloud", |
| 67 | 79 | "version": "link:ts-cloud", |
| 68 | 80 | "source": "npm" |
| 81 | }, | |
| 82 | "typescriptlang.org@^5.9.3": { | |
| 83 | "name": "typescriptlang.org", | |
| 84 | "version": "^5.9.3", | |
| 85 | "source": "npm" | |
| 86 | }, | |
| 87 | "ziglang.org@0.16.0-dev": { | |
| 88 | "name": "ziglang.org", | |
| 89 | "version": "0.16.0-dev", | |
| 90 | "source": "pantry" | |
| 69 | 91 | } |
| 70 | 92 | } |
| 71 | 93 | } |