ReviewOS

also looking at this

mail-os/mail

Harden encrypted mail storage, recovery, and forwarding

#8
Merged chris wants to merge codex/mail-storage-hardening into main
50 files +1701 -9001
docs/ENCRYPTION_AT_REST.mdadded+186-0
Changes to docs/ENCRYPTION_AT_REST.md
@@ -0,0 +1,186 @@
1# Encryption at rest
2
3Production installs encrypt persisted mail data by default with a dedicated
4LUKS2 filesystem on an independently attached Hetzner volume. The protected
5paths include Maildirs, SQLite databases, DKIM material, forwarding rules,
6local backups, certificate backups, Restic credentials, and the mail log.
7
8The volume uses AES-XTS with a 512-bit key (two AES-256 keys, as required by
9XTS). A cryptographically random volume key is generated during provisioning.
10Initial installs wrap it with `systemd-creds` as the machine-bound encrypted
11credential:
12
13```text
14/etc/credstore.encrypted/mail-storage-key.cred
15```
16
17The plaintext key exists only briefly in `/run` during initial provisioning.
18`mail-storage.service` receives it as a private runtime credential, unlocks the
19volume, and mounts it at `/var/lib/mail-storage` before `mail.service` starts.
20The unprivileged mail process can access the mounted data but cannot read the
21persistent key credential.
22
23The server retains its established paths under `/opt/mail`; those paths point
24into the encrypted mount so SMTP, IMAP, POP3, CalDAV, CLI tools, and backups all
25see the same data layout.
26
27## Verification
28
29```bash
30systemctl is-active mail-storage mail
31findmnt /var/lib/mail-storage
32cryptsetup status mail-storage
33systemd-creds decrypt --name=mail-storage-key \
34 /etc/credstore.encrypted/mail-storage-key.cred - >/dev/null
35```
36
37## Object storage
38
39Cloud deployments apply a customer-managed KMS key with automatic rotation to
40private email and backup buckets. This is separate from LUKS2 and Restic: LUKS
41protects live host data, Restic encrypts and authenticates backup content
42before upload, and KMS-backed bucket encryption adds provider-side protection.
43
44Create or reconcile the KMS key and bucket policies:
45
46```bash
47buddy mail:storage:kms:ensure --profile stacks
48```
49
50## External key mode
51
52Hetzner VMs currently expose no TPM. For protection against theft of the entire
53root disk, production can externalize the LUKS key into AWS Secrets Manager,
54where it is encrypted by AWS KMS:
55
56```bash
57buddy mail:storage:externalize --profile stacks
58```
59
60External mode removes the mail volume credential from the host. After a reboot,
61mail remains stopped until an authorized workstation streams the key from
62Secrets Manager directly into `cryptsetup`:
63
64```bash
65buddy mail:storage:status
66buddy mail:storage:key:verify --profile stacks
67buddy mail:storage:unlock --profile stacks
68buddy mail:storage:lock
69```
70
71The key is never placed in a command-line argument, environment variable, log,
72or persistent file on the mail host. `mail:storage:unlock` holds it in the
73operator process briefly and sends it over SSH stdin.
74
75An independent recovery key can be added to a second LUKS2 keyslot and stored
76in the operator's macOS Keychain:
77
78```bash
79buddy mail:storage:key:recovery:add --profile stacks
80buddy mail:storage:key:recovery:verify
81```
82
83The command streams both keys over SSH stdin, verifies the new keyslot, writes
84the recovery key to Keychain without placing it in argv, reads it back, and
85tests it against the LUKS header. If a failed enrollment leaves an unwanted
86keyslot, inspect the LUKS header and remove only the explicitly selected slot:
87
88```bash
89buddy mail:storage:key:slot:remove --slot <number> --profile stacks
90```
91
92The removal command refuses the primary slot and requires the remaining
93external key to unlock the volume, so it cannot silently remove the last
94usable credential.
95
96## Dedicated volume
97
98Move an existing root-disk LUKS image to a separately attached Hetzner volume:
99
100```bash
101buddy mail:storage:volume:migrate --size 20 --profile stacks
102```
103
104The operation creates or reuses `stacks-production-mail`, stops mail, performs
105a byte-verified sparse copy, preserves the LUKS UUID and keyslots, unlocks with
106the externally stored key, grows the encrypted filesystem, validates the
107service, and only then removes the root-disk rollback image. A failed
108validation automatically restores the original image.
109
110## Recovery
111
112Before externalization, the encrypted credential is bound to the machine's
113systemd host secret. A machine-image or disaster-recovery backup must retain
114both:
115
116```text
117/etc/credstore.encrypted/mail-storage-key.cred
118/var/lib/systemd/credential.secret
119```
120
121After `mail:storage:externalize`, neither file is sufficient to unlock the mail
122volume because the local credential has been removed. Recovery then uses the
123AWS Secrets Manager entry (by default
124`stacks/production/mail-storage-key`):
125
126```bash
127buddy mail:storage:key:verify --profile stacks
128buddy mail:storage:unlock --profile stacks
129```
130
131Back up the LUKS header separately after provisioning or changing key slots.
132The Buddy command streams it to S3 and requests KMS encryption without leaving
133a persistent local copy:
134
135```bash
136buddy mail:storage:header:backup --profile stacks
137```
138
139The header backup is not a volume key, but it is sensitive and must be kept in
140encrypted offline storage. Losing every valid external key or destroying the
141LUKS header makes the volume unrecoverable by design.
142
143## Backup operations
144
145Local backups are created inside the encrypted filesystem, use SQLite's online
146backup API, and verify the archive before publishing it:
147
148```bash
149buddy mail:storage:backup
150systemctl status mail-backup.timer
151```
152
153Cloud object storage encryption is provider-side and independent from LUKS.
154Keep bucket default encryption enabled even when uploaded archives came from
155the encrypted host volume.
156
157Authenticated off-host backups use Restic with a dedicated least-privilege IAM
158user. The Restic password and AWS credentials are stored both inside the
159unlocked encrypted filesystem for unattended backups and in the
160KMS-encrypted `stacks/production/mail-restic` recovery secret.
161
162```bash
163buddy mail:storage:restic:configure --profile stacks
164buddy mail:storage:restore:check --profile stacks
165```
166
167The daily job creates a verified SQLite-consistent local snapshot, uploads it
168through Restic, applies 7 daily / 5 weekly / 12 monthly retention, prunes,
169and performs a repository data check. The restore check downloads the newest
170snapshot to a temporary directory on the encrypted filesystem, validates both
171archives, runs SQLite `quick_check`, and confirms that mail and forwarding data
172are present. An hourly health timer fails when the encrypted mount is missing,
173usage crosses configured thresholds, or the newest successful off-host backup
174is stale.
175
176## Recovery order
177
1781. Attach and mount the Hetzner data volume.
1792. Retrieve either the AWS secret or independent Keychain recovery key.
1803. Unlock `/var/lib/mail-storage.luks`.
1814. Start `mail.service` and validate SQLite plus Maildir counts.
1825. If the volume is unavailable, restore the latest Restic snapshot and its
183 environment archive onto a newly encrypted volume.
184
185Implementation and operational follow-up are tracked in
186[mail-os/mail#4](https://github.com/mail-os/mail/issues/4).