also looking at this
Harden encrypted mail storage, recovery, and forwarding
#8
50 files
+1701
-9001
| @@ -0,0 +1,186 @@ | ||
| 1 | # Encryption at rest | |
| 2 | ||
| 3 | Production installs encrypt persisted mail data by default with a dedicated | |
| 4 | LUKS2 filesystem on an independently attached Hetzner volume. The protected | |
| 5 | paths include Maildirs, SQLite databases, DKIM material, forwarding rules, | |
| 6 | local backups, certificate backups, Restic credentials, and the mail log. | |
| 7 | ||
| 8 | The volume uses AES-XTS with a 512-bit key (two AES-256 keys, as required by | |
| 9 | XTS). A cryptographically random volume key is generated during provisioning. | |
| 10 | Initial installs wrap it with `systemd-creds` as the machine-bound encrypted | |
| 11 | credential: | |
| 12 | ||
| 13 | ```text | |
| 14 | /etc/credstore.encrypted/mail-storage-key.cred | |
| 15 | ``` | |
| 16 | ||
| 17 | The plaintext key exists only briefly in `/run` during initial provisioning. | |
| 18 | `mail-storage.service` receives it as a private runtime credential, unlocks the | |
| 19 | volume, and mounts it at `/var/lib/mail-storage` before `mail.service` starts. | |
| 20 | The unprivileged mail process can access the mounted data but cannot read the | |
| 21 | persistent key credential. | |
| 22 | ||
| 23 | The server retains its established paths under `/opt/mail`; those paths point | |
| 24 | into the encrypted mount so SMTP, IMAP, POP3, CalDAV, CLI tools, and backups all | |
| 25 | see the same data layout. | |
| 26 | ||
| 27 | ## Verification | |
| 28 | ||
| 29 | ```bash | |
| 30 | systemctl is-active mail-storage mail | |
| 31 | findmnt /var/lib/mail-storage | |
| 32 | cryptsetup status mail-storage | |
| 33 | systemd-creds decrypt --name=mail-storage-key \ | |
| 34 | /etc/credstore.encrypted/mail-storage-key.cred - >/dev/null | |
| 35 | ``` | |
| 36 | ||
| 37 | ## Object storage | |
| 38 | ||
| 39 | Cloud deployments apply a customer-managed KMS key with automatic rotation to | |
| 40 | private email and backup buckets. This is separate from LUKS2 and Restic: LUKS | |
| 41 | protects live host data, Restic encrypts and authenticates backup content | |
| 42 | before upload, and KMS-backed bucket encryption adds provider-side protection. | |
| 43 | ||
| 44 | Create or reconcile the KMS key and bucket policies: | |
| 45 | ||
| 46 | ```bash | |
| 47 | buddy mail:storage:kms:ensure --profile stacks | |
| 48 | ``` | |
| 49 | ||
| 50 | ## External key mode | |
| 51 | ||
| 52 | Hetzner VMs currently expose no TPM. For protection against theft of the entire | |
| 53 | root disk, production can externalize the LUKS key into AWS Secrets Manager, | |
| 54 | where it is encrypted by AWS KMS: | |
| 55 | ||
| 56 | ```bash | |
| 57 | buddy mail:storage:externalize --profile stacks | |
| 58 | ``` | |
| 59 | ||
| 60 | External mode removes the mail volume credential from the host. After a reboot, | |
| 61 | mail remains stopped until an authorized workstation streams the key from | |
| 62 | Secrets Manager directly into `cryptsetup`: | |
| 63 | ||
| 64 | ```bash | |
| 65 | buddy mail:storage:status | |
| 66 | buddy mail:storage:key:verify --profile stacks | |
| 67 | buddy mail:storage:unlock --profile stacks | |
| 68 | buddy mail:storage:lock | |
| 69 | ``` | |
| 70 | ||
| 71 | The key is never placed in a command-line argument, environment variable, log, | |
| 72 | or persistent file on the mail host. `mail:storage:unlock` holds it in the | |
| 73 | operator process briefly and sends it over SSH stdin. | |
| 74 | ||
| 75 | An independent recovery key can be added to a second LUKS2 keyslot and stored | |
| 76 | in the operator's macOS Keychain: | |
| 77 | ||
| 78 | ```bash | |
| 79 | buddy mail:storage:key:recovery:add --profile stacks | |
| 80 | buddy mail:storage:key:recovery:verify | |
| 81 | ``` | |
| 82 | ||
| 83 | The command streams both keys over SSH stdin, verifies the new keyslot, writes | |
| 84 | the recovery key to Keychain without placing it in argv, reads it back, and | |
| 85 | tests it against the LUKS header. If a failed enrollment leaves an unwanted | |
| 86 | keyslot, inspect the LUKS header and remove only the explicitly selected slot: | |
| 87 | ||
| 88 | ```bash | |
| 89 | buddy mail:storage:key:slot:remove --slot <number> --profile stacks | |
| 90 | ``` | |
| 91 | ||
| 92 | The removal command refuses the primary slot and requires the remaining | |
| 93 | external key to unlock the volume, so it cannot silently remove the last | |
| 94 | usable credential. | |
| 95 | ||
| 96 | ## Dedicated volume | |
| 97 | ||
| 98 | Move an existing root-disk LUKS image to a separately attached Hetzner volume: | |
| 99 | ||
| 100 | ```bash | |
| 101 | buddy mail:storage:volume:migrate --size 20 --profile stacks | |
| 102 | ``` | |
| 103 | ||
| 104 | The operation creates or reuses `stacks-production-mail`, stops mail, performs | |
| 105 | a byte-verified sparse copy, preserves the LUKS UUID and keyslots, unlocks with | |
| 106 | the externally stored key, grows the encrypted filesystem, validates the | |
| 107 | service, and only then removes the root-disk rollback image. A failed | |
| 108 | validation automatically restores the original image. | |
| 109 | ||
| 110 | ## Recovery | |
| 111 | ||
| 112 | Before externalization, the encrypted credential is bound to the machine's | |
| 113 | systemd host secret. A machine-image or disaster-recovery backup must retain | |
| 114 | both: | |
| 115 | ||
| 116 | ```text | |
| 117 | /etc/credstore.encrypted/mail-storage-key.cred | |
| 118 | /var/lib/systemd/credential.secret | |
| 119 | ``` | |
| 120 | ||
| 121 | After `mail:storage:externalize`, neither file is sufficient to unlock the mail | |
| 122 | volume because the local credential has been removed. Recovery then uses the | |
| 123 | AWS Secrets Manager entry (by default | |
| 124 | `stacks/production/mail-storage-key`): | |
| 125 | ||
| 126 | ```bash | |
| 127 | buddy mail:storage:key:verify --profile stacks | |
| 128 | buddy mail:storage:unlock --profile stacks | |
| 129 | ``` | |
| 130 | ||
| 131 | Back up the LUKS header separately after provisioning or changing key slots. | |
| 132 | The Buddy command streams it to S3 and requests KMS encryption without leaving | |
| 133 | a persistent local copy: | |
| 134 | ||
| 135 | ```bash | |
| 136 | buddy mail:storage:header:backup --profile stacks | |
| 137 | ``` | |
| 138 | ||
| 139 | The header backup is not a volume key, but it is sensitive and must be kept in | |
| 140 | encrypted offline storage. Losing every valid external key or destroying the | |
| 141 | LUKS header makes the volume unrecoverable by design. | |
| 142 | ||
| 143 | ## Backup operations | |
| 144 | ||
| 145 | Local backups are created inside the encrypted filesystem, use SQLite's online | |
| 146 | backup API, and verify the archive before publishing it: | |
| 147 | ||
| 148 | ```bash | |
| 149 | buddy mail:storage:backup | |
| 150 | systemctl status mail-backup.timer | |
| 151 | ``` | |
| 152 | ||
| 153 | Cloud object storage encryption is provider-side and independent from LUKS. | |
| 154 | Keep bucket default encryption enabled even when uploaded archives came from | |
| 155 | the encrypted host volume. | |
| 156 | ||
| 157 | Authenticated off-host backups use Restic with a dedicated least-privilege IAM | |
| 158 | user. The Restic password and AWS credentials are stored both inside the | |
| 159 | unlocked encrypted filesystem for unattended backups and in the | |
| 160 | KMS-encrypted `stacks/production/mail-restic` recovery secret. | |
| 161 | ||
| 162 | ```bash | |
| 163 | buddy mail:storage:restic:configure --profile stacks | |
| 164 | buddy mail:storage:restore:check --profile stacks | |
| 165 | ``` | |
| 166 | ||
| 167 | The daily job creates a verified SQLite-consistent local snapshot, uploads it | |
| 168 | through Restic, applies 7 daily / 5 weekly / 12 monthly retention, prunes, | |
| 169 | and performs a repository data check. The restore check downloads the newest | |
| 170 | snapshot to a temporary directory on the encrypted filesystem, validates both | |
| 171 | archives, runs SQLite `quick_check`, and confirms that mail and forwarding data | |
| 172 | are present. An hourly health timer fails when the encrypted mount is missing, | |
| 173 | usage crosses configured thresholds, or the newest successful off-host backup | |
| 174 | is stale. | |
| 175 | ||
| 176 | ## Recovery order | |
| 177 | ||
| 178 | 1. Attach and mount the Hetzner data volume. | |
| 179 | 2. Retrieve either the AWS secret or independent Keychain recovery key. | |
| 180 | 3. Unlock `/var/lib/mail-storage.luks`. | |
| 181 | 4. Start `mail.service` and validate SQLite plus Maildir counts. | |
| 182 | 5. If the volume is unavailable, restore the latest Restic snapshot and its | |
| 183 | environment archive onto a newly encrypted volume. | |
| 184 | ||
| 185 | Implementation and operational follow-up are tracked in | |
| 186 | [mail-os/mail#4](https://github.com/mail-os/mail/issues/4). | |