also looking at this

Harden encrypted mail storage, recovery, and forwarding

Merged chris wants to merge codex/mail-storage-hardening into main

50 files +1701 -9001

Review threads live on the whole diff, not on one commit, so none are shown here - a thread's line means something in the branch's final form, and painting it into an intermediate step would put it on code it is not about.

docs/ENCRYPTION_AT_REST.mdadded+186-0

Changes to docs/ENCRYPTION_AT_REST.md
		@@ -0,0 +1,186 @@
	1	# Encryption at rest
	2
	3	Production installs encrypt persisted mail data by default with a dedicated
	4	LUKS2 filesystem on an independently attached Hetzner volume. The protected
	5	paths include Maildirs, SQLite databases, DKIM material, forwarding rules,
	6	local backups, certificate backups, Restic credentials, and the mail log.
	7
	8	The volume uses AES-XTS with a 512-bit key (two AES-256 keys, as required by
	9	XTS). A cryptographically random volume key is generated during provisioning.
	10	Initial installs wrap it with `systemd-creds` as the machine-bound encrypted
	11	credential:
	12
	13	```text
	14	/etc/credstore.encrypted/mail-storage-key.cred
	15	```
	16
	17	The plaintext key exists only briefly in `/run` during initial provisioning.
	18	`mail-storage.service` receives it as a private runtime credential, unlocks the
	19	volume, and mounts it at `/var/lib/mail-storage` before `mail.service` starts.
	20	The unprivileged mail process can access the mounted data but cannot read the
	21	persistent key credential.
	22
	23	The server retains its established paths under `/opt/mail`; those paths point
	24	into the encrypted mount so SMTP, IMAP, POP3, CalDAV, CLI tools, and backups all
	25	see the same data layout.
	26
	27	## Verification
	28
	29	```bash
	30	systemctl is-active mail-storage mail
	31	findmnt /var/lib/mail-storage
	32	cryptsetup status mail-storage
	33	systemd-creds decrypt --name=mail-storage-key \
	34	/etc/credstore.encrypted/mail-storage-key.cred - >/dev/null
	35	```
	36
	37	## Object storage
	38
	39	Cloud deployments apply a customer-managed KMS key with automatic rotation to
	40	private email and backup buckets. This is separate from LUKS2 and Restic: LUKS
	41	protects live host data, Restic encrypts and authenticates backup content
	42	before upload, and KMS-backed bucket encryption adds provider-side protection.
	43
	44	Create or reconcile the KMS key and bucket policies:
	45
	46	```bash
	47	buddy mail:storage:kms:ensure --profile stacks
	48	```
	49
	50	## External key mode
	51
	52	Hetzner VMs currently expose no TPM. For protection against theft of the entire
	53	root disk, production can externalize the LUKS key into AWS Secrets Manager,
	54	where it is encrypted by AWS KMS:
	55
	56	```bash
	57	buddy mail:storage:externalize --profile stacks
	58	```
	59
	60	External mode removes the mail volume credential from the host. After a reboot,
	61	mail remains stopped until an authorized workstation streams the key from
	62	Secrets Manager directly into `cryptsetup`:
	63
	64	```bash
	65	buddy mail:storage:status
	66	buddy mail:storage:key:verify --profile stacks
	67	buddy mail:storage:unlock --profile stacks
	68	buddy mail:storage:lock
	69	```
	70
	71	The key is never placed in a command-line argument, environment variable, log,
	72	or persistent file on the mail host. `mail:storage:unlock` holds it in the
	73	operator process briefly and sends it over SSH stdin.
	74
	75	An independent recovery key can be added to a second LUKS2 keyslot and stored
	76	in the operator's macOS Keychain:
	77
	78	```bash
	79	buddy mail:storage:key:recovery:add --profile stacks
	80	buddy mail:storage:key:recovery:verify
	81	```
	82
	83	The command streams both keys over SSH stdin, verifies the new keyslot, writes
	84	the recovery key to Keychain without placing it in argv, reads it back, and
	85	tests it against the LUKS header. If a failed enrollment leaves an unwanted
	86	keyslot, inspect the LUKS header and remove only the explicitly selected slot:
	87
	88	```bash
	89	buddy mail:storage:key:slot:remove --slot <number> --profile stacks
	90	```
	91
	92	The removal command refuses the primary slot and requires the remaining
	93	external key to unlock the volume, so it cannot silently remove the last
	94	usable credential.
	95
	96	## Dedicated volume
	97
	98	Move an existing root-disk LUKS image to a separately attached Hetzner volume:
	99
	100	```bash
	101	buddy mail:storage:volume:migrate --size 20 --profile stacks
	102	```
	103
	104	The operation creates or reuses `stacks-production-mail`, stops mail, performs
	105	a byte-verified sparse copy, preserves the LUKS UUID and keyslots, unlocks with
	106	the externally stored key, grows the encrypted filesystem, validates the
	107	service, and only then removes the root-disk rollback image. A failed
	108	validation automatically restores the original image.
	109
	110	## Recovery
	111
	112	Before externalization, the encrypted credential is bound to the machine's
	113	systemd host secret. A machine-image or disaster-recovery backup must retain
	114	both:
	115
	116	```text
	117	/etc/credstore.encrypted/mail-storage-key.cred
	118	/var/lib/systemd/credential.secret
	119	```
	120
	121	After `mail:storage:externalize`, neither file is sufficient to unlock the mail
	122	volume because the local credential has been removed. Recovery then uses the
	123	AWS Secrets Manager entry (by default
	124	`stacks/production/mail-storage-key`):
	125
	126	```bash
	127	buddy mail:storage:key:verify --profile stacks
	128	buddy mail:storage:unlock --profile stacks
	129	```
	130
	131	Back up the LUKS header separately after provisioning or changing key slots.
	132	The Buddy command streams it to S3 and requests KMS encryption without leaving
	133	a persistent local copy:
	134
	135	```bash
	136	buddy mail:storage:header:backup --profile stacks
	137	```
	138
	139	The header backup is not a volume key, but it is sensitive and must be kept in
	140	encrypted offline storage. Losing every valid external key or destroying the
	141	LUKS header makes the volume unrecoverable by design.
	142
	143	## Backup operations
	144
	145	Local backups are created inside the encrypted filesystem, use SQLite's online
	146	backup API, and verify the archive before publishing it:
	147
	148	```bash
	149	buddy mail:storage:backup
	150	systemctl status mail-backup.timer
	151	```
	152
	153	Cloud object storage encryption is provider-side and independent from LUKS.
	154	Keep bucket default encryption enabled even when uploaded archives came from
	155	the encrypted host volume.
	156
	157	Authenticated off-host backups use Restic with a dedicated least-privilege IAM
	158	user. The Restic password and AWS credentials are stored both inside the
	159	unlocked encrypted filesystem for unattended backups and in the
	160	KMS-encrypted `stacks/production/mail-restic` recovery secret.
	161
	162	```bash
	163	buddy mail:storage:restic:configure --profile stacks
	164	buddy mail:storage:restore:check --profile stacks
	165	```
	166
	167	The daily job creates a verified SQLite-consistent local snapshot, uploads it
	168	through Restic, applies 7 daily / 5 weekly / 12 monthly retention, prunes,
	169	and performs a repository data check. The restore check downloads the newest
	170	snapshot to a temporary directory on the encrypted filesystem, validates both
	171	archives, runs SQLite `quick_check`, and confirms that mail and forwarding data
	172	are present. An hourly health timer fails when the encrypted mount is missing,
	173	usage crosses configured thresholds, or the newest successful off-host backup
	174	is stale.
	175
	176	## Recovery order
	177
	178	1. Attach and mount the Hetzner data volume.
	179	2. Retrieve either the AWS secret or independent Keychain recovery key.
	180	3. Unlock `/var/lib/mail-storage.luks`.
	181	4. Start `mail.service` and validate SQLite plus Maildir counts.
	182	5. If the volume is unavailable, restore the latest Restic snapshot and its
	183	environment archive onto a newly encrypted volume.
	184
	185	Implementation and operational follow-up are tracked in
	186	[mail-os/mail#4](https://github.com/mail-os/mail/issues/4).

packages/cloud/cloud.config.tsmodified+9-0

Changes to packages/cloud/cloud.config.ts
		@@ -70,6 +70,7 @@ const mailConfig = {
70	70	'python3',
71	71	'python3-pip',
72	72	'openssl',
	73	'cryptsetup',
73	74	'sqlite',
74	75	'fail2ban',
75	76	'bind-utils',
		@@ -327,6 +328,14 @@ chown -R mail-server:mail-server ${cfg.paths.logDir}
327	328	chown -R mail-server:mail-server ${cfg.paths.mailDir}
328	329	chown -R mail-server:mail-server ${cfg.paths.installDir}/mail
329	330
	331	# Encrypt every persistent mail path by default. The LUKS2 volume key is
	332	# machine-bound in systemd's encrypted credential store and is never exposed
	333	# to the mail-server service as a persistent plaintext file.
	334	chmod +x ${cfg.paths.installDir}/scripts/setup-encrypted-storage.sh
	335	MAIL_ROOT=${cfg.paths.installDir} \
	336	MAIL_ENCRYPTED_IMAGE_SIZE="\${MAIL_ENCRYPTED_IMAGE_SIZE:-4G}" \
	337	${cfg.paths.installDir}/scripts/setup-encrypted-storage.sh
	338
330	339	# Get instance metadata
331	340	echo "Getting instance metadata..."
332	341	TOKEN=\$(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")

packages/zig/scripts/hetzner-provision.shmodified+15-14

Changes to packages/zig/scripts/hetzner-provision.sh
		@@ -11,6 +11,8 @@ BACKUP_DIR="${BACKUP_DIR:-/opt/mail/backups}"
11	11	RETAIN_DAYS="${RETAIN_DAYS:-7}"
12	12	SWAP_GB="${SWAP_GB:-2}"
13	13	FAIL2BAN_IGNORE_IPS="${FAIL2BAN_IGNORE_IPS:-127.0.0.1/8 ::1}"
	14	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
	15	REPO_ROOT="$(cd "${SCRIPT_DIR}/../../.." && pwd)"
14	16
15	17	echo "== memory safety =="
16	18	# The production box is shared with Bun applications. Give short deployment
		@@ -79,26 +81,25 @@ systemctl enable --now fail2ban >/dev/null 2>&1 \|\| true
79	81	systemctl restart fail2ban \|\| true
80	82	echo "fail2ban: $(systemctl is-active fail2ban)"
81	83
	84	echo "== encrypted storage =="
	85	if ! command -v cryptsetup >/dev/null 2>&1; then
	86	export DEBIAN_FRONTEND=noninteractive
	87	apt-get update -qq && apt-get install -y -qq cryptsetup >/dev/null
	88	fi
	89	chmod +x "${REPO_ROOT}/scripts/setup-encrypted-storage.sh"
	90	MAIL_ROOT="${MAIL_DIR}" "${REPO_ROOT}/scripts/setup-encrypted-storage.sh"
	91
82	92	echo "== backups =="
83	93	mkdir -p "$BACKUP_DIR"
84		cat > "$MAIL_DIR/backup.sh" <<BK
85		#!/usr/bin/env bash
86		set -euo pipefail
87		TS=\$(date +%Y%m%d-%H%M%S)
88		OUT="$BACKUP_DIR/mail-backup-\$TS.tgz"
89		# mailboxes + databases + DKIM key + config (exclude the backups dir itself)
90		tar czf "\$OUT" \\
91		-C "$MAIL_DIR" mail smtp.db caldav.db dkim 2>/dev/null \|\| true
92		[ -f /etc/mail/mail.env ] && tar rf "\${OUT%.tgz}.env.tar" -C /etc/mail mail.env 2>/dev/null \|\| true
93		# prune old backups
94		find "$BACKUP_DIR" -name 'mail-backup-*.tgz' -mtime +$RETAIN_DAYS -delete 2>/dev/null \|\| true
95		echo "backup written: \$OUT (\$(du -h "\$OUT" 2>/dev/null \| cut -f1))"
96		BK
97		chmod +x "$MAIL_DIR/backup.sh"
	94	install -m 0755 "${REPO_ROOT}/scripts/mail-backup.sh" "$MAIL_DIR/backup.sh"
	95	install -m 0755 "${REPO_ROOT}/scripts/mail-forward-compile" /usr/local/sbin/mail-forward-compile
	96	install -m 0755 "${REPO_ROOT}/scripts/maildir-migrate.sh" /usr/local/sbin/maildir-migrate
98	97
99	98	cat > /etc/systemd/system/mail-backup.service <<'U'
100	99	[Unit]
101	100	Description=Mail server backup
	101	Requires=mail-storage.service
	102	After=mail-storage.service
102	103	[Service]
103	104	Type=oneshot
104	105	ExecStart=/opt/mail/backup.sh

scripts/mail-backup.shadded+64-0

Changes to scripts/mail-backup.sh
		@@ -0,0 +1,64 @@
	1	#!/usr/bin/env bash
	2	set -euo pipefail
	3
	4	MAIL_DIR="${MAIL_DIR:-/opt/mail}"
	5	BACKUP_DIR="${BACKUP_DIR:-${MAIL_DIR}/backups}"
	6	RETAIN_DAYS="${RETAIN_DAYS:-7}"
	7
	8	[[ "${MAIL_DIR}" == /* && "${MAIL_DIR}" != "/" ]] \|\| {
	9	echo "mail-backup: MAIL_DIR must be an absolute non-root path" >&2
	10	exit 1
	11	}
	12	[[ "${BACKUP_DIR}" == /* && "${BACKUP_DIR}" != "/" ]] \|\| {
	13	echo "mail-backup: BACKUP_DIR must be an absolute non-root path" >&2
	14	exit 1
	15	}
	16
	17	install -d -m 0700 -o root -g root "${BACKUP_DIR}"
	18
	19	timestamp="$(date -u +%Y%m%d-%H%M%S)"
	20	output="${BACKUP_DIR}/mail-backup-${timestamp}.tgz"
	21	partial="${output}.partial"
	22	work_dir="$(mktemp -d "${BACKUP_DIR}/.mail-backup-${timestamp}.XXXXXX")"
	23
	24	cleanup() {
	25	rm -f "${partial}"
	26	find "${work_dir}" -depth -delete 2>/dev/null \|\| true
	27	}
	28	trap cleanup EXIT
	29
	30	declare -a snapshots=()
	31	for database in smtp.db caldav.db; do
	32	if [[ -f "${MAIL_DIR}/${database}" ]]; then
	33	sqlite3 "${MAIL_DIR}/${database}" ".backup '${work_dir}/${database}'"
	34	sqlite3 -readonly "${work_dir}/${database}" "PRAGMA quick_check;" \
	35	\| grep -qx ok
	36	snapshots+=("${database}")
	37	fi
	38	done
	39
	40	declare -a persistent=()
	41	for path in mail dkim forwards.json forwards.sieve cert-backups; do
	42	[[ -e "${MAIL_DIR}/${path}" ]] && persistent+=("${path}")
	43	done
	44
	45	tar --dereference --exclude='mail//.' -czf "${partial}" \
	46	-C "${MAIL_DIR}" "${persistent[@]}" \
	47	-C "${work_dir}" "${snapshots[@]}"
	48	tar tzf "${partial}" >/dev/null
	49	chmod 0600 "${partial}"
	50	mv "${partial}" "${output}"
	51
	52	if [[ -f /etc/mail/mail.env ]]; then
	53	env_output="${output%.tgz}.env.tar"
	54	tar cf "${env_output}.partial" -C /etc/mail mail.env
	55	tar tf "${env_output}.partial" >/dev/null
	56	chmod 0600 "${env_output}.partial"
	57	mv "${env_output}.partial" "${env_output}"
	58	fi
	59
	60	find "${BACKUP_DIR}" -maxdepth 1 -type f \
	61	$ -name 'mail-backup-.tgz' -o -name 'mail-backup-.env.tar' $ \
	62	-mtime "+${RETAIN_DAYS}" -delete
	63
	64	echo "backup written: ${output} ($(du -h "${output}" \| cut -f1))"

scripts/mail-restic-backup.shadded+73-0

Changes to scripts/mail-restic-backup.sh
		@@ -0,0 +1,73 @@
	1	#!/usr/bin/env bash
	2	set -euo pipefail
	3
	4	# Create a transactionally consistent local archive, then copy it into an
	5	# authenticated, client-side encrypted Restic repository. S3's SSE-KMS remains
	6	# enabled as a second encryption layer; Restic encryption happens first.
	7
	8	MAIL_ROOT="${MAIL_ROOT:-/opt/mail}"
	9	RESTIC_CONFIG_DIR="${MAIL_RESTIC_CONFIG_DIR:-/var/lib/mail-storage/data/restic}"
	10	RESTIC_ENV="${MAIL_RESTIC_ENV:-${RESTIC_CONFIG_DIR}/restic.env}"
	11	BACKUP_SCRIPT="${MAIL_BACKUP_SCRIPT:-${MAIL_ROOT}/backup.sh}"
	12	STATUS_FILE="${MAIL_RESTIC_STATUS_FILE:-${RESTIC_CONFIG_DIR}/last-success}"
	13	CHECK_SUBSET="${MAIL_RESTIC_CHECK_SUBSET:-5%}"
	14
	15	die() {
	16	echo "mail-restic-backup: $*" >&2
	17	exit 1
	18	}
	19
	20	[[ "${EUID}" -eq 0 ]] \|\| die "must run as root"
	21	mountpoint -q /var/lib/mail-storage \|\| die "encrypted mail storage is not mounted"
	22	[[ -x "${BACKUP_SCRIPT}" ]] \|\| die "${BACKUP_SCRIPT} is not executable"
	23	[[ -r "${RESTIC_ENV}" ]] \|\| die "${RESTIC_ENV} is unavailable"
	24	command -v restic >/dev/null 2>&1 \|\| die "restic is not installed"
	25
	26	set -a
	27	# shellcheck disable=SC1090
	28	source "${RESTIC_ENV}"
	29	set +a
	30
	31	: "${RESTIC_REPOSITORY:?RESTIC_REPOSITORY is required}"
	32	: "${RESTIC_PASSWORD_FILE:?RESTIC_PASSWORD_FILE is required}"
	33	: "${AWS_ACCESS_KEY_ID:?AWS_ACCESS_KEY_ID is required}"
	34	: "${AWS_SECRET_ACCESS_KEY:?AWS_SECRET_ACCESS_KEY is required}"
	35
	36	umask 077
	37	"${BACKUP_SCRIPT}"
	38
	39	latest="$(find -L "${MAIL_ROOT}/backups" -maxdepth 1 -type f -name 'mail-backup-*.tgz' \
	40	-printf '%T@ %p\n' \| sort -nr \| head -1 \| cut -d' ' -f2-)"
	41	[[ -n "${latest}" && -f "${latest}" ]] \|\| die "no local backup archive was produced"
	42	tar -tzf "${latest}" >/dev/null
	43
	44	declare -a inputs=("${latest}")
	45	env_archive="${latest%.tgz}.env.tar"
	46	[[ -f "${env_archive}" ]] && inputs+=("${env_archive}")
	47
	48	restic backup \
	49	--host "$(hostname -f 2>/dev/null \|\| hostname)" \
	50	--tag mail \
	51	--tag production \
	52	"${inputs[@]}"
	53
	54	restic forget \
	55	--tag mail \
	56	--keep-daily 7 \
	57	--keep-weekly 5 \
	58	--keep-monthly 12 \
	59	--prune
	60
	61	restic check --read-data-subset="${CHECK_SUBSET}"
	62
	63	install -d -m 0700 "$(dirname "${STATUS_FILE}")"
	64	status_tmp="${STATUS_FILE}.tmp"
	65	{
	66	echo "timestamp=$(date -u +%s)"
	67	echo "archive=$(basename "${latest}")"
	68	echo "repository=${RESTIC_REPOSITORY}"
	69	} > "${status_tmp}"
	70	chmod 0600 "${status_tmp}"
	71	mv "${status_tmp}" "${STATUS_FILE}"
	72
	73	echo "mail-restic-backup: uploaded and verified $(basename "${latest}")"

scripts/mail-restic-restore-check.shadded+58-0

Changes to scripts/mail-restic-restore-check.sh
		@@ -0,0 +1,58 @@
	1	#!/usr/bin/env bash
	2	set -euo pipefail
	3
	4	# Restore the newest Restic snapshot into a disposable directory on the
	5	# encrypted filesystem and validate the archive, databases, messages, and
	6	# forwarding configuration without touching live data.
	7
	8	RESTIC_CONFIG_DIR="${MAIL_RESTIC_CONFIG_DIR:-/var/lib/mail-storage/data/restic}"
	9	RESTIC_ENV="${MAIL_RESTIC_ENV:-${RESTIC_CONFIG_DIR}/restic.env}"
	10	RESTORE_PARENT="${MAIL_RESTORE_CHECK_PARENT:-/var/lib/mail-storage}"
	11
	12	die() {
	13	echo "mail-restic-restore-check: $*" >&2
	14	exit 1
	15	}
	16
	17	[[ "${EUID}" -eq 0 ]] \|\| die "must run as root"
	18	mountpoint -q /var/lib/mail-storage \|\| die "encrypted mail storage is not mounted"
	19	[[ -r "${RESTIC_ENV}" ]] \|\| die "${RESTIC_ENV} is unavailable"
	20	command -v restic >/dev/null 2>&1 \|\| die "restic is not installed"
	21
	22	set -a
	23	# shellcheck disable=SC1090
	24	source "${RESTIC_ENV}"
	25	set +a
	26
	27	: "${RESTIC_REPOSITORY:?RESTIC_REPOSITORY is required}"
	28	: "${RESTIC_PASSWORD_FILE:?RESTIC_PASSWORD_FILE is required}"
	29
	30	restore_dir="$(mktemp -d "${RESTORE_PARENT}/restic-restore-check.XXXXXX")"
	31	archive_dir="${restore_dir}/archive"
	32	unpack_dir="${restore_dir}/unpacked"
	33	cleanup() {
	34	find "${restore_dir}" -depth -delete 2>/dev/null \|\| true
	35	}
	36	trap cleanup EXIT
	37
	38	install -d -m 0700 "${archive_dir}" "${unpack_dir}"
	39	restic restore latest --tag mail --target "${archive_dir}"
	40
	41	archive="$(find "${archive_dir}" -type f -name 'mail-backup-*.tgz' \
	42	-printf '%T@ %p\n' \| sort -nr \| head -1 \| cut -d' ' -f2-)"
	43	[[ -n "${archive}" && -f "${archive}" ]] \|\| die "restored snapshot has no mail archive"
	44	tar -tzf "${archive}" >/dev/null
	45	tar -xzf "${archive}" -C "${unpack_dir}"
	46
	47	for database in smtp.db caldav.db; do
	48	if [[ -f "${unpack_dir}/${database}" ]]; then
	49	sqlite3 -readonly "${unpack_dir}/${database}" 'PRAGMA quick_check;' \| grep -qx ok \
	50	\|\| die "${database} failed quick_check"
	51	fi
	52	done
	53
	54	message_count="$(find "${unpack_dir}/mail" -type f 2>/dev/null \| wc -l)"
	55	[[ "${message_count}" -gt 0 ]] \|\| die "restored archive contains no messages"
	56	[[ -f "${unpack_dir}/forwards.json" ]] \|\| die "restored archive lacks forwarding rules"
	57
	58	echo "mail-restic-restore-check: ok (${message_count} messages)"

scripts/mail-storage-external.shadded+253-0

Changes to scripts/mail-storage-external.sh
		@@ -0,0 +1,253 @@
	1	#!/usr/bin/env bash
	2	set -euo pipefail
	3
	4	# Manage an externally escrowed LUKS2 mail volume. In external mode no volume
	5	# key or cloud credential is stored on the mail host. `unlock` accepts the raw
	6	# key on stdin, passes it directly to cryptsetup, and never writes it to disk.
	7
	8	COMMAND="${1:-status}"
	9	MAIL_SERVICE="${MAIL_SERVICE:-mail}"
	10	IMAGE_PATH="${MAIL_ENCRYPTED_IMAGE:-/var/lib/mail-storage.luks}"
	11	MOUNT_PATH="${MAIL_ENCRYPTED_MOUNT:-/var/lib/mail-storage}"
	12	MAPPER_NAME="${MAIL_ENCRYPTED_MAPPER:-mail-storage}"
	13	CREDENTIAL_PATH="${MAIL_ENCRYPTED_CREDENTIAL:-/etc/credstore.encrypted/mail-storage-key.cred}"
	14	UNIT_PATH="/etc/systemd/system/mail-storage.service"
	15	UNIT_BACKUP="/etc/systemd/system/mail-storage.service.machine-bound-backup"
	16	MAIL_DROPIN="/etc/systemd/system/${MAIL_SERVICE}.service.d/encrypted-storage.conf"
	17	BACKUP_DROPIN="/etc/systemd/system/mail-backup.service.d/encrypted-storage.conf"
	18
	19	die() {
	20	echo "mail-storage: $*" >&2
	21	return 1
	22	}
	23
	24	[[ "${EUID}" -eq 0 ]] \|\| die "must run as root"
	25	[[ "${IMAGE_PATH}" == /* && "${IMAGE_PATH}" != "/" ]] \|\| die "invalid image path"
	26	[[ "${MOUNT_PATH}" == /* && "${MOUNT_PATH}" != "/" ]] \|\| die "invalid mount path"
	27	[[ "${MAPPER_NAME}" =~ ^[A-Za-z0-9_.+-]+$ ]] \|\| die "invalid mapper name"
	28
	29	CRYPTSETUP="$(command -v cryptsetup \|\| true)"
	30	MOUNT_BIN="$(command -v mount \|\| true)"
	31	UMOUNT_BIN="$(command -v umount \|\| true)"
	32	MOUNTPOINT_BIN="$(command -v mountpoint \|\| true)"
	33	LOSETUP_BIN="$(command -v losetup \|\| true)"
	34	[[ -n "${CRYPTSETUP}" && -n "${MOUNT_BIN}" && -n "${UMOUNT_BIN}" && -n "${MOUNTPOINT_BIN}" && -n "${LOSETUP_BIN}" ]] \
	35	\|\| die "cryptsetup and mount utilities are required"
	36
	37	configure_external() {
	38	[[ -f "${IMAGE_PATH}" ]] \|\| die "${IMAGE_PATH} does not exist"
	39	[[ -f "${CREDENTIAL_PATH}" ]] \|\| die "machine-bound credential is unavailable for rollback"
	40
	41	systemctl stop "${MAIL_SERVICE}.service" 2>/dev/null \|\| true
	42	systemctl stop mail-backup.service 2>/dev/null \|\| true
	43	systemctl stop mail-storage.service 2>/dev/null \|\| true
	44
	45	if [[ -f "${UNIT_PATH}" && ! -f "${UNIT_BACKUP}" ]]; then
	46	cp -p "${UNIT_PATH}" "${UNIT_BACKUP}"
	47	fi
	48
	49	cat > "${UNIT_PATH}" <<EOF
	50	[Unit]
	51	Description=Externally unlocked encrypted mail data filesystem
	52	Before=${MAIL_SERVICE}.service mail-backup.service
	53
	54	[Service]
	55	Type=oneshot
	56	RemainAfterExit=yes
	57	ExecStart=${MOUNTPOINT_BIN} -q ${MOUNT_PATH}
	58	ExecStop=-${UMOUNT_BIN} ${MOUNT_PATH}
	59	ExecStop=-${CRYPTSETUP} close ${MAPPER_NAME}
	60	TimeoutStopSec=120
	61	EOF
	62
	63	install -d -m 0755 "$(dirname "${MAIL_DROPIN}")" "$(dirname "${BACKUP_DROPIN}")"
	64	cat > "${MAIL_DROPIN}" <<EOF
	65	[Unit]
	66	After=mail-storage.service
	67
	68	[Service]
	69	ExecCondition=${MOUNTPOINT_BIN} -q ${MOUNT_PATH}
	70	EOF
	71	cp "${MAIL_DROPIN}" "${BACKUP_DROPIN}"
	72
	73	systemctl disable mail-storage.service >/dev/null 2>&1 \|\| true
	74	systemctl daemon-reload
	75	echo "mail-storage: external mode configured; volume is locked"
	76	}
	77
	78	unlock_external() {
	79	[[ -f "${IMAGE_PATH}" ]] \|\| die "${IMAGE_PATH} does not exist"
	80
	81	if ! "${CRYPTSETUP}" status "${MAPPER_NAME}" >/dev/null 2>&1; then
	82	backing_file="$(readlink -f "${IMAGE_PATH}")"
	83	loop_device="$("${LOSETUP_BIN}" -j "${backing_file}" \| awk -F: 'NR == 1 { print $1 }')"
	84	created_loop=false
	85	if [[ -z "${loop_device}" ]]; then
	86	loop_device="$("${LOSETUP_BIN}" --find --show "${backing_file}")"
	87	created_loop=true
	88	fi
	89	# cryptsetup consumes the raw key from this process's stdin. No shell
	90	# variable, argument, environment entry, or persistent file holds it.
	91	if ! "${CRYPTSETUP}" open --type luks2 --key-file - "${loop_device}" "${MAPPER_NAME}"; then
	92	[[ "${created_loop}" == "true" ]] && "${LOSETUP_BIN}" -d "${loop_device}" \|\| true
	93	return 1
	94	fi
	95	else
	96	# Consume stdin even when already unlocked so callers never get a broken
	97	# pipe while streaming a secret.
	98	dd of=/dev/null status=none
	99	fi
	100
	101	if ! "${MOUNTPOINT_BIN}" -q "${MOUNT_PATH}"; then
	102	install -d -m 0750 -o root -g mail-server "${MOUNT_PATH}"
	103	"${MOUNT_BIN}" -o noatime,nodev,nosuid "/dev/mapper/${MAPPER_NAME}" "${MOUNT_PATH}"
	104	fi
	105
	106	systemctl reset-failed mail-storage.service "${MAIL_SERVICE}.service" 2>/dev/null \|\| true
	107	systemctl start mail-storage.service
	108	systemctl start "${MAIL_SERVICE}.service"
	109	sleep 2
	110	systemctl is-active --quiet mail-storage.service \|\| die "storage unit is not active"
	111	systemctl is-active --quiet "${MAIL_SERVICE}.service" \|\| die "mail service is not active"
	112	echo "mail-storage: unlocked; mail is active"
	113	}
	114
	115	lock_external() {
	116	systemctl stop mail-backup.service 2>/dev/null \|\| true
	117	systemctl stop "${MAIL_SERVICE}.service" 2>/dev/null \|\| true
	118	if systemctl is-active --quiet mail-storage.service; then
	119	systemctl stop mail-storage.service
	120	else
	121	"${MOUNTPOINT_BIN}" -q "${MOUNT_PATH}" && "${UMOUNT_BIN}" "${MOUNT_PATH}" \|\| true
	122	"${CRYPTSETUP}" status "${MAPPER_NAME}" >/dev/null 2>&1 \
	123	&& "${CRYPTSETUP}" close "${MAPPER_NAME}" \|\| true
	124	fi
	125	backing_file="$(readlink -f "${IMAGE_PATH}")"
	126	while IFS=: read -r loop_device _; do
	127	[[ -n "${loop_device}" ]] && "${LOSETUP_BIN}" -d "${loop_device}" 2>/dev/null \|\| true
	128	done < <("${LOSETUP_BIN}" -j "${backing_file}")
	129	echo "mail-storage: locked; mail is stopped"
	130	}
	131
	132	finalize_external() {
	133	"${MOUNTPOINT_BIN}" -q "${MOUNT_PATH}" \|\| die "refusing to remove local credential while storage is locked"
	134	systemctl is-active --quiet "${MAIL_SERVICE}.service" \
	135	\|\| die "refusing to remove local credential while mail is inactive"
	136	[[ -f "${UNIT_BACKUP}" ]] \|\| die "machine-bound unit backup is missing"
	137
	138	shred -u "${CREDENTIAL_PATH}" 2>/dev/null \|\| rm -f "${CREDENTIAL_PATH}"
	139	chmod 0600 "${UNIT_BACKUP}"
	140	echo "mail-storage: local volume credential removed; external mode finalized"
	141	}
	142
	143	rollback_machine() {
	144	[[ -f "${UNIT_BACKUP}" ]] \|\| die "machine-bound unit backup is missing"
	145	[[ -f "${CREDENTIAL_PATH}" ]] \|\| die "machine-bound credential is missing"
	146
	147	systemctl stop "${MAIL_SERVICE}.service" 2>/dev/null \|\| true
	148	systemctl stop mail-storage.service 2>/dev/null \|\| true
	149	cp -p "${UNIT_BACKUP}" "${UNIT_PATH}"
	150	cat > "${MAIL_DROPIN}" <<EOF
	151	[Unit]
	152	Requires=mail-storage.service
	153	After=mail-storage.service
	154	EOF
	155	cp "${MAIL_DROPIN}" "${BACKUP_DROPIN}"
	156	systemctl daemon-reload
	157	systemctl enable --now mail-storage.service
	158	systemctl reset-failed "${MAIL_SERVICE}.service" 2>/dev/null \|\| true
	159	systemctl start "${MAIL_SERVICE}.service"
	160	echo "mail-storage: restored machine-bound unlock mode"
	161	}
	162
	163	status_external() {
	164	local mode="machine-bound"
	165	if [[ ! -f "${CREDENTIAL_PATH}" ]]; then
	166	mode="external"
	167	elif ! grep -q '^LoadCredentialEncrypted=' "${UNIT_PATH}" 2>/dev/null; then
	168	mode="external-pending-finalization"
	169	fi
	170	echo "mode=${mode}"
	171	if [[ -f "${CREDENTIAL_PATH}" ]]; then
	172	echo "local_credential=present"
	173	else
	174	echo "local_credential=absent"
	175	fi
	176	if "${CRYPTSETUP}" status "${MAPPER_NAME}" >/dev/null 2>&1; then
	177	echo "mapper=active"
	178	else
	179	echo "mapper=locked"
	180	fi
	181	if "${MOUNTPOINT_BIN}" -q "${MOUNT_PATH}"; then
	182	echo "mount=active"
	183	else
	184	echo "mount=inactive"
	185	fi
	186	echo "mail=$(systemctl is-active "${MAIL_SERVICE}.service" 2>/dev/null \|\| true)"
	187	echo "backup_timer=$(systemctl is-active mail-backup.timer 2>/dev/null \|\| true)"
	188	}
	189
	190	add_recovery_key() {
	191	local key_dir current_key recovery_key
	192	key_dir="$(mktemp -d /run/mail-storage-recovery.XXXXXX)"
	193	current_key="${key_dir}/current"
	194	recovery_key="${key_dir}/recovery"
	195	cleanup_recovery_keys() {
	196	[[ -n "${current_key:-}" ]] && shred -u "${current_key}" 2>/dev/null \|\| true
	197	[[ -n "${recovery_key:-}" ]] && shred -u "${recovery_key}" 2>/dev/null \|\| true
	198	[[ -n "${current_key:-}" ]] && rm -f "${current_key}" \|\| true
	199	[[ -n "${recovery_key:-}" ]] && rm -f "${recovery_key}" \|\| true
	200	[[ -n "${key_dir:-}" ]] && rmdir "${key_dir}" 2>/dev/null \|\| true
	201	}
	202	trap cleanup_recovery_keys EXIT
	203
	204	dd if=/dev/stdin of="${current_key}" bs=64 count=1 iflag=fullblock status=none
	205	dd if=/dev/stdin of="${recovery_key}" bs=64 count=1 iflag=fullblock status=none
	206	[[ "$(stat -c %s "${current_key}")" -eq 64 ]] \|\| die "invalid current key length"
	207	[[ "$(stat -c %s "${recovery_key}")" -eq 64 ]] \|\| die "invalid recovery key length"
	208	chmod 0400 "${current_key}" "${recovery_key}"
	209
	210	"${CRYPTSETUP}" luksAddKey \
	211	--key-file "${current_key}" \
	212	--new-keyfile "${recovery_key}" \
	213	"${IMAGE_PATH}"
	214	"${CRYPTSETUP}" open \
	215	--test-passphrase \
	216	--type luks2 \
	217	--key-file "${recovery_key}" \
	218	"${IMAGE_PATH}"
	219	echo "mail-storage: independent recovery keyslot added and verified"
	220	}
	221
	222	remove_keyslot() {
	223	local slot="${2:-}" key_dir current_key
	224	[[ "${slot}" =~ ^[0-9]+$ ]] \|\| die "a numeric LUKS keyslot is required"
	225	key_dir="$(mktemp -d /run/mail-storage-keyslot.XXXXXX)"
	226	current_key="${key_dir}/current"
	227	cleanup_keyslot_key() {
	228	[[ -n "${current_key:-}" ]] && shred -u "${current_key}" 2>/dev/null \|\| true
	229	[[ -n "${current_key:-}" ]] && rm -f "${current_key}" \|\| true
	230	[[ -n "${key_dir:-}" ]] && rmdir "${key_dir}" 2>/dev/null \|\| true
	231	}
	232	trap cleanup_keyslot_key EXIT
	233
	234	dd if=/dev/stdin of="${current_key}" bs=64 count=1 iflag=fullblock status=none
	235	[[ "$(stat -c %s "${current_key}")" -eq 64 ]] \|\| die "invalid current key length"
	236	chmod 0400 "${current_key}"
	237	"${CRYPTSETUP}" luksKillSlot --key-file "${current_key}" "${IMAGE_PATH}" "${slot}"
	238	echo "mail-storage: removed keyslot ${slot}"
	239	}
	240
	241	case "${COMMAND}" in
	242	configure-external) configure_external ;;
	243	unlock) unlock_external ;;
	244	lock) lock_external ;;
	245	finalize-external) finalize_external ;;
	246	rollback-machine) rollback_machine ;;
	247	add-recovery-key) add_recovery_key ;;
	248	remove-keyslot) remove_keyslot "$@" ;;
	249	status) status_external ;;
	250	*)
	251	die "usage: $0 {configure-external\|unlock\|lock\|finalize-external\|rollback-machine\|add-recovery-key\|remove-keyslot <slot>\|status}"
	252	;;
	253	esac

scripts/mail-storage-health.shadded+31-0

Changes to scripts/mail-storage-health.sh
		@@ -0,0 +1,31 @@
	1	#!/usr/bin/env bash
	2	set -euo pipefail
	3
	4	MOUNT_PATH="${MAIL_ENCRYPTED_MOUNT:-/var/lib/mail-storage}"
	5	STATUS_FILE="${MAIL_RESTIC_STATUS_FILE:-${MOUNT_PATH}/data/restic/last-success}"
	6	WARN_PERCENT="${MAIL_STORAGE_WARN_PERCENT:-70}"
	7	CRITICAL_PERCENT="${MAIL_STORAGE_CRITICAL_PERCENT:-85}"
	8	MAX_BACKUP_AGE="${MAIL_BACKUP_MAX_AGE_SECONDS:-90000}"
	9
	10	die() {
	11	echo "mail-storage-health: $*" >&2
	12	exit 1
	13	}
	14
	15	mountpoint -q "${MOUNT_PATH}" \|\| die "${MOUNT_PATH} is not mounted"
	16	usage="$(df -P "${MOUNT_PATH}" \| awk 'NR == 2 { gsub(/%/, "", $5); print $5 }')"
	17	[[ "${usage}" =~ ^[0-9]+$ ]] \|\| die "could not read filesystem usage"
	18
	19	if (( usage >= CRITICAL_PERCENT )); then
	20	die "encrypted storage is ${usage}% full (critical threshold ${CRITICAL_PERCENT}%)"
	21	elif (( usage >= WARN_PERCENT )); then
	22	echo "mail-storage-health: warning: encrypted storage is ${usage}% full" >&2
	23	fi
	24
	25	[[ -r "${STATUS_FILE}" ]] \|\| die "no successful off-host backup status exists"
	26	timestamp="$(awk -F= '$1 == "timestamp" { print $2; exit }' "${STATUS_FILE}")"
	27	[[ "${timestamp}" =~ ^[0-9]+$ ]] \|\| die "backup status timestamp is invalid"
	28	age="$(( $(date -u +%s) - timestamp ))"
	29	(( age <= MAX_BACKUP_AGE )) \|\| die "latest verified off-host backup is ${age}s old"
	30
	31	echo "mail-storage-health: ok (usage=${usage}%, backup_age=${age}s)"

scripts/migrate-encrypted-storage-volume.shadded+109-0

Changes to scripts/migrate-encrypted-storage-volume.sh
		@@ -0,0 +1,109 @@
	1	#!/usr/bin/env bash
	2	set -euo pipefail
	3
	4	# Move the existing LUKS image onto a dedicated block volume without changing
	5	# its UUID, keyslots, or public path. `prepare` intentionally leaves mail
	6	# locked; the external-key controller must unlock and validate before
	7	# `finalize` removes the root-disk rollback image.
	8
	9	COMMAND="${1:-status}"
	10	DEVICE="${MAIL_VOLUME_DEVICE:-${2:-}}"
	11	VOLUME_MOUNT="${MAIL_VOLUME_MOUNT:-/var/lib/mail-volume}"
	12	IMAGE_PATH="${MAIL_ENCRYPTED_IMAGE:-/var/lib/mail-storage.luks}"
	13	TARGET_SIZE="${MAIL_ENCRYPTED_IMAGE_SIZE:-16G}"
	14	VOLUME_IMAGE="${VOLUME_MOUNT}/mail-storage.luks"
	15	ROOT_BACKUP="${IMAGE_PATH}.root-backup"
	16
	17	die() {
	18	echo "mail-volume-migrate: $*" >&2
	19	exit 1
	20	}
	21
	22	[[ "${EUID}" -eq 0 ]] \|\| die "must run as root"
	23	[[ "${VOLUME_MOUNT}" == /* && "${VOLUME_MOUNT}" != "/" ]] \|\| die "invalid mount path"
	24	[[ "${IMAGE_PATH}" == /* && "${IMAGE_PATH}" != "/" ]] \|\| die "invalid image path"
	25
	26	ensure_volume_mount() {
	27	[[ -n "${DEVICE}" ]] \|\| die "MAIL_VOLUME_DEVICE or device argument is required"
	28	[[ -b "${DEVICE}" ]] \|\| die "${DEVICE} is not a block device"
	29	install -d -m 0700 "${VOLUME_MOUNT}"
	30
	31	if ! blkid "${DEVICE}" >/dev/null 2>&1; then
	32	mkfs.ext4 -q -L mail-volume "${DEVICE}"
	33	fi
	34	uuid="$(blkid -s UUID -o value "${DEVICE}")"
	35	[[ -n "${uuid}" ]] \|\| die "could not resolve volume UUID"
	36
	37	if ! grep -qE "^[^#]+[[:space:]]+${VOLUME_MOUNT//\//\\/}[[:space:]]" /etc/fstab; then
	38	echo "UUID=${uuid} ${VOLUME_MOUNT} ext4 defaults,noatime,nodev,nosuid 0 2" >> /etc/fstab
	39	fi
	40	mountpoint -q "${VOLUME_MOUNT}" \|\| mount "${VOLUME_MOUNT}"
	41	chmod 0700 "${VOLUME_MOUNT}"
	42	}
	43
	44	prepare() {
	45	[[ -f "${IMAGE_PATH}" && ! -L "${IMAGE_PATH}" ]] \
	46	\|\| die "${IMAGE_PATH} must be the current root-disk image"
	47	[[ ! -e "${ROOT_BACKUP}" ]] \|\| die "${ROOT_BACKUP} already exists"
	48	ensure_volume_mount
	49
	50	/usr/local/sbin/mail-storage-external lock
	51	if [[ -e "${VOLUME_IMAGE}" ]]; then
	52	root_size="$(stat -c %s "${IMAGE_PATH}")"
	53	if ! cmp -n "${root_size}" "${IMAGE_PATH}" "${VOLUME_IMAGE}"; then
	54	cp --sparse=always --reflink=auto "${IMAGE_PATH}" "${VOLUME_IMAGE}.new"
	55	chmod 0600 "${VOLUME_IMAGE}.new"
	56	cmp -s "${IMAGE_PATH}" "${VOLUME_IMAGE}.new" \|\| die "refreshed image failed byte comparison"
	57	sync "${VOLUME_IMAGE}.new"
	58	mv "${VOLUME_IMAGE}.new" "${VOLUME_IMAGE}"
	59	fi
	60	else
	61	cp --sparse=always --reflink=auto "${IMAGE_PATH}" "${VOLUME_IMAGE}.new"
	62	chmod 0600 "${VOLUME_IMAGE}.new"
	63	cmp -s "${IMAGE_PATH}" "${VOLUME_IMAGE}.new" \|\| die "copied image failed byte comparison"
	64	sync "${VOLUME_IMAGE}.new"
	65	mv "${VOLUME_IMAGE}.new" "${VOLUME_IMAGE}"
	66	fi
	67	mv "${IMAGE_PATH}" "${ROOT_BACKUP}"
	68	ln -s "${VOLUME_IMAGE}" "${IMAGE_PATH}"
	69	truncate -s "${TARGET_SIZE}" "${VOLUME_IMAGE}"
	70	sync "${VOLUME_IMAGE}"
	71	echo "mail-volume-migrate: prepared; external unlock is required"
	72	}
	73
	74	finalize() {
	75	mountpoint -q /var/lib/mail-storage \|\| die "encrypted mail storage is not mounted"
	76	systemctl is-active --quiet mail.service \|\| die "mail is not active"
	77	[[ -L "${IMAGE_PATH}" && "$(readlink "${IMAGE_PATH}")" == "${VOLUME_IMAGE}" ]] \
	78	\|\| die "mail image does not point to dedicated volume"
	79
	80	loop_device="$(cryptsetup status mail-storage \| awk '/device:/ { print $2; exit }')"
	81	[[ -n "${loop_device}" ]] \|\| die "could not identify loop device"
	82	losetup -c "${loop_device}"
	83	resize2fs /dev/mapper/mail-storage
	84	rm -f "${ROOT_BACKUP}"
	85	echo "mail-volume-migrate: finalized"
	86	}
	87
	88	rollback() {
	89	/usr/local/sbin/mail-storage-external lock
	90	[[ -f "${ROOT_BACKUP}" ]] \|\| die "root rollback image is unavailable"
	91	[[ -L "${IMAGE_PATH}" ]] \|\| die "${IMAGE_PATH} is not a symlink"
	92	unlink "${IMAGE_PATH}"
	93	mv "${ROOT_BACKUP}" "${IMAGE_PATH}"
	94	echo "mail-volume-migrate: rolled back; external unlock is required"
	95	}
	96
	97	status() {
	98	findmnt "${VOLUME_MOUNT}" \|\| true
	99	ls -l "${IMAGE_PATH}" "${VOLUME_IMAGE}" "${ROOT_BACKUP}" 2>/dev/null \|\| true
	100	df -h "${VOLUME_MOUNT}" 2>/dev/null \|\| true
	101	}
	102
	103	case "${COMMAND}" in
	104	prepare) prepare ;;
	105	finalize) finalize ;;
	106	rollback) rollback ;;
	107	status) status ;;
	108	*) die "usage: $0 {prepare\|finalize\|rollback\|status} [device]" ;;
	109	esac

scripts/setup-encrypted-storage.shadded+256-0

Changes to scripts/setup-encrypted-storage.sh
		@@ -0,0 +1,256 @@
	1	#!/usr/bin/env bash
	2	set -euo pipefail
	3
	4	# Provision a LUKS2-backed filesystem for mail data and migrate an existing
	5	# /opt/mail installation without changing the paths used by the server.
	6	#
	7	# The random volume key is never left on persistent storage in plaintext.
	8	# systemd-creds wraps it with the host credential secret and exposes it only to
	9	# mail-storage.service through a private, read-only runtime credential.
	10
	11	MAIL_ROOT="${MAIL_ROOT:-/opt/mail}"
	12	MAIL_SERVICE="${MAIL_SERVICE:-mail}"
	13	IMAGE_PATH="${MAIL_ENCRYPTED_IMAGE:-/var/lib/mail-storage.luks}"
	14	IMAGE_SIZE="${MAIL_ENCRYPTED_IMAGE_SIZE:-2G}"
	15	MOUNT_PATH="${MAIL_ENCRYPTED_MOUNT:-/var/lib/mail-storage}"
	16	MAPPER_NAME="${MAIL_ENCRYPTED_MAPPER:-mail-storage}"
	17	CREDENTIAL_PATH="${MAIL_ENCRYPTED_CREDENTIAL:-/etc/credstore.encrypted/mail-storage-key.cred}"
	18	UNIT_PATH="/etc/systemd/system/mail-storage.service"
	19	DROPIN_PATH="/etc/systemd/system/${MAIL_SERVICE}.service.d/encrypted-storage.conf"
	20
	21	die() {
	22	echo "encrypted-storage: $*" >&2
	23	return 1
	24	}
	25
	26	[[ "${EUID}" -eq 0 ]] \|\| die "must run as root"
	27	[[ "${MAIL_ROOT}" == /* && "${MAIL_ROOT}" != "/" ]] \|\| die "MAIL_ROOT must be an absolute non-root path"
	28	[[ "${IMAGE_PATH}" == /* && "${IMAGE_PATH}" != "/" ]] \|\| die "MAIL_ENCRYPTED_IMAGE must be an absolute file path"
	29	[[ "${MOUNT_PATH}" == /* && "${MOUNT_PATH}" != "/" ]] \|\| die "MAIL_ENCRYPTED_MOUNT must be an absolute non-root path"
	30	[[ "${MAPPER_NAME}" =~ ^[A-Za-z0-9_.+-]+$ ]] \|\| die "invalid mapper name"
	31
	32	CRYPTSETUP="$(command -v cryptsetup \|\| true)"
	33	MKFS_EXT4="$(command -v mkfs.ext4 \|\| true)"
	34	SYSTEMD_CREDS="$(command -v systemd-creds \|\| true)"
	35	MOUNT_BIN="$(command -v mount \|\| true)"
	36	UMOUNT_BIN="$(command -v umount \|\| true)"
	37
	38	[[ -n "${CRYPTSETUP}" ]] \|\| die "cryptsetup is required"
	39	[[ -n "${MKFS_EXT4}" ]] \|\| die "mkfs.ext4 is required"
	40	[[ -n "${SYSTEMD_CREDS}" ]] \|\| die "systemd-creds is required"
	41	[[ -n "${MOUNT_BIN}" && -n "${UMOUNT_BIN}" ]] \|\| die "mount and umount are required"
	42
	43	install -d -m 0700 "$(dirname "${CREDENTIAL_PATH}")"
	44	install -d -m 0750 -o root -g mail-server "${MOUNT_PATH}"
	45	install -d -m 0755 "$(dirname "${IMAGE_PATH}")"
	46
	47	created_image=0
	48	temporary_key=""
	49	cleanup_key() {
	50	if [[ -n "${temporary_key}" && -f "${temporary_key}" ]]; then
	51	shred -u "${temporary_key}" 2>/dev/null \|\| rm -f "${temporary_key}"
	52	fi
	53	}
	54	trap cleanup_key EXIT
	55
	56	if [[ ! -f "${IMAGE_PATH}" ]]; then
	57	[[ ! -e "${CREDENTIAL_PATH}" ]] \|\| die "credential exists but encrypted image does not"
	58
	59	temporary_key="$(mktemp /run/mail-storage-key.XXXXXX)"
	60	chmod 0400 "${temporary_key}"
	61	dd if=/dev/urandom of="${temporary_key}" bs=64 count=1 status=none
	62
	63	echo "encrypted-storage: creating ${IMAGE_SIZE} LUKS2 image"
	64	fallocate -l "${IMAGE_SIZE}" "${IMAGE_PATH}"
	65	chmod 0600 "${IMAGE_PATH}"
	66	"${CRYPTSETUP}" luksFormat \
	67	--batch-mode \
	68	--type luks2 \
	69	--cipher aes-xts-plain64 \
	70	--key-size 512 \
	71	--hash sha512 \
	72	--pbkdf argon2id \
	73	--iter-time 5000 \
	74	--key-file "${temporary_key}" \
	75	"${IMAGE_PATH}"
	76
	77	"${CRYPTSETUP}" open --type luks2 --key-file "${temporary_key}" "${IMAGE_PATH}" "${MAPPER_NAME}"
	78	"${MKFS_EXT4}" -q -L mail-storage "/dev/mapper/${MAPPER_NAME}"
	79	"${CRYPTSETUP}" close "${MAPPER_NAME}"
	80
	81	"${SYSTEMD_CREDS}" encrypt \
	82	--name=mail-storage-key \
	83	"${temporary_key}" \
	84	"${CREDENTIAL_PATH}"
	85	chmod 0600 "${CREDENTIAL_PATH}"
	86	created_image=1
	87	elif [[ ! -f "${CREDENTIAL_PATH}" ]]; then
	88	die "encrypted image exists but ${CREDENTIAL_PATH} is missing"
	89	fi
	90
	91	cat > "${UNIT_PATH}" <<EOF
	92	[Unit]
	93	Description=Encrypted mail data filesystem
	94	After=local-fs.target
	95	Before=${MAIL_SERVICE}.service
	96
	97	[Service]
	98	Type=oneshot
	99	RemainAfterExit=yes
	100	LoadCredentialEncrypted=mail-storage-key:${CREDENTIAL_PATH}
	101	ExecStart=${CRYPTSETUP} open --type luks2 --key-file %d/mail-storage-key ${IMAGE_PATH} ${MAPPER_NAME}
	102	ExecStart=${MOUNT_BIN} -o noatime,nodev,nosuid /dev/mapper/${MAPPER_NAME} ${MOUNT_PATH}
	103	ExecStop=-${UMOUNT_BIN} ${MOUNT_PATH}
	104	ExecStop=-${CRYPTSETUP} close ${MAPPER_NAME}
	105	TimeoutStartSec=0
	106	TimeoutStopSec=120
	107
	108	[Install]
	109	WantedBy=multi-user.target
	110	EOF
	111
	112	install -d -m 0755 "$(dirname "${DROPIN_PATH}")"
	113	cat > "${DROPIN_PATH}" <<EOF
	114	[Unit]
	115	Requires=mail-storage.service
	116	After=mail-storage.service
	117	EOF
	118
	119	systemctl daemon-reload
	120	systemctl enable mail-storage.service >/dev/null
	121	if ! systemctl is-active --quiet mail-storage.service; then
	122	systemctl start mail-storage.service
	123	fi
	124	mountpoint -q "${MOUNT_PATH}" \|\| die "${MOUNT_PATH} is not mounted"
	125
	126	install -d -m 0750 -o mail-server -g mail-server "${MOUNT_PATH}/data"
	127
	128	was_active=0
	129	if systemctl is-active --quiet "${MAIL_SERVICE}.service"; then
	130	was_active=1
	131	systemctl stop "${MAIL_SERVICE}.service"
	132	fi
	133
	134	timestamp="$(date -u +%Y%m%dT%H%M%SZ)"
	135	plaintext_backup="${MAIL_ROOT}/.plaintext-mail-migration-${timestamp}"
	136	install -d -m 0700 "${plaintext_backup}"
	137	declare -a migrated=()
	138
	139	rollback() {
	140	local rel source backup
	141	echo "encrypted-storage: migration failed; restoring plaintext paths" >&2
	142	systemctl stop "${MAIL_SERVICE}.service" 2>/dev/null \|\| true
	143	for rel in "${migrated[@]}"; do
	144	source="${MAIL_ROOT}/${rel}"
	145	backup="${plaintext_backup}/${rel}"
	146	if [[ -L "${source}" && -e "${backup}" ]]; then
	147	unlink "${source}"
	148	install -d -m 0700 "$(dirname "${source}")"
	149	mv "${backup}" "${source}"
	150	fi
	151	done
	152	if [[ "${was_active}" -eq 1 ]]; then
	153	systemctl start "${MAIL_SERVICE}.service" 2>/dev/null \|\| true
	154	fi
	155	}
	156	trap rollback ERR
	157
	158	declare -a paths=(mail dkim backups cert-backups mail.log)
	159	while IFS= read -r candidate; do
	160	paths+=("$(basename "${candidate}")")
	161	done < <(
	162	find "${MAIL_ROOT}" -maxdepth 1 -type f \
	163	$ -name 'smtp.db' -o -name 'caldav.db' -o -name 'forwards.json' -o -name 'forwards.sieve' $ \
	164	-print \| sort
	165	)
	166
	167	for rel in "${paths[@]}"; do
	168	source="${MAIL_ROOT}/${rel}"
	169	destination="${MOUNT_PATH}/data/${rel}"
	170	backup="${plaintext_backup}/${rel}"
	171
	172	if [[ -L "${source}" ]]; then
	173	[[ "$(readlink "${source}")" == "${destination}" ]] \
	174	\|\| die "${source} points somewhere other than encrypted storage"
	175	continue
	176	fi
	177	[[ -e "${source}" ]] \|\| continue
	178	[[ ! -e "${destination}" ]] \|\| die "${destination} already exists while ${source} is plaintext"
	179
	180	install -d -m 0750 "$(dirname "${destination}")"
	181	cp -a "${source}" "${destination}.new"
	182
	183	if [[ -d "${source}" ]]; then
	184	diff -qr "${source}" "${destination}.new" >/dev/null
	185	else
	186	cmp -s "${source}" "${destination}.new"
	187	fi
	188
	189	mv "${source}" "${backup}"
	190	mv "${destination}.new" "${destination}"
	191	ln -s "${destination}" "${source}"
	192	migrated+=("${rel}")
	193	done
	194
	195	if [[ -d "${MOUNT_PATH}/data/mail" ]]; then
	196	chown -R mail-server:mail-server "${MOUNT_PATH}/data/mail"
	197	find "${MOUNT_PATH}/data/mail" -type d -exec chmod 0700 {} +
	198	find "${MOUNT_PATH}/data/mail" -type f -exec chmod 0600 {} +
	199	fi
	200
	201	for db in "${MOUNT_PATH}"/data/smtp.db* "${MOUNT_PATH}"/data/caldav.db*; do
	202	[[ -e "${db}" ]] \|\| continue
	203	chown mail-server:mail-server "${db}"
	204	chmod 0600 "${db}"
	205	done
	206
	207	if [[ -d "${MOUNT_PATH}/data/dkim" ]]; then
	208	chown -R mail-server:mail-server "${MOUNT_PATH}/data/dkim"
	209	find "${MOUNT_PATH}/data/dkim" -type d -exec chmod 0700 {} +
	210	find "${MOUNT_PATH}/data/dkim" -type f -exec chmod 0600 {} +
	211	fi
	212
	213	for forwards in "${MOUNT_PATH}"/data/forwards.json* "${MOUNT_PATH}"/data/forwards.sieve*; do
	214	[[ -e "${forwards}" ]] \|\| continue
	215	chown mail-server:mail-server "${forwards}"
	216	chmod 0600 "${forwards}"
	217	done
	218
	219	if [[ -d "${MOUNT_PATH}/data/backups" ]]; then
	220	chown -R root:root "${MOUNT_PATH}/data/backups"
	221	find "${MOUNT_PATH}/data/backups" -type d -exec chmod 0700 {} +
	222	find "${MOUNT_PATH}/data/backups" -type f -exec chmod 0600 {} +
	223	fi
	224
	225	if [[ -f /etc/mail/mail.env ]]; then
	226	chown root:root /etc/mail/mail.env
	227	chmod 0600 /etc/mail/mail.env
	228	fi
	229
	230	sync
	231
	232	if systemctl list-unit-files "${MAIL_SERVICE}.service" --no-legend 2>/dev/null \| grep -q "${MAIL_SERVICE}.service"; then
	233	systemctl start "${MAIL_SERVICE}.service"
	234	sleep 2
	235	systemctl is-active --quiet "${MAIL_SERVICE}.service" \
	236	\|\| die "${MAIL_SERVICE}.service did not restart after migration"
	237	fi
	238
	239	# The encrypted copy has been byte-checked and the service restarted from it.
	240	# Remove the temporary plaintext rollback copy so message data is not left
	241	# exposed beside the LUKS image.
	242	find "${plaintext_backup}" -depth -delete
	243	trap - ERR
	244
	245	echo "encrypted-storage: ready"
	246	echo "encrypted-storage: image=${IMAGE_PATH}"
	247	echo "encrypted-storage: mount=${MOUNT_PATH}"
	248	echo "encrypted-storage: credential=${CREDENTIAL_PATH}"
	249	if [[ "${created_image}" -eq 1 ]]; then
	250	echo "encrypted-storage: generated a new machine-bound volume key"
	251	fi
	252
	253	script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
	254	if [[ -f "${script_dir}/mail-storage-external.sh" ]]; then
	255	install -m 0700 "${script_dir}/mail-storage-external.sh" /usr/local/sbin/mail-storage-external
	256	fi

scripts/setup-restic-backup.shadded+93-0

Changes to scripts/setup-restic-backup.sh
		@@ -0,0 +1,93 @@
	1	#!/usr/bin/env bash
	2	set -euo pipefail
	3
	4	SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
	5	RESTIC_CONFIG_DIR="${MAIL_RESTIC_CONFIG_DIR:-/var/lib/mail-storage/data/restic}"
	6
	7	[[ "${EUID}" -eq 0 ]] \|\| {
	8	echo "setup-restic-backup: must run as root" >&2
	9	exit 1
	10	}
	11	mountpoint -q /var/lib/mail-storage \|\| {
	12	echo "setup-restic-backup: encrypted mail storage is not mounted" >&2
	13	exit 1
	14	}
	15
	16	if ! command -v restic >/dev/null 2>&1; then
	17	if command -v apt-get >/dev/null 2>&1; then
	18	export DEBIAN_FRONTEND=noninteractive
	19	apt-get update -qq
	20	apt-get install -y -qq restic
	21	elif command -v dnf >/dev/null 2>&1; then
	22	dnf install -y restic
	23	else
	24	echo "setup-restic-backup: unsupported package manager" >&2
	25	exit 1
	26	fi
	27	fi
	28
	29	install -d -m 0700 -o root -g root "${RESTIC_CONFIG_DIR}"
	30	install -m 0755 "${SCRIPT_DIR}/mail-restic-backup.sh" /usr/local/sbin/mail-restic-backup
	31	install -m 0755 "${SCRIPT_DIR}/mail-restic-restore-check.sh" /usr/local/sbin/mail-restic-restore-check
	32	install -m 0755 "${SCRIPT_DIR}/mail-storage-health.sh" /usr/local/sbin/mail-storage-health
	33	install -m 0755 "${SCRIPT_DIR}/maildir-migrate.sh" /usr/local/sbin/maildir-migrate
	34	install -m 0755 "${SCRIPT_DIR}/mail-forward-compile" /usr/local/sbin/mail-forward-compile
	35
	36	cat > /etc/systemd/system/mail-restic-backup.service <<'UNIT'
	37	[Unit]
	38	Description=Authenticated off-host mail backup
	39	After=network-online.target mail-storage.service
	40	Wants=network-online.target
	41	ConditionPathIsMountPoint=/var/lib/mail-storage
	42
	43	[Service]
	44	Type=oneshot
	45	ExecStart=/usr/local/sbin/mail-restic-backup
	46	Nice=10
	47	IOSchedulingClass=best-effort
	48	IOSchedulingPriority=7
	49	PrivateTmp=true
	50	ProtectSystem=strict
	51	ReadWritePaths=/var/lib/mail-storage /opt/mail
	52	UNIT
	53
	54	cat > /etc/systemd/system/mail-restic-backup.timer <<'UNIT'
	55	[Unit]
	56	Description=Daily authenticated off-host mail backup
	57
	58	[Timer]
	59	OnCalendar=--* 03:20:00 UTC
	60	Persistent=true
	61	RandomizedDelaySec=15m
	62
	63	[Install]
	64	WantedBy=timers.target
	65	UNIT
	66
	67	cat > /etc/systemd/system/mail-storage-health.service <<'UNIT'
	68	[Unit]
	69	Description=Mail storage capacity and backup freshness check
	70	After=mail-storage.service
	71	ConditionPathIsMountPoint=/var/lib/mail-storage
	72
	73	[Service]
	74	Type=oneshot
	75	ExecStart=/usr/local/sbin/mail-storage-health
	76	UNIT
	77
	78	cat > /etc/systemd/system/mail-storage-health.timer <<'UNIT'
	79	[Unit]
	80	Description=Hourly mail storage health check
	81
	82	[Timer]
	83	OnBootSec=10m
	84	OnUnitActiveSec=1h
	85	Persistent=true
	86
	87	[Install]
	88	WantedBy=timers.target
	89	UNIT
	90
	91	systemctl daemon-reload
	92	systemctl enable mail-restic-backup.timer mail-storage-health.timer >/dev/null
	93	echo "setup-restic-backup: installed"