A self-hosted mail server that actually reaches the inbox

What this builds

A complete single-domain mail server: Postfix for transport, Dovecot for IMAP and local delivery, rspamd for filtering and DKIM signing, and the three DNS records that decide whether large receivers put your mail in the inbox or in a folder nobody opens. It sends, it receives, it authenticates, and it is not an open relay.

Running your own mail is unfashionable advice and mostly for good reason. Deliverability is the hard part, not the software. Get the DNS wrong and the most carefully configured Postfix in the world posts messages straight into the void.

Before you start

• An R-8 or larger. Filtering is CPU-bound and rspamd will happily use every core you give it during a spam wave.
• A domain whose DNS you control, and patience for propagation.
• Reverse DNS set on the instance address to your mail hostname. It is editable from the panel and it is mandatory: receivers reject unresolvable senders.
• One ticket to support asking whether outbound port 25 is open on your instance. Ask before you build, not after.
• mail.example.com throughout. Substitute your own.

1. Hostname and DNS

hostnamectl set-hostname mail.example.com
apt update && apt full-upgrade -y

Publish these records now, because the certificate step needs the A record and the signing step needs somewhere to put a key. Values in the table are what each record does, not what to paste.

The SPF record says: my MX hosts send my mail, everything else is forged. Hard fail rather than soft fail, because ~all invites receivers to make their own decision and their decision will not be the generous one. DMARC starts at p=none so that you collect reports for a week before anything gets rejected on your behalf.

2. Certificates

apt install -y certbot
certbot certonly --standalone -d mail.example.com

Nothing is listening on port 80 yet, so standalone mode is the simplest path. Renewal hooks come at the end, once the daemons exist to reload.

3. Postfix

DEBIAN_FRONTEND=noninteractive apt install -y postfix postfix-pcre dovecot-imapd dovecot-lmtpd rspamd redis-server

Now set the transport configuration. Every line below is deliberate; postconf writes them in place so there is no editor to get wrong:

postconf -e "myhostname = mail.example.com"
postconf -e "mydestination = localhost"
postconf -e "smtpd_banner = \$myhostname ESMTP"
postconf -e "inet_interfaces = all"
postconf -e "smtpd_tls_cert_file = /etc/letsencrypt/live/mail.example.com/fullchain.pem"
postconf -e "smtpd_tls_key_file = /etc/letsencrypt/live/mail.example.com/privkey.pem"
postconf -e "smtpd_tls_security_level = may"
postconf -e "smtp_tls_security_level = dane"
postconf -e "smtpd_tls_mandatory_protocols = >=TLSv1.2"
postconf -e "smtpd_sasl_type = dovecot"
postconf -e "smtpd_sasl_path = private/auth"
postconf -e "smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated reject_unauth_destination"
postconf -e "smtpd_recipient_restrictions = permit_mynetworks permit_sasl_authenticated reject_unknown_recipient_domain reject_unauth_destination"
postconf -e "virtual_transport = lmtp:unix:private/dovecot-lmtp"
postconf -e "virtual_mailbox_domains = example.com"
postconf -e "smtpd_milters = inet:127.0.0.1:11332"
postconf -e "non_smtpd_milters = inet:127.0.0.1:11332"
postconf -e "milter_default_action = accept"

The single most important line is smtpd_relay_restrictions. It ends with reject_unauth_destination, which is what stops the entire internet using your machine to send its mail. An open relay gets you delisted within hours and is a breach of the acceptable use policy besides.

Enable the submission port so your own clients can send, with authentication required and no anonymous access:

postconf -M submission/inet="submission inet n - y - - smtpd"
postconf -P "submission/inet/syslog_name=postfix/submission"
postconf -P "submission/inet/smtpd_tls_security_level=encrypt"
postconf -P "submission/inet/smtpd_sasl_auth_enable=yes"
postconf -P "submission/inet/smtpd_client_restrictions=permit_sasl_authenticated,reject"

4. Dovecot

Mailboxes in Maildir under a single system user keeps this build small. Create the user and the storage:

groupadd -g 5000 vmail
useradd -g vmail -u 5000 -d /var/vmail -m vmail
install -d -o vmail -g vmail -m 700 /var/vmail/example.com

Write /etc/dovecot/conf.d/99-local.conf:

mail_location = maildir:/var/vmail/%d/%n
mail_uid = vmail
mail_gid = vmail
ssl = required
ssl_cert = </etc/letsencrypt/live/mail.example.com/fullchain.pem
ssl_key = </etc/letsencrypt/live/mail.example.com/privkey.pem
ssl_min_protocol = TLSv1.2
auth_mechanisms = plain login
disable_plaintext_auth = yes

passdb {
  driver = passwd-file
  args = scheme=ARGON2ID username_format=%u /etc/dovecot/users
}
userdb {
  driver = static
  args = uid=vmail gid=vmail home=/var/vmail/%d/%n
}

service lmtp {
  unix_listener /var/spool/postfix/private/dovecot-lmtp {
    mode = 0600
    user = postfix
    group = postfix
  }
}

service auth {
  unix_listener /var/spool/postfix/private/auth {
    mode = 0660
    user = postfix
    group = postfix
  }
}

Create a mailbox. The command prompts twice and writes a hash, never the password itself:

echo -n "[email protected]:" >> /etc/dovecot/users
doveadm pw -s ARGON2ID >> /etc/dovecot/users
chmod 640 /etc/dovecot/users
chgrp dovecot /etc/dovecot/users

5. rspamd and DKIM

Generate the signing key, then publish the public half:

install -d -o _rspamd -g _rspamd -m 750 /var/lib/rspamd/dkim
rspamadm dkim_keygen -s mail -b 2048 -d example.com -k /var/lib/rspamd/dkim/example.com.mail.key
chown _rspamd:_rspamd /var/lib/rspamd/dkim/example.com.mail.key
chmod 400 /var/lib/rspamd/dkim/example.com.mail.key

That command prints a DNS record on standard output. Publish it as a TXT record at mail._domainkey.example.com, exactly as printed, with the quoted segments joined the way your DNS provider expects. Then tell rspamd where the key lives, in /etc/rspamd/local.d/dkim_signing.conf:

path = "/var/lib/rspamd/dkim/$domain.$selector.key";
selector = "mail";
allow_username_mismatch = true;
use_domain = "header";
sign_authenticated = true;
sign_local = true;

Bind the milter and set a password for the web interface:

echo "bind_socket = \"127.0.0.1:11332\";" > /etc/rspamd/local.d/worker-proxy.inc
rspamadm pw

Paste the resulting hash into /etc/rspamd/local.d/worker-controller.inc as password = "...". Reach the interface over an SSH tunnel rather than opening port 11334 to the world.

6. Start everything

systemctl enable --now redis-server rspamd dovecot postfix
postfix check
systemctl --no-pager --failed

Add a renewal hook so a new certificate reaches both daemons:

echo "systemctl reload postfix dovecot" > /etc/letsencrypt/renewal-hooks/deploy/reload-mail.sh
chmod +x /etc/letsencrypt/renewal-hooks/deploy/reload-mail.sh

Verify it

Three tests. The first is local, the second is the one that matters, the third is the one that keeps you off blocklists.

echo | openssl s_client -starttls smtp -connect mail.example.com:25 -servername mail.example.com 2>/dev/null | grep -E "subject=|Verify return"

Then send a message from your new server to a mailbox you hold at a large webmail provider, open the received message and read its raw headers. You are looking for one line:

Authentication-Results: mx.receiver.example;
  dkim=pass [email protected];
  spf=pass smtp.mailfrom=example.com;
  dmarc=pass (p=NONE) header.from=example.com

Three passes and you are done. A dkim=none means rspamd is not signing, so check rspamadm configdump dkim_signing. An spf=fail almost always means the message left over IPv6 while your SPF record only covers the v4 MX address.

Last, prove you are not a relay. From any other machine, connect and try to send mail between two domains you do not own:

apt install -y swaks
swaks --to [email protected] --from [email protected] --server mail.example.com

The correct outcome is a rejection at the recipient stage: 554 5.7.1 Relay access denied. Anything else means you have configured a machine that will be abused within the day, and you should stop and re-read step three.

Afterwards

Leave DMARC at p=none for a week and read the aggregate reports. Once they show only your own server signing your own mail, move to p=quarantine, then p=reject a fortnight later. Add off-node backup for /var/vmail while you are at it, because mail is the one thing users never have a second copy of, and our backup add-on writes to a different site than the instance.