Moving two terabytes between two of our sites without downtime

What this builds

Two terabytes of files and a live Postgres database moved from one site to another, with the application answering requests throughout. The method is a bulk pre-copy while the source keeps serving, a streaming replica that stays in step, and a cutover measured in seconds with the old host proxying to the new one until DNS catches up.

Distance is the interesting part. Between AMS-01 and FRA-01 the round trip is seven milliseconds and a single stream will fill a port. Run that same copy out to GRU-01, where the round trip is close to two hundred, and an untuned transfer crawls along at a small fraction of what you are paying for while both machines sit almost idle. That gap is entirely down to how much data one connection is allowed to have in flight.

Before you start

• Source and target instances, a tunnel between them, and root on both.
• DNS you control, with the record’s TTL dropped to sixty seconds at least a day beforehand. This is the step people skip and then wait six hours for.
• Enough disk on the target for the whole dataset plus the replica.

1. Work out what the link can do

The bandwidth-delay product decides everything. Multiply the round-trip time by the rate you want and you get the amount of data that must be unacknowledged in flight at any moment:

ping -c20 target.example.com | tail -1

On both machines:

cat > /etc/sysctl.d/75-transfer.conf <<EOF
net.core.rmem_max = 134217728
net.core.wmem_max = 134217728
net.ipv4.tcp_rmem = 4096 262144 134217728
net.ipv4.tcp_wmem = 4096 262144 134217728
net.ipv4.tcp_congestion_control = bbr
net.core.default_qdisc = fq
net.ipv4.tcp_mtu_probing = 1
EOF
sysctl --system

BBR rather than the default congestion control matters over long paths with any loss at all, because loss-based algorithms interpret a single dropped packet as congestion and halve their window. Over two hundred milliseconds, recovering from that takes seconds, and it happens repeatedly.

Measure before you commit two terabytes to an assumption:

# target
apt install -y iperf3 && iperf3 -s
# source
iperf3 -c target.example.com -P 8 -t 30

Eight parallel streams should approach the port speed. If one stream manages a hundred megabits and eight manage eight hundred, the window settings did not take effect; a single stream should get most of the way there on its own once they have.

2. Bulk pre-copy

Run this against a live source. It will take hours, and nothing is interrupted while it does:

apt install -y rsync
rsync -aHAX --numeric-ids --partial --inplace --info=progress2 \
  -e "ssh -c [email protected] -o Compression=no" \
  /srv/data/ [email protected]:/srv/data/

The cipher choice is not superstition. On these processors the hardware-accelerated GCM mode moves several gigabytes a second per core, while the default negotiated cipher on some client versions is markedly slower and turns your transfer into a single-core benchmark. Compression is off because the data is already compressed and, if it is not, the CPU spends its time on that instead of on moving bytes.

For millions of small files, one stream is the wrong shape. Split by top-level directory and run several:

ls /srv/data | xargs -P 8 -I{} rsync -aHAX --numeric-ids \
  -e "ssh -c [email protected] -o Compression=no" \
  /srv/data/{}/ [email protected]:/srv/data/{}/

Eight parallel copies on eight dedicated cores is the right number here; sixteen is not twice as fast and will make the source machine unpleasant for whatever it is still serving.

3. The database, replicated rather than copied

Dumping and restoring two hundred gigabytes of Postgres means a long window where writes are lost. Stream it instead. On the source:

sudo -u postgres psql -c "CREATE USER repl WITH REPLICATION PASSWORD '<a long password>';"
sudo -u postgres psql -c "SELECT pg_create_physical_replication_slot('target_site');"
echo "host replication repl 10.9.0.2/32 scram-sha-256" >> /etc/postgresql/17/main/pg_hba.conf
systemctl reload postgresql

On the target:

systemctl stop postgresql
rm -rf /var/lib/postgresql/17/main/*
sudo -u postgres pg_basebackup -h 10.9.0.1 -U repl -D /var/lib/postgresql/17/main \
  -S target_site -X stream -R -P -c fast
systemctl start postgresql

The -R flag writes the connection settings and the standby signal file, so the target comes up as a replica and begins following. From here it stays within a second or two of the source indefinitely, which means the cutover no longer involves moving the database at all.

4. Cutover

Start a probe from a third machine and leave it running. It is both your verification and your record of what the switch cost:

while true; do curl -s -o /dev/null -w "%{http_code} %{time_total}\n" https://app.example.com/health; sleep 1; done | tee cutover.log

Then, in this order and without pauses:

# 1. final delta, source still live
rsync -aHAX --numeric-ids --delete -e "ssh -c [email protected]" /srv/data/ [email protected]:/srv/data/

# 2. stop writes on the source
systemctl stop app

# 3. one more delta, now tiny
rsync -aHAX --numeric-ids --delete -e "ssh -c [email protected]" /srv/data/ [email protected]:/srv/data/

# 4. promote the replica
ssh [email protected] "sudo -u postgres pg_ctl promote -D /var/lib/postgresql/17/main"

# 5. start the application on the target
ssh [email protected] "systemctl start app"

Steps two through five take a few seconds. Now stop the old host from serving stale content while DNS propagates, by turning it into a proxy for the new one:

server {
  listen 443 ssl;
  server_name app.example.com;
  ssl_certificate     /etc/letsencrypt/live/app.example.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/app.example.com/privkey.pem;
  location / {
    proxy_pass https://10.9.0.2;
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $remote_addr;
  }
}

This is the piece that makes the whole thing non-disruptive. Anyone still resolving the old address gets a correct answer over the tunnel rather than an error or, worse, a copy of the data that stopped updating at cutover. Now change the DNS record.

Verify it

Start with the probe log you have been collecting:

sort cutover.log | uniq -c | sort -rn | head

Every line should read 200. A handful of slower responses during the switch is expected; any non-200 means step five started after step two by too wide a margin, and the proxy in the previous section is what shrinks that gap to nothing.

Then prove the data is identical rather than merely present:

rsync -aHAXn --delete --itemize-changes /srv/data/ [email protected]:/srv/data/ | head
du -s --bytes /srv/data
ssh [email protected] "du -s --bytes /srv/data"

A dry run that itemises nothing means the two trees agree on content, permissions, ownership and extended attributes. Byte totals should match exactly.

Database next:

ssh [email protected] "sudo -u postgres psql -c 'SELECT pg_is_in_recovery();'"
ssh [email protected] "sudo -u postgres psql app -c 'SELECT count(*) FROM orders;'"
sudo -u postgres psql app -c "SELECT count(*) FROM orders;"

Recovery should now be false on the target, and the row counts should agree. Finally, confirm traffic has genuinely moved rather than being quietly proxied forever:

dig +short app.example.com
tail -f /var/log/nginx/access.log | wc -l   # on the old host, an hour later

When the old host has been idle for a full TTL plus a comfortable margin, take the proxy down and destroy the instance. Do not destroy it the same afternoon; the cheapest insurance in this entire procedure is leaving the source machine intact and switched off for a week.

Afterwards

Two terabytes over a well-tuned long-haul path is roughly an hour at ten gigabits and rather longer in practice, since the source is also serving. Between neighbouring European sites, plan for an afternoon; between continents, start it in the evening and do the cutover the next day. The locations page lists the round-trip times we measure, and those are the numbers to put into the table in step one.