Find packet loss with MTR, and read it correctly

Install it

apt install -y mtr-tiny     # Debian
dnf install -y mtr          # AlmaLinux

Run it properly

mtr -rwbc 200 <destination>

• -r report mode, so it exits with something you can paste
• -w wide output, so long hostnames survive
• -b show the name and the address for each hop
• -c 200 two hundred cycles, about three and a half minutes, and the difference between evidence and an anecdote

Ten packets prove nothing at all. If a ten-cycle run shows loss, run two hundred before you conclude anything from it.

Read the last hop first

Routers de-prioritise packets addressed to themselves. A transit router carrying hundreds of gigabits will answer your expired-TTL probes when it has a spare cycle and drop them when it does not, which paints loss across the middle of your report while every packet you actually care about arrives intact.

Two rules cover most reports:

1. Loss at hop seven that is absent at hop eight and beyond is a rate limit. Ignore it.
2. Loss that starts at hop seven and continues all the way to the destination is real, and the hop where it starts is where to look.

Latency behaves the same way. A hop that adds eighty milliseconds and then hands traffic to the next hop at the previous figure is answering slowly, not forwarding slowly.

ICMP is not your traffic

If the service runs on TCP, test on TCP:

mtr -rwbc 200 --tcp --port 443 <destination>

Networks that rate-limit ICMP hard often pass TCP untouched. The reverse case is worth discovering before you blame anybody’s network: a firewall that answers ping happily and drops the port your users actually reach for.

Both directions, every time

Routing is asymmetric more often than not. The path from you to us is not the path from us to you, and a report from one end describes exactly one of them. Run MTR from the instance towards the client and from the client towards the instance, then attach both. Our half of the path can also be probed without a shell anywhere, from the looking glass.

Corroborate it in the TCP counters

MTR describes the path. The socket statistics describe the consequence:

ss -ti state established
nstat -az TcpRetransSegs TcpExtTCPLostRetransmit

A retransmission rate above a percent or so, on a connection that should be clean, supports the report. Zero retransmissions alongside an MTR full of loss means you are looking at a rate limit and nothing else.

What to send us

• Both MTR reports, two hundred cycles, as text rather than as a screenshot
• Destination, transport and port
• Timestamps with a timezone, and whether the loss is constant or in bursts
• The instance ID

Roughly half of these resolve to a rate-limited middle hop, and we say so in the first reply. The rest we can usually route around within the hour: loss inside our own network is our problem, and loss in one transit path is a matter of moving your traffic to a different one.