For five years, buying a server here took a median of three minutes and twenty seconds from the settlement webhook to credentials in your inbox. Nobody complained. It was faster than most of the industry and considerably faster than anything requiring a human to approve you.
It was also four serial steps that had no business being serial, and once you notice that you cannot stop noticing it. The rewrite shipped in March and the median is now forty-seven seconds, with the ninety-fifth percentile just under four minutes.
What the old one did
| Step | Median | What was happening |
|---|---|---|
| Image copy | 84 s | Pulling a disk image from a per-site store onto the target node |
| Volume create | 31 s | Allocating and formatting the NVMe volume |
| Address allocation | 46 s | Taking a lock on the site address pool, picking an IPv4, writing reverse DNS |
| First boot and cloud-init | 39 s | Generating keys, expanding the filesystem, sending mail |
Four steps, one global worker pool, and a lock on the address pool that every order in the same site had to queue behind. On a quiet Tuesday it was fine. During a promotion, or during the hour after a large customer scripted forty creates, the median doubled and the tail went past twelve minutes.
What we changed
Images are pre-seeded, not copied. Every node holds a thin clone base for every image in the catalogue, refreshed nightly. Creating a volume is now a copy-on-write clone against a local Gen4 NVMe mirror rather than a network pull. That step went from eighty-four seconds to under two.
Addresses are reserved in advance. Each site keeps a warm pool of allocated addresses with reverse DNS already written, sized to about six hours of demand at that site. Allocation is now a pop from a pre-built pool instead of a lock, a scan and a DNS write. Forty-six seconds became about four hundred milliseconds.
Queues are per-site. A rush in Frankfurt no longer slows a create in São Paulo, which sounds obvious and was not the original design because the original design had four sites and one engineer.
Workers are idempotent and resumable. Every step is keyed, so a worker dying halfway leaves a resumable job rather than a half-built instance and a support ticket. Manual intervention on a failed create fell from about one order in four hundred to one in nine thousand.
The result
| Metric | Before | After |
|---|---|---|
| Median | 3 min 20 s | 47 s |
| 95th percentile | 12 min 10 s | 3 min 56 s |
| 99th percentile | 41 min | 8 min 20 s |
| Failed creates needing a human | 1 in 400 | 1 in 9,000 |
| Concurrency per site before the median moves | 6 | 90 |
What went wrong on the way
In March, for eleven hours, the nightly refresh of the thin clone bases failed silently at four sites and the pre-seeded Debian image served a point release that was nine days stale. About ninety instances were built from it. None of them were broken in any interesting way, since a stale point release is a package update away from a current one, but nobody buying a server should have to check.
We rebuilt the affected instances on request, emailed all ninety whether they asked or not, and added a check that compares the base image hash against the catalogue before any node is allowed to serve creates. A silent failure in a nightly job is the most boring possible cause and it is worth writing down precisely because it is boring.
What is still slow
- Custom ISO installs. Still manual, still measured in tens of minutes, usually completed within the hour. The bottleneck is a person confirming that the image does what its uploader says it does.
- Windows. Licence activation adds two to three minutes and is not under our control.
- Bare metal. Same-day rather than same-minute. A physical machine has a physical rebuild, and we would rather quote you honestly than start a clock we cannot beat.
- IPv6 /48 delegation. Manual at three sites where the fabric configuration is older. Being fixed, slowly.
Why we stopped here
We could get the median to about twenty seconds by keeping instances pre-booted and simply handing you one on payment. That means holding idle capacity, which means paying for cores nobody is using, which means charging everyone a little more so that new orders can feel eleven seconds faster.
Forty-seven seconds is short enough that the constraint is now the chain confirming your payment rather than anything we do. Optimising past the point where the customer notices is a hobby, not engineering.