Summary
Between 29 October and 1 November 2024, forty-one enterprise NVMe drives across four sites stopped accepting commands at a fixed power-on-hour count, a firmware defect in one manufacturing batch. Mirroring absorbed thirty-nine of those failures without customer impact. One node in Warsaw lost both halves of a mirror inside forty minutes, because both drives came from the same batch and were racked on the same day. Two instances on that node lost data. That second part was our fault, not the vendor’s.
Timeline
All times UTC.
| Time | Event |
|---|---|
| 29 Oct 03:11 | Drive drops off the bus on a Warsaw node. Mirror degrades, hot spare begins rebuilding. Routine, no page. |
| 29 Oct 03:49 | Second drive in the same mirror drops. Node loses its root pool and stops. |
| 29 Oct 03:52 | Page fires. On-call online at 03:55. |
| 29 Oct 04:20 | Node declared unrecoverable in place. Rebuild from the hot spare is impossible; the spare was mid-rebuild from a source that no longer answers. |
| 29 Oct 05:40 | Engineer reviewing drive logs notices the two failures are eighteen power-on hours apart, not eighteen months. Suspicion moves from bad luck to a cohort. |
| 29 Oct 06:15 | Fleet-wide query by batch identifier and power-on hours. Two hundred and six drives are in the affected batch. Thirty-one have already passed the count and failed; the rest are between forty and nine hundred hours away from it. |
| 29 Oct 07:30 | Vendor contacted. Defect confirmed within four hours: a counter in the wear-levelling telemetry overflows at 1,536 power-on hours and wedges the controller. A firmware revision fixing it had shipped, quietly, six weeks earlier. |
| 29 Oct 09:00 | Rolling firmware update begins, prioritised by hours remaining. |
| 30 Oct 22:40 | Last drive in the batch updated or replaced. |
| 1 Nov 14:00 | Affected customer instances restored or refunded. |
Root cause
Two causes, and only one of them belongs to the drive manufacturer.
Theirs: a telemetry counter overflowed at a fixed power-on-hour count and hung the controller. The drive survives a power cycle, comes back, and wedges again within minutes. Data on the platter is intact and unreachable, which is the worst combination for anyone trying to diagnose it at four in the morning.
Ours: we built mirrors from drives that arrived in the same delivery. A mirror is meant to be two independent failure domains, and two drives from one batch, racked the same afternoon, powered on within the same hour, are not independent in any sense that matters. Eleven mirrors across the fleet were built this way. Ten got lucky in that the hot spare finished rebuilding first. One did not.
We had known this in principle for years. Nobody had written it into the build procedure, and the build procedure is what people follow at two in the morning under a delivery deadline.
Impact
- Thirty-nine failures absorbed by mirrors with no customer-visible effect.
- One node offline for nine hours and eighteen minutes.
- Fourteen instances on that node restored from off-node backups, losing at most eleven minutes of writes.
- Two instances with no backup of any kind. Both lost everything on the node.
What we changed
- Purchase batches are split. No two drives from the same batch may form a mirror, and the hot spare protecting a pair comes from a third batch. Enforced by the build tooling, not by a document. Done 4 November.
- Cohort alerting on power-on hours. We now alert when more than four drives sharing a batch identifier are within a hundred hours of each other and approaching any round number of hours. It is a crude heuristic and it would have caught this one nineteen days early.
- Firmware soak before production. A new drive family runs two thousand power-on hours in a test rack before it carries customer data. This defect would have surfaced at 1,536.
- We now read vendor firmware release notes on a schedule. The fix existed six weeks before we needed it. Nobody was assigned to look, so nobody looked.
What we did not change, and why
We did not change drive family or vendor. The defect was real and the disclosure was poor, but our loss came from correlated batching, which we would have reproduced with any manufacturer. Switching would have felt decisive and fixed nothing.
Off-node backup stays an add-on at nine euro per five hundred gigabytes. Making it universal means charging every customer for a service most of them replicate themselves, and we are not going to bill people for the appearance of safety. What did change is the order form, which now states plainly that instance storage is mirrored and mirrored is not a backup, and the confirmation step no longer lets that sentence be skipped past silently.
We did not move to triple mirroring. It costs a third more per gigabyte and does not address correlated failure, which was the actual mechanism here. Two independent domains beat three dependent ones.
The two customers
Both were refunded in full for the affected term, in the asset they paid with, without being asked to justify anything. One left. The other stayed and now buys the backup add-on, having read the same sentence on the order form that had been there in a weaker form all along.
Neither of those outcomes was in our gift. The only thing that was in our gift was building the mirror properly, and we did not.