Summary
On 19 January 2026, between 14:02 and 14:28 UTC, roughly two thirds of all attempts to log into the panel or authenticate against the API failed. Existing sessions were also invalidated at random. Running instances, their networking and their traffic were untouched throughout. Provisioning paused for twenty-six minutes and then drained; no order was lost.
Timeline
All times UTC, 19 January 2026.
| Time | Event |
|---|---|
| 14:02 | Session token schema change applied to the control plane. Rollout marked as configuration, so it went to all six panel nodes at once. |
| 14:03 | Login error rate rises from near zero to sixty-one percent. Automated alerting has a two-minute window and does not fire yet. |
| 14:06 | Alert fires. On-call engineer paged. |
| 14:08 | First responder online. Sees an error rate, no deploy in the code-change log, and starts looking at the database. |
| 14:14 | Second engineer joins, checks the configuration change log rather than the code log, and finds the 14:02 entry. |
| 14:16 | Cause understood: two nodes are validating tokens with the previous reader. |
| 14:19 | Rollback begins. |
| 14:24 | Tokens issue and validate correctly on all six nodes. Error rate falls to zero. |
| 14:28 | Queued provisioning jobs drain. Recovery complete. |
| 15:10 | Status page updated. Forty-two minutes late, which is its own failure and is addressed below. |
Root cause
The session token gained a field. Writers emitted the new format immediately; the reader on two of six panel nodes had not been restarted and rejected anything carrying the new field. Requests are distributed across all six nodes, so a session minted on a new node had roughly a one-in-three chance of being validated by an old one on any given request. The result looked intermittent, which is why the first six minutes of diagnosis went into the database rather than into the deploy log.
The deeper cause is a rule we wrote in 2023. Changes that touch a schema file rather than application code were classified as configuration, and configuration skipped the canary stage on the grounds that it was, quote, just values. That rule was reasonable when the only schema files were feature flags. Nobody revisited it as the definition of a schema file expanded, and this change was correctly classified under a rule that had quietly become wrong.
It was not a mistake by the engineer who applied it. The classification was followed exactly as written.
Blast radius
- Panel logins: sixty-one percent failure rate for twenty-six minutes.
- API tokens: same failure rate, same window. Idempotency keys meant retried creates did not duplicate.
- Running instances: unaffected. No packet loss, no reboots, no storage impact.
- Provisioning: paused, not failed. Four orders settled during the window and all four completed by 14:28.
What we changed
- The canary stage is now unconditional. Anything that deploys, of any classification, goes to one node for five minutes with synthetic traffic before it goes anywhere else. Done 21 January.
- Readers accept the previous format for thirty days. Token validation is now explicitly versioned with an overlap window, so a partially rolled-out change degrades to nothing at all. Done 23 January.
- A synthetic login runs every fifteen seconds from outside our network, in three regions, and pages on the second consecutive failure rather than after a two-minute averaging window. Done 22 January.
- The status page publishes automatically when the synthetic login fails twice, without waiting for a human to write a sentence. A human sentence follows it. Done 26 January.
What we did not change, and why
We still do not log client IPs or request bodies for panel traffic. Having them would have shown the one-in-three distribution across nodes immediately and probably saved nine minutes. Nine minutes is not worth a permanent record of where every customer logs in from. The retention table on the logging page stands unchanged.
We did not move sessions into a shared cross-site store. A single session store spanning every site would have avoided the mixed-reader problem entirely by having one reader. It would also create precisely the centralised, always-hot record of who is connected to what that we have spent six years not building.
We did not add an outage announcement channel outside the status page. Several people suggested social media. The status page is the only thing we operate that we can commit to keeping accurate, and adding a second surface means a second surface that goes stale.
The credit
The SLA covers instance availability. Instances were available for all twenty-six minutes, so under the contract nobody was owed anything. We applied a one-day credit to every account that had a failed authentication in the window, which came to about three thousand one hundred euro, because arguing the distinction with people who could not reach their servers is a worse use of everyone’s afternoon.