E-Paper
Failed health check, elevated load, server restart: What led to the Microsoft outage
Automated server shutdowns created an infrastructure bottleneck, even as engineers deploy alternatives, while an initial fix attempt don’t seem to be delivering results
Prefer HTon Google
Microsoft services, including Microsoft 365 for Business and Enterprise, as well as consumer products such as Microsoft Teams, Outlook, OneDrive, and Office on the web, continue to experience what the tech company described as a “service degradation” following a major global outage that has stretched across several hours.
Microsoft has yet to confirm why the services went offline in the early hours of Friday, but it attributes the outage to several servers unexpectedly becoming unavailable after failing automated health checks, which placed elevated strain on the remaining infrastructure worldwide.
The Microsoft Service Health Status currently lists all customer services as “operational”, but Microsoft’s update on X indicated there were issues to resolve.
Deep Dive
“We’ve monitored the status of the changes applied to improve serviceability and confirmed they’re not taking as expected. We’re identifying alternate workstreams that can be applied to increase capacity in the face of sustained traffic management imbalances,” said Microsoft 365 Status account on X.
Why the services went offline
By all accounts, the outage was caused by a cascading infrastructure failure.
It began with automated health checks across its server infrastructure, which failed on certain servers and caused them to shut down. This increased the load on the remaining online servers, causing a cascading effect. Microsoft engineers have attempted to restart these servers, and Microsoft 365 Status does indicate that a portion of the affected servers has been restarted and remains stable.
Restarting a server requires as long as 30 minutes, depending on hardware and workload. After an unexpected crash or hard reboot, the server may run a disk check process, which can take a long time on large drives. There is no guarantee that the issue which caused the server to shut down or become unresponsive will be fixed by a restart.
“We’re testing a configuration change to address a problem that’s causing servers to fail automated health checks and become unavailable,” Microsoft said before it started deploying certain fixes. That attempt hasn’t delivered results so far.
Microsoft said the disruption wasn’t driven by a spike in user traffic or a cyberattack. Instead, a core process on affected servers became overwhelmed, causing capacity loss when servers failed automated health checks and dropped offline. As a result, services became unavailable and unstable.
