Region us-central1 partial degradation

Lesson: Data center cooling failures can lead to rapid thermal shutdowns, requiring robust disaster recovery plans for whole-region power loss and manual recovery processes.

What happened

[https://nebius.com/blog/posts/incident-post-mortem-analysis-us-central1-service-disruption-on-august-19](https://nebius.com/blog/posts/incident-post-mortem-analysis-us-central1-service-disruption-on-august-19)

1. Postmortem · Sep 1, 09:15 UTC
   [https://nebius.com/blog/posts/incident-post-mortem-analysis-us-central1-service-disruption-on-august-19](https://nebius.com/blog/posts/incident-post-mortem-analysis-us-central1-service-disruption-on-august-19)
2. Resolved · Aug 20, 08:16 UTC
   This incident has been resolved.
3. Identified · Aug 19, 21:59 UTC
   Most critical issues have been resolved and the region is operational, though a subset of nodes remains unavailable.
4. Identified · Aug 19, 20:50 UTC
   The Managed Service for Kubernetes is nearly fully restored, though a subset of nodes remains unavailable.
5. Identified · Aug 19, 19:13 UTC
   Most nodes are operating normally. The Managed Service for Kubernetes® is still degraded, but the situation is improving.
6. Identified · Aug 19, 17:26 UTC
   Most nodes are operating normally. The Managed Service for Kubernetes® is experiencing a partial outage, and the team is actively working to restore it.
7. Identified · Aug 19, 16:00 UTC
   Services are still gradually continuing to recover. Many of the virtual machines are already running, but not all of them.
8. Identified · Aug 19, 15:06 UTC
   Services are still gradually continuing to recover. Some virtual machines are already running, but not all of them.
9. Identified · Aug 19, 13:49 UTC
   Services are gradually continuing to recover.
10. Identified · Aug 19, 12:52 UTC
    The root cause was fixed, and the situation is stabilizing. The services are recovering.
11. Identified · Aug 19, 12:20 UTC
    We are still experiencing temperature issues in the us-central1. Team is working on it
12. Identified · Aug 19, 11:18 UTC
    The root cause has been identified as overheating hardware. GPU performance is impacted, and some nodes may be unavailable.
13. Investigating · Aug 19, 10:43 UTC
    We are currently investigating this issue.