Service Disruption: Gradient-Core NY2

Final update

# Paperspace NY2 network outage Postmortem ## Incident Summary On June 16th, a significant number of virtual machines in the NY2 region became unavailable and went into Read-Only \(RO\) mode, which affected network availability. By 18:20 UTC on the same day, network connectivity was restored to the affected virtual machines. However, affected virtual machines continued to be unavailable until 12:23 UTC on June 17th as they were still in RO mode. The affected virtual machines were restored to Read-Write \(RW\) mode between 5:58 and 12:23 UTC on June 17th. By 12:24 UTC, affected virtual machines were available and network access to customers was restored.  ## Incident Details ### Root Cause The root cause of this unexpected service unavailability is related to a core switch failure in NY2 that caused a large number of machines to go into RO mode. ### Impact As a result of the failure of the core switch, there was a spike in traffic that adversely impacted network performance. The virtual machines went into RO mode as a result of the loss of network access to the Network File System on which they resided. Since virtual machines were in RO mode, customers were unable to perform any wri

1. Postmortem · Jun 21, 18:46 UTC
   # Paperspace NY2 network outage Postmortem ## Incident Summary On June 16th, a significant number of virtual machines in the NY2 region became unavailable and went into Read-Only \(RO\) mode, which affected network availability. By 18:20 UTC on the same day, network connectivity was restored to the affected virtual machines. However, affected virtual machines continued to be unavailable until 12:23 UTC on June 17th as they were still in RO mode. The affected virtual machines were restored to Read-Write \(RW\) mode between 5:58 and 12:23 UTC on June 17th. By 12:24 UTC, affected virtual machines were available and network access to customers was restored.  ## Incident Details ### Root Cause The root cause of this unexpected service unavailability is related to a core switch failure in NY2 that caused a large number of machines to go into RO mode. ### Impact As a result of the failure of the core switch, there was a spike in traffic that adversely impacted network performance. The virtual machines went into RO mode as a result of the loss of network access to the Network File System on which they resided. Since virtual machines were in RO mode, customers were unable to perform any write operations on them, resulting in service disruptions around 10:30 UTC on June 16th, 2024.  ## Remediation Actions A number of efforts are underway to try to prevent these types of failures from occurring again, including a network redesign and installation of new equipment. On behalf of Paperspac
2. Resolved · Jun 17, 06:05 UTC
   Our engineers have resolved the issue. If you continue to experience issues, please contact our Support Team.
3. Monitoring · Jun 17, 00:11 UTC
   We've implemented a series of fixes and are monitoring the results. While Core is already fixed, the Gradient platform will continue to have degraded performance since it is still taking time to recover. We will continue working on fixes for it.
4. Identified · Jun 16, 21:55 UTC
   The issue has been identified, it appears to be an issue with our network in NY2. A fix is being implemented.
5. Investigating · Jun 16, 19:03 UTC
   We are continuing to investigate this issue.
6. Investigating · Jun 16, 15:23 UTC
   We are continuing to investigate this issue.
7. Investigating · Jun 16, 12:22 UTC
   We are continuing to investigate this issue.
8. Investigating · Jun 16, 12:13 UTC
   We are continuing to investigate this issue.
9. Investigating · Jun 16, 11:16 UTC
   We are currently investigating a network issue that is preventing users from interacting with Gradient notebooks and Core VMs based in NY2 region.