Partial service outage
Mar 11, 06:00 UTCMar 11, 07:50 UTC
Duration
1h 50m
Impact
Major
Root cause
Not disclosed
Humanloop, 90 days
0 incidents
Affected
Not listed by the vendor.
Final update
Services are all now back to normal
Timeline
- Resolved · Mar 11, 07:50 UTC
Services are all now back to normal
- Monitoring · Mar 11, 07:00 UTC
Monitoring services after fix deployed and failover to read replica completed.
- Identified · Mar 11, 06:50 UTC
Post deployment database backfill activities caused anomalous database load leading to severe contention and downtime. Contention was resolved by promoting read replica to master writer instance.
- Investigating · Mar 11, 06:00 UTC
Early warning alarms around master database load triggered. On call engineer pinged, investigation started.
More from Humanloop
Full historyPerformance Degradation on the APIlasted 1h 26mPerformance degradation in the EU instancelasted 7mPerformance Degradation - Increased Load Timeslasted 52mFailed deployment migration impacting one of our customer specific environmentslasted 1h 46mSlow API response timeslasted 28h 10mSlow API response timeslasted 17m