Failed deployment migration impacting one of our customer specific environments
Jan 3, 18:44 UTCJan 3, 20:30 UTC
Duration
1h 46m
Impact
Major
Root cause
Not disclosed
Humanloop, 90 days
0 incidents
Affected
Not listed by the vendor.
Final update
**Post mortem completed** Actions identified for resolution: - Resolve the issue that allowed a silent failure for a migration of this kind during a regular deployment. - Resolve the issue with the alarms on the customer environment that delayed the response.
Timeline
- Resolved · Jan 3, 20:30 UTC
**Post mortem completed** Actions identified for resolution: - Resolve the issue that allowed a silent failure for a migration of this kind during a regular deployment. - Resolve the issue with the alarms on the customer environment that delayed the response.
- Monitoring · Jan 3, 19:40 UTC
Customer notified that issue has been resolved and logging in should be operational again.
- Monitoring · Jan 3, 19:36 UTC
Deployment successful, impacted environment fully operational again.
- Investigating · Jan 3, 19:15 UTC
Preparing a fix for the migration and re-deploying to the impacted environment.
- Investigating · Jan 3, 19:05 UTC
The issue reported was caused by a recent deployment. It was identified as a minor database migration that failed silently but didn't block the deployment. The successfully deployed code then caused issues preventing the user from logging in. Only a single customer environment was impacted by the issue, the deployment across other environments, including the main app.humanloop.com, was successful with no issues.
- Investigating · Jan 3, 18:57 UTC
We provided first response to the user in Slack acknowledging the issue and starting the investigation.
- Identified · Jan 3, 18:44 UTC
User reported issue in Slack with failed SSO when attempting to log in.