Who is down right now

26 down now · 292 this week

Humanloop · AIJan 3, 2025, 18:44 UTC 1 year ago

Failed deployment migration impacting one of our customer specific environments

MajorNot disclosedUpdated 4d ago
Jan 3, 18:44 UTCJan 3, 20:30 UTC
Duration
1h 46m
Impact
Major
Root cause
Not disclosed
Humanloop, 90 days
0 incidents
Affected
Not listed by the vendor.
Status page

Final update

**Post mortem completed** Actions identified for resolution: - Resolve the issue that allowed a silent failure for a migration of this kind during a regular deployment. - Resolve the issue with the alarms on the customer environment that delayed the response.

Timeline

  1. Resolved · Jan 3, 20:30 UTC
    **Post mortem completed** Actions identified for resolution: - Resolve the issue that allowed a silent failure for a migration of this kind during a regular deployment. - Resolve the issue with the alarms on the customer environment that delayed the response.
  2. Monitoring · Jan 3, 19:40 UTC
    Customer notified that issue has been resolved and logging in should be operational again.
  3. Monitoring · Jan 3, 19:36 UTC
    Deployment successful, impacted environment fully operational again.
  4. Investigating · Jan 3, 19:15 UTC
    Preparing a fix for the migration and re-deploying to the impacted environment.
  5. Investigating · Jan 3, 19:05 UTC
    The issue reported was caused by a recent deployment. It was identified as a minor database migration that failed silently but didn't block the deployment. The successfully deployed code then caused issues preventing the user from logging in. Only a single customer environment was impacted by the issue, the deployment across other environments, including the main app.humanloop.com, was successful with no issues.
  6. Investigating · Jan 3, 18:57 UTC
    We provided first response to the user in Slack acknowledging the issue and starting the investigation.
  7. Identified · Jan 3, 18:44 UTC
    User reported issue in Slack with failed SSO when attempting to log in.

More from Humanloop

Full history