Authentication errors in prod-us-west-0

Final update

This incident has been resolved.

1. Resolved · Oct 14, 15:45 UTC
   This incident has been resolved.
2. Monitoring · Oct 14, 15:25 UTC
   The fix has been implemented, and we are monitoring the results.
3. Identified · Oct 14, 15:00 UTC
   We have noticed improvements across services. We will now be applying a more permanent fix. Impacted users may experience a disruption in the auth-api service of up to 15min while we apply a DB patch that will improve performance.
4. Identified · Oct 14, 13:58 UTC
   The issue has been identified, and we have applied mitigations on our end. We are currently observing the progress.
5. Investigating · Oct 14, 12:57 UTC
   We are investigating increased authentication errors affecting Grafana Cloud in the prod-us-west-0 region. Customers may experience intermittent failures for requests that require authentication. Our engineering team has identified database performance degradation affecting the authentication service and is investigating the underlying cause. We will provide another update as more information becomes available.