Metrics Alert Rule Evaluation Failures in AWS Australia (prod-au-southeast-1)
Nov 2, 09:20 UTCNov 2, 11:08 UTC
Duration
1h 48m
Impact
Minor
Root cause
Not disclosed
Grafana Labs, 90 days
64 incidents
Affected
AWS Australia - prod-au-southeast-1: Rule EvaluationAlerting and SLO - AWS Australia - prod-au-southeast-1: Rule Evaluation
Final update
Between approximately 08:00 and 09:40 UTC (about 1 hour 40 minutes), some customers in AWS Australia (prod-au-southeast-1) experienced intermittent failures when evaluating alert and recording rules, which may have caused missed or delayed alert notifications and gaps in recorded series. This has been fully resolved and rule evaluation is operating normally.
Timeline
- Resolved · Nov 2, 11:08 UTC
Between approximately 08:00 and 09:40 UTC (about 1 hour 40 minutes), some customers in AWS Australia (prod-au-southeast-1) experienced intermittent failures when evaluating alert and recording rules, which may have caused missed or delayed alert notifications and gaps in recorded series. This has been fully resolved and rule evaluation is operating normally.
- Investigating · Nov 2, 10:17 UTC
We continue to investigate intermittent failures when evaluating alert and recording rules in AWS Australia (prod-au-southeast-1), first observed at approximately 08:00 UTC. Some customers may see rules fail to evaluate or be delayed, which can result in missed or delayed alert notifications and gaps in recorded series. We are continuing to investigate. Next update by 11:15 UTC.
- Investigating · Nov 2, 09:20 UTC
We are investigating an elevated rate of failures when evaluating Metrics alert and recording rules in AWS Australia (prod-au-southeast-1). Some customers may see rules missed or delayed rule evaluations, which can result in missed or delayed alert notifications and gaps in recorded series. Next update by 10:20 UTC.
More from Grafana Labs
Full historyMissing Metrics for Some Synthetic Monitoring Checks using the Montreal, North Virginia, and North California public probeslasted 0mIRM Unable to Voice Call Brazil Numbersstill openFrontend Observability Outage in prod-eu-west-2lasted 28mDegraded Cloud Integrationslasted 4h 42mingestion errors in mimir-prod-66lasted 1hSome Grafana Instances Experiencing Issues Coming Online after Restartlasted 4h 43m