Delayed Monitors Notifications
May 8, 00:06 UTCMay 8, 14:07 UTC
Duration
14h 1m
Impact
Critical
Root cause
Dependency
Datadog, 90 days
9 incidents
Affected
APMCI VisibilityError TrackingLog ManagementMetrics and Infra MonitoringMonitorsRUM
Final update
This incident has been resolved.
Timeline
- Resolved · May 8, 14:07 UTC
This incident has been resolved.
- Monitoring · May 8, 06:21 UTC
We have identified upstream provider issues. Metrics, Logs, APM, RUM and CI visibility data are being processed normally for all customers. Alerting is also functional. We are monitoring the situation. https://health.aws.amazon.com/health/status
- Identified · May 8, 05:44 UTC
We have identified upstream provider issues. Metrics, Logs, APM and RUM data are being processed normally for all customers. Alerting is also functional. There are still processing delays for CI visibility. We are monitoring the situation. https://health.aws.amazon.com/health/status
- Identified · May 8, 05:30 UTC
We have identified upstream provider issues and are continuing to experience delays in processing data across multiple products. We are continuing to work on a fix. https://health.aws.amazon.com/health/status For metrics, distribution metrics and point metrics are being processed normally for all customers.
- Identified · May 8, 04:48 UTC
We have identified upstream provider issues and are continuing to experience delays in processing data across multiple products. We are continuing to work on a fix. https://health.aws.amazon.com/health/status For metrics, distribution metrics and point metrics are being processed normally for all customers.
- Identified · May 8, 04:13 UTC
We have identified upstream provider issues and are continuing to experience delays in processing data across multiple products. We are continuing to work on a fix. https://health.aws.amazon.com/health/status For metrics, distribution metrics and point metrics are being processed normally for all customers.
- Identified · May 8, 02:59 UTC
We have identified due to upstream provider issues, we are continuing to see unavailability of telemetry data coming from AWS into Datadog. We are continuing to work on a fix. https://health.aws.amazon.com/health/status For metrics we are still seeing delays in distribution metrics. Counts, rates and gauge metrics are being processed normally for most customers
- Identified · May 8, 02:16 UTC
We have identified due to upstream provider issues, we are continuing to see unavailability of telemetry data coming from AWS into Datadog. We are continuing to work on a fix. https://health.aws.amazon.com/health/status
- Identified · May 8, 02:02 UTC
We have identified upstream provider issues and are continuing to experience delays in processing data across multiple products. We are working on a fix. https://health.aws.amazon.com/health/status
- Identified · May 8, 01:20 UTC
We have identified upstream provider issues and are continuing to experience delays in processing data across multiple products. We are working on a fix. https://health.aws.amazon.com/health/status
- Investigating · May 8, 00:47 UTC
Due to upstream provider issues, we are also continuing to see unavailability of telemetry data coming from AWS into Datadog. https://health.aws.amazon.com/health/status
- Investigating · May 8, 00:24 UTC
We are investigating increased latency across multiple products which began at 23:39 UTC. As a result of this issue, some users may see delays in data across the platform.
- Investigating · May 8, 00:06 UTC
We are investigating delays in Monitors Notifications, which began at 23:39 UTC.
More from Datadog
Full historyMetrics query are disruptedlasted 2h 29mElevated Error Rates for Workflow Automation, App Builder, Bits Code, Bits Agent Builderlasted 57mDelayed RUM sessionslasted 2h 17mDelayed Eventslasted 48mUsers are unable to acknowledge, escalate, or resolve On-Call Pageslasted 27mDelayed Monitors Notificationslasted 46m