Monitors and Alerting delayed or missed

Final update

This issue is now resolved. The main impact to customers was delay in our modeling and alerting pipeline that would have caused any potential alert to be delayed potentially multiple hours. Our custom cron scheduling system was impacted, causing us to miss observing any custom cron monitors from from 12:17 EST -> 8AM EST

1. Resolved · Jan 29, 13:52 UTC
   This issue is now resolved. The main impact to customers was delay in our modeling and alerting pipeline that would have caused any potential alert to be delayed potentially multiple hours. Our custom cron scheduling system was impacted, causing us to miss observing any custom cron monitors from from 12:17 EST -> 8AM EST
2. Monitoring · Jan 29, 13:46 UTC
   Alerts are sending in realtime again
3. Identified · Jan 29, 13:13 UTC
   From 12:17 EST -> 8AM EST monitoring for customers was highly lagged, causing a delay or miss in alerting. Metrics with a custom schedule also failed to run. We have identified the issue and are implementing a fix so we can recover and alert on as much data as possible.