Investigating intermittent timeouts on historical APIs

Final update

This incident has been resolved.

1. Resolved · Apr 28, 22:26 UTC
   This incident has been resolved.
2. Monitoring · Apr 28, 00:34 UTC
   We have taken additional steps to resolve the slow operations on the storage cluster. We will continue to monitor for issues.
3. Identified · Apr 27, 14:39 UTC
   We are still experiencing performance issues on our storage cluster. This is causing intermittent issues on the historical API and delays to historical data availability. We are working on a fix.
4. Monitoring · Apr 27, 12:56 UTC
   We have taken steps to resolve the slow operations on the storage cluster. We will continue to monitor for issues.
5. Identified · Apr 27, 10:38 UTC
   We have taken action to mitigate the impact. Our systems engineers have identified the root cause and are working to resolve the issue. Customers may continue to experience timeouts and errors from the historical API.
6. Investigating · Apr 27, 10:14 UTC
   We are continuing to investigate this issue.
7. Investigating · Apr 27, 10:13 UTC
   We are currently investigating intermittent issues on our historical API. We believe this is due to slow reads on our storage cluster.