Batch job processing delayed
Jul 29, 08:51 UTCAug 5, 01:35 UTC
Duration
6d 16h
Impact
Critical
Root cause
Not disclosed
Databento, 90 days
15 incidents
Affected
Historical
Final update
This incident has been resolved.
Timeline
- Resolved · Aug 5, 01:35 UTC
This incident has been resolved.
- Monitoring · Jul 30, 16:58 UTC
We've implemented some improvements to our batch job processing infrastructure, and have observed that it is keeping up with incoming batch jobs. We're continuing to monitor for further delays or issues.
- Identified · Jul 30, 12:33 UTC
The fixes implemented earlier helped the situation temporarily, but the batch system has fallen behind again. This is impacting the rate at which customer jobs are being completed. We are working on a solution.
- Monitoring · Jul 29, 10:23 UTC
A fix has been implemented and we are monitoring the results.
- Identified · Jul 29, 08:51 UTC
Our batch job processing system fell behind, processing is resuming with the backlog of jobs. (There is no need to re-submit any jobs.) We're expecting the system to be caught up by 10:00 UTC.