Incidentsby IbisSome customers in us-west-2 may experience elevated rate of errors
Dec 18, 20:35 UTCDec 18, 20:35 UTC
Duration
0m
Impact
Minor
Root cause
Not disclosed
Temporal, 90 days
27 incidents
Affected
Not listed by the vendor.
Final update
Some customers in us-west-2 may have seen resource exhausted errors on poll RPCs and increased schedule-to-start latency, task backlogs for a short period 20:35 UTC Investigating 21:05 UTC Resolved A small number of customers experienced intermittent ResourceExhausted errors when polling for tasks, along with brief increases in schedule-to-start latency and task backlogs. This occurred due to elevated load on one of our core services (the part of the system that manages task queues and delivers tasks to workers). The issue was short-lived, and system performance has since returned to normal.
Timeline
- Resolved · Dec 18, 20:35 UTC
Some customers in us-west-2 may have seen resource exhausted errors on poll RPCs and increased schedule-to-start latency, task backlogs for a short period 20:35 UTC Investigating 21:05 UTC Resolved A small number of customers experienced intermittent ResourceExhausted errors when polling for tasks, along with brief increases in schedule-to-start latency and task backlogs. This occurred due to elevated load on one of our core services (the part of the system that manages task queues and delivers tasks to workers). The issue was short-lived, and system performance has since returned to normal.
More from Temporal
Full historyNew namespace creation restricted across all GCP regionsstill openElevated latency and errors in GCP us-west1lasted 17mElevated latency and errors in AWS us-east-1lasted 2h 17mElevated error rates for cross-region traffic between US AWS regionslasted 0mElevated GCP storage latency in us-central1 – precautionary namespace failoverlasted 4h 25mIncreased errors on Temporal Cloud control plane, Cloud Ops API, and Temporal UIlasted 29m