Event Streams Service Disruption
Sep 24, 20:40 UTCSep 25, 02:24 UTC
Duration
5h 44m
Impact
Minor
Root cause
Not disclosed
Twilio, 90 days
554 incidents
Affected
Event WebhooksSendGrid Marketing CampaignsSendGrid StatisticsSendGrid Email ActivitySendGrid Webhooks - Event Webhooks
Final update
Our engineers have monitored the fix and confirmed the issue with event streams has been resolved. All services are now operating normally at this time.
Timeline
- Resolved · Sep 25, 02:24 UTC
Our engineers have monitored the fix and confirmed the issue with event streams has been resolved. All services are now operating normally at this time.
- Identified · Sep 25, 02:05 UTC
Our team is continuing to work on restoring a small subset of events that are delayed from 12:12 PM PT and customers may see some duplicate events happened for a few minutes around 12:13 PM PT. We expect to provide another update in 1 hour or as soon as more information becomes available.
- Identified · Sep 25, 01:03 UTC
Our team is continuing to work on restoring a small subset of events that are delayed from 12:12 PM PT and customers may see some duplicate events happened for a few minutes around 12:13 PM PT. We expect to provide another update in 1 hour or as soon as more information becomes available.
- Identified · Sep 24, 23:45 UTC
Our team is continuing to work on restoring a small subset of events that are delayed from 12:12 PM PT and customers may see some duplicate events happened for a few minutes around 12:13 PM PT. We expect to provide another update in 1 hour or as soon as more information becomes available.
- Identified · Sep 24, 22:13 UTC
Our engineers have identified the issue and are working on restoring delayed events. A subset of mail-send related events (such as Delivered, Deferred, Bounces, Spam Reports) for Email Activity, Marketing Campaigns, Group Unsubscribes, and Event Webhook are delayed for a specific time window (TBD). Email stats for new mail recieved after 12:15 PM PT is not impacted and no event data was lost. We will provide an update in 1 hour or as soon as more information becomes available.
- Investigating · Sep 24, 20:44 UTC
Our engineers are continuing to investigate the issue causing inconsistencies with Event Streams. We expect to provide another update in 2 hours or as soon as more information becomes available.
- Investigating · Sep 24, 20:40 UTC
We are investigating a service interruption with Event Streams. Customers may experience event data loss or delays in event delivery. Our team is actively investigating the scope of impact and data recovery options. We expect to provide another update in 2 hours or as soon as more information becomes available.
More from Twilio
Full history| Started | Incident | Impact | Duration |
|---|---|---|---|
| Sep 2503:49 UTC | SMS Delivery Delays from Twilio to Claro Brazil | minor | Ongoing |
| Sep 2502:24 UTC | SMS Delivery Delays from a Subset of Twilio Short Codes to Cellularone NE Arizona United States | minor | Ongoing |
| Sep 2417:55 UTC | SMS Delivery Receipt Delays from a Subset of Twilio Alphanumeric Sender IDs to Ooredoo Algeria | minor | Ongoing |
| Sep 2417:19 UTC | Some Transfer Calls in Elastic SIP Trunking Failing With Error Code 32204 | minor | 43m |
| Sep 2416:22 UTC | Link Branding Creation Fails with HTTP 500 Internal Server Error (Auto SSL Enabled) | minor | 2h 4m |
| Sep 2414:37 UTC | Customers Using Conversation Intelligence (Classic) May Experience Delays With Transcript and Operator Results Being Available | minor | 4h 58m |
Sources: vendors' own status pages, published postmortems and SEC 8-K Item 1.05 filings, read daily. Times as reported. Logos via logo.dev; trademarks belong to their owners.