CDC ClickPipes stuck in maintenance state
Jul 22, 11:53 UTCJul 22, 14:36 UTC
Duration
2h 43m
Impact
Major
Root cause
Not disclosed
ClickHouse, 90 days
18 incidents
Affected
Not listed by the vendor.
Final update
The CDC ClickPipes upgrade rollout remains on hold while we confirm we can safely proceed without triggering the underlying pod networking failure mode that triggered the incident.
Timeline
- Resolved · Jul 22, 14:36 UTC
The CDC ClickPipes upgrade rollout remains on hold while we confirm we can safely proceed without triggering the underlying pod networking failure mode that triggered the incident.
- Monitoring · Jul 22, 13:39 UTC
All services have recovered; we will continue monitoring the affected nodes and consolidating our findings. Please reach out to our support team to request access to the RCA, when available.
- Identified · Jul 22, 13:05 UTC
Most services have fully recovered. Continuing to roll out the fix to remaining 4 services (all in `eu-central-1`, `us-east-2`).
- Identified · Jul 22, 12:36 UTC
Services in `eu-central-1` have recovered. We're in the process of manually recovering ~20 services in other regions.
- Investigating · Jul 22, 11:53 UTC
As part of our weekly upgrade process, CDC ClickPipes in `eu-central-1` became stuck in maintenance state. The team is working on a mitigation.
More from ClickHouse
Full history| Started | Incident | Impact | Duration |
|---|---|---|---|
| Sep 2408:42 UTC | Streaming ClickPipes degradation in AWS eu-central-1 region | minor | 29m |
| Sep 2408:31 UTC | Some ClickPipes CDC pipes paused and can't be resumed | minor | 33m |
| Sep 2315:28 UTC | We are investigating an issue affecting Azure eastus2 region | minor | 30m |
| Sep 2213:19 UTC | We are investigating an issue with CH servers in Azure germanywestcentral region | minor | 5h 6m |
| Sep 1610:16 UTC | Intermittent Issues with the Support tab | minor | 13h 50m |
| Sep 1518:57 UTC | ClickPipes failing on Kinesis in AWS us-east-1 | critical | 29h 46m |
Sources: vendors' own status pages, published postmortems and SEC 8-K Item 1.05 filings, read daily. Times as reported. Logos via logo.dev; trademarks belong to their owners.