Incidentsby GlyphCDC ClickPipes stuck in maintenance state
Jul 22, 11:53 UTCJul 22, 14:36 UTC
Duration
2h 43m
Impact
Major
Root cause
Not disclosed
ClickHouse, 90 days
15 incidents
Affected
Not listed by the vendor.
Final update
The CDC ClickPipes upgrade rollout remains on hold while we confirm we can safely proceed without triggering the underlying pod networking failure mode that triggered the incident.
Timeline
- Resolved · Jul 22, 14:36 UTC
The CDC ClickPipes upgrade rollout remains on hold while we confirm we can safely proceed without triggering the underlying pod networking failure mode that triggered the incident.
- Monitoring · Jul 22, 13:39 UTC
All services have recovered; we will continue monitoring the affected nodes and consolidating our findings. Please reach out to our support team to request access to the RCA, when available.
- Identified · Jul 22, 13:05 UTC
Most services have fully recovered. Continuing to roll out the fix to remaining 4 services (all in `eu-central-1`, `us-east-2`).
- Identified · Jul 22, 12:36 UTC
Services in `eu-central-1` have recovered. We're in the process of manually recovering ~20 services in other regions.
- Investigating · Jul 22, 11:53 UTC
As part of our weekly upgrade process, CDC ClickPipes in `eu-central-1` became stuck in maintenance state. The team is working on a mitigation.
More from ClickHouse
Full historySQL Console database connection outagelasted 50mDegraded support ticket submission and responselasted 2h 48mGoogle Cloud Outagelasted 6h 44mSupport case creation unavailablelasted 31mStreaming ClickPipes degradation in AWS eu-central-1 regionlasted 29mSome ClickPipes CDC pipes paused and can't be resumedlasted 33m