Incidentsby GlyphKPO podspec changes for GPU-enabled nodes may cause KPO pod startup failures
Aug 4, 15:11 UTCAug 4, 16:40 UTC
Duration
1h 30m
Impact
Minor
Root cause
Not disclosed
Astronomer, 90 days
17 incidents
Affected
Not listed by the vendor.
Final update
Mitigations have been successful. A Platform release that corrects this errant behavior is being worked on now. Issue is resolved.
Timeline
- Resolved · Aug 4, 16:40 UTC
Mitigations have been successful. A Platform release that corrects this errant behavior is being worked on now. Issue is resolved.
- Monitoring · Aug 4, 15:15 UTC
We have rolled back the changes that triggered this issue and are monitoring the issue for confirmation that this issue is resolved.
- Identified · Aug 4, 15:11 UTC
We've identified a change that enabled GPU support for KubernetesPodOperator pods had an unintended side effect when reading pod specs without proper limits set in the expected format. We're working on a fix, if you notice any unexpected failures for KPO tasks in your Astro deployments, please open a support ticket.
More from Astronomer
Full historyPHP Security Advisory: Astro and Astro Private Cloud Not Affected; Review Custom Imagesstill openSecurity Update: Runtime Releases and Guidance for the eMudhra/emSign Certificate Issuelasted 45h 27mDelayed delivery of Astro Alertslasted 26h 9mOrganization Dashboards are failing to load with Invalid kid parameter errorlasted 1h 23mAstro UI and API are appearing offline for some userslasted 1h 51mCapacity constraints in GCP us-central1 may delay or fail some task executionslasted 7d 21h