Red Hat Connect Major Outage - Investigating

Final update

Executive Summary A certificate update by the backend team triggered automated system alerts this morning. The Partner Certification Team investigated the alerts and identified the undocumented certificate rotation as the root cause of the incident. Detection & Investigation: The Partner Certification Team detected the issue via automated alerts. The team immediately initiated debugging procedures and successfully isolated the certificate update as the underlying cause. Recommended Action Items: Implement a mandatory communication protocol and sign-off process between the platform team and the Partner Certification Team for all future certificate rotations. Review monitoring systems to ensure certificate mismatches trigger immediate alerts during deployment windows, rather than the following morning.

1. Postmortem · Sep 25, 05:25 UTC
   Executive Summary A certificate update by the backend team triggered automated system alerts this morning. The Partner Certification Team investigated the alerts and identified the undocumented certificate rotation as the root cause of the incident. Detection & Investigation: The Partner Certification Team detected the issue via automated alerts. The team immediately initiated debugging procedures and successfully isolated the certificate update as the underlying cause. Recommended Action Items: Implement a mandatory communication protocol and sign-off process between the platform team and the Partner Certification Team for all future certificate rotations. Review monitoring systems to ensure certificate mismatches trigger immediate alerts during deployment windows, rather than the following morning.
2. Resolved · Sep 25, 05:22 UTC
   Issue has resolved Hydra has updated the certificates which caused our outage in the partner portal. we have updated the certificates configs in our application in alignment to it.
3. Investigating · Sep 25, 05:02 UTC
   We are currently investigating this issue.
4. Identified · Sep 25, 05:01 UTC
   We are currently experiencing an outage affecting the Red Hat Partner Connect. Our teams are actively investigating the issue and working to restore full functionality as soon as possible. We apologize for any inconvenience and appreciate your patience. Updates will be provided as more information becomes available.