Who is down right now

26 down now · 292 this week

Metabase · Data and observabilityNov 9, 2023, 19:37 UTC 3 years ago

Issue with SSL certificates

CriticalConfig changeUpdated 6d ago
Nov 9, 19:37 UTCNov 9, 20:02 UTC
Duration
25m
Impact
Critical
Root cause
Config change
Metabase, 90 days
0 incidents
Affected
Metabase Cloud Platform
Status page

Final update

**Summary** On Thursday Nov 9th the default certificate for the external Nginx ingress was replaced with a dummy certificate. Users attempting to reach their hosted instances would get an invalid certificate error. This was due to a misconfiguration in the new CI for releasing our helm charts that used the internal Nginx ingress values files for the external Nginx ingress. **Impact** All hosted customers not using custom domains were affected. **How was the root cause diagnosed?** We identified that the values were properly set in the values files. We Investigated the CI steps to determine why the correct values were not set. It was determined that the wrong values files were being used for external Nginx ingress. **How we’ll make this not happen again?** * Update staging and production value files to match. * Add alert to pagerduty for non-custom domains \(catch alerts faster\)

Timeline

  1. Postmortem · Nov 15, 13:15 UTC
    **Summary** On Thursday Nov 9th the default certificate for the external Nginx ingress was replaced with a dummy certificate. Users attempting to reach their hosted instances would get an invalid certificate error. This was due to a misconfiguration in the new CI for releasing our helm charts that used the internal Nginx ingress values files for the external Nginx ingress. **Impact** All hosted customers not using custom domains were affected. **How was the root cause diagnosed?** We identified that the values were properly set in the values files. We Investigated the CI steps to determine why the correct values were not set. It was determined that the wrong values files were being used for external Nginx ingress. **How we’ll make this not happen again?** * Update staging and production value files to match. * Add alert to pagerduty for non-custom domains \(catch alerts faster\)
  2. Resolved · Nov 9, 20:02 UTC
    Ingress controller certificates are now back to normal. You should be able to get the correct certificate in the browser now. We're very sorry for the inconvenience, we'll publish a retrospective about the issue in the next few days.
  3. Investigating · Nov 9, 19:37 UTC
    We're investigating an issue in our cloud with certificate management that's causing browsers to receive incorrect certificates

More from Metabase

Full history

Also caused by configuration change

All