Degraded Ingest due to database connectivity issues

Final update

This incident is now resolved. We are also noting that some other parts of honeycomb (the query UI) may have seen spurious failures as well during the impacted period (13:22, 13:44 UTC).

1. Resolved · Mar 10, 14:14 UTC
   This incident is now resolved. We are also noting that some other parts of honeycomb (the query UI) may have seen spurious failures as well during the impacted period (13:22, 13:44 UTC).
2. Monitoring · Mar 10, 13:56 UTC
   We have stabilized the cluster and are monitoring the state while investigating to understand what triggered the degradation. Overall, 0.04% traffic was rejected with an error due to this degradation during the 20 minute window we were impaired.
3. Investigating · Mar 10, 13:50 UTC
   We are currently seeing elevated errors in connecting to some of our databases, which results in low amounts of Ingest traffic being rejected. We are looking into correcting the situation.