Checkout service degradation

Final update

This incident has been resolved.

1. Resolved · Oct 21, 20:00 UTC
   This incident has been resolved.
2. Monitoring · Oct 20, 23:53 UTC
   Affirm US Checkout service has sustained high availability for an extended period of time. We are changing the status of this incident to `Monitoring` and will resolve the incident after a longer period of observation.
3. Identified · Oct 20, 22:13 UTC
   We are continuing to closely monitor the service health of our dependencies and their impact on our services. We will not declare our incident resolved until we are convinced that all issues with Affirm's dependencies are 100% resolved.
4. Identified · Oct 20, 21:39 UTC
   We have applied internal mitigations successfully and Affirm US Checkout service continues to gradually improve in availability. Core Affirm services remain available for the vast majority of users. We are continuing to closely monitor the service health of our dependencies.
5. Identified · Oct 20, 21:01 UTC
   We are continuing to measure the risks and benefits of direct mitigations to remove the last traces of impact from Affirm US Checkout service. Core Affirm services remain available for the vast majority of users.
6. Identified · Oct 20, 20:18 UTC
   To eliminate lingering incident impact we are attempting some more direct mitigations. Core Affirm services remain available for the vast majority of users. Our next update will be posted in 30 minutes.
7. Identified · Oct 20, 19:44 UTC
   We continue to observe minor impact on some endpoints within our US Checkout service. Core Affirm services remain available for the vast majority of users. Our next update will be posted in 30 minutes.
8. Identified · Oct 20, 19:13 UTC
   We are continuing to monitor this incident. Consumers may be experiencing elevated latency and sporadic, recoverable errors, but core Affirm services remain available. Our next update will be posted in 30 minutes.
9. Identified · Oct 20, 18:29 UTC
   Affirm US service health is continuing to improve. Availability and latency metrics are gradually returning to normal levels. We will continue to update every 30m until the issue is resolved.
10. Identified · Oct 20, 17:55 UTC
    We have identified the root causes of impact to Affirm Checkout in US & CA. We have applied a mitigation in CA and have observed a return to normal availability. We are continuing to work on this issue for our US integrations and will continue to update on status.
11. Investigating · Oct 20, 17:21 UTC
    We are continuing to investigate this issue, and exploring possible failover mitigation options where possible. We will provide another update in 30 minutes.
12. Investigating · Oct 20, 16:39 UTC
    We are investigating an increase in latencies and recoverable errors on Affirm Checkout. This is related to the critical infrastructure partner outage that impacted Affirm last night. Users currently impacted will be prompted to refresh their checkouts and are able to succeed after retrying. We are determining which integrations are impacted, specifically, now.