Degraded networking to AWS IPs

Final update

This incident has been resolved, connectivity from AWS to all of our hosts is back to normal.

1. Resolved · Oct 26, 20:40 UTC
   This incident has been resolved, connectivity from AWS to all of our hosts is back to normal.
2. Identified · Oct 26, 19:23 UTC
   Connectivity from AWS to Fly.io on impacted hosts is continuing to recover. Approximately 80% of the hosts that were seeing connectivity issues are fully recovered at this time.
3. Identified · Oct 26, 18:08 UTC
   AWS has implemented changes to unblock connections from their network from reaching our machines. We are seeing connectivity starting to resume on a host-by-host basis as these changes take effect.
4. Identified · Oct 26, 16:26 UTC
   We are continuing to work with our upstream networking partners and AWS on remediation. Impacted customers are recommended to allocate app-scoped static egress IPs using **fly ips allocate-egress.** Customers seeing issues are encouraged to also contact AWS support or their AWS hosted vendors support with details of the connection errors and their fly machine IPs.
5. Identified · Oct 26, 15:15 UTC
   We are continuing to see connectivity issues from AWS IPs to a subset of our hosts. From our investigation these connections appear to be being blocked from the AWS side. We are continuing to work with our upstream networking partners and AWS on remediation. In the meantime impacted customers are recommended to allocate app-scoped static egress IPs using **fly ips allocate-egress.**
6. Investigating · Oct 26, 14:54 UTC
   Our initial investigations indicate connection errors are occuring the return path from AWS IPs. We are escalating the issue with AWS and continuing to investigate. Customers seeing issues are encouraged to also contact AWS support or their AWS hosted vendors support with details of the connection errors and their fly machine IPs.
7. Investigating · Oct 26, 14:34 UTC
   We are investigating reports of elevated networking issues between our network and AWS IPs. We are working to determine the source of the issue.