Delays starting Depot Builders in IAD
Jul 9, 15:21 UTCJul 9, 16:18 UTC
Duration
57m
Impact
Minor
Root cause
Not disclosed
Fly.io, 90 days
52 incidents
Affected
DeploymentsRemote Builds
Final update
This incident has been resolved.
Timeline
- Resolved · Jul 9, 16:18 UTC
This incident has been resolved.
- Monitoring · Jul 9, 15:43 UTC
A fix has been implemented and we are seeing improvements in builder performance, latency, and error rates. We are continuing to monitor for a full recovery. Customers still seeing issues can trigger a fly-hosted builder based deploy with `fly deploy --depot=false`.
- Identified · Jul 9, 15:21 UTC
We have identified an issue causing delays or failures when starting depoting builders located in the IAD region. Customers with builders in IAD may see delays or timeouts starting builds during `fly deploy`. We are working on a fix. In the meantime customers can trigger a fly-hosted builder based deploy with `fly deploy --depot=false`. You can also change your builder region away from IAD via the `settings` tab of your Fly dashboard. We recommend DFW or ORD as alternate builder regions at this time. Please note, your builder region can differ from the region your machines are in.
More from Fly.io
Full history| Started | Incident | Impact | Duration |
|---|---|---|---|
| Sep 2416:31 UTC | State database issues in GRU | major | 2h 40m |
| Sep 2414:51 UTC | Private Networking issues (6PN) | major | 46m |
| Sep 2318:44 UTC | Partial Sprites outage | major | 7h 28m |
| Sep 2314:47 UTC | Elevated private networking errors | major | 36m |
| Sep 2312:42 UTC | Upstash unavailability in FRA region | major | 1h 22m |
| Sep 2306:18 UTC | IPv6 Networking Issues in DFW | major | 5h 57m |
Sources: vendors' own status pages, published postmortems and SEC 8-K Item 1.05 filings, read daily. Times as reported. Logos via logo.dev; trademarks belong to their owners.