US - Service Availability Issue
Jul 24, 11:38 UTCJul 24, 21:28 UTC
Duration
9h 49m
Impact
Major
Root cause
Not disclosed
NinjaOne, 90 days
14 incidents
Affected
United States (US)
Final update
This incident has been resolved.
Timeline
- Resolved · Jul 24, 21:28 UTC
This incident has been resolved.
- Monitoring · Jul 24, 20:10 UTC
Mitigation efforts have shown positive results and we are seeing agents reporting status correctly at this time. We are continuing to monitor the incident.
- Monitoring · Jul 24, 18:21 UTC
To help mitigate some agents still showing offline we are restarting our frontend web systems. Some users may see a slight increase in response time during this process.
- Monitoring · Jul 24, 15:38 UTC
During the AWS Outage, Ninja Agents attempted and failed to reconnect to our backend services. This automatically triggers a backoff/retry mechanism in our agents that is designed to prevent traffic flooding or thundering herd behavior during significant outages. Agents are slowly reconnecting, but will randomize their reconnection attempts until successful. Additionally the service that was responsible for processing device status was struggling with the volume of agent status changes. We have scaled that capacity out and are seeing healthy metrics. Over the last hour we have seen our connected devices increase by over 150k devices, but recovery will likely be slow while the agents attempt to gracefully reestablish service
- Monitoring · Jul 24, 14:54 UTC
We are continuing to monitor for any further issues.
- Monitoring · Jul 24, 13:08 UTC
We have identified a broader connectivity issue within our cloud provider's infrastructure that is causing intermittent service disruptions for some customers. Recovery efforts are ongoing and service stability has improved, though some impact may persist while restoration work continues.
- Identified · Jul 24, 12:10 UTC
We have identified a broader connectivity issue within our cloud provider's infrastructure that is causing intermittent service disruptions for some customers, and we are actively monitoring recovery efforts.
- Investigating · Jul 24, 11:38 UTC
We are currently investigating this issue.
More from NinjaOne
Full historyDevice Local Time Schedules Not Running in Parts of Canadalasted 4d 18hNA/US2: Ninja Support and Billing Ticketslasted 3h 57mSome NinjaOne Agents Reporting Offline - OClasted 48mApps & Books deployment issues on Apple deviceslasted 2h 16mMicrosoft Edge and WebView2 errorslasted 0mNinjaOne Remote issueslasted 59m