AWS Outage Impacting Wave Services
Nov 25, 15:02 UTCNov 25, 23:32 UTC
Duration
8h 31m
Impact
Major
Root cause
Not disclosed
Aqua Security, 90 days
2 incidents
Affected
Cloud Account ScanningReal-Time Events ServiceRemediation ServiceCSPM - Cloud Account ScanningCSPM - Real-Time Events ServiceCSPM - Remediation Service
Final update
AWS services appear to be recovering and at this time, all core Aqua Wave functionality has returned to normal operation.
Timeline
- Resolved · Nov 25, 23:32 UTC
AWS services appear to be recovering and at this time, all core Aqua Wave functionality has returned to normal operation.
- Monitoring · Nov 25, 20:06 UTC
Aqua Wave is continuing to monitor an ongoing incident in AWS. The impact is currently limited to CSPM background scan schedules and the triggering of real-time events from user accounts in us-east-1. We will continue to provide updates when we have new information to share, but at this point, will not publish any new updates unless we receive a status change from AWS or different issues are detected.
- Monitoring · Nov 25, 18:47 UTC
We are continuing to monitor elevated AWS error rates across numerous AWS services. All Aqua Wave APIs are currently functioning as expected. CSPM background scans continue to run at a reduced interval. Technical details: CSPM background scans are normally triggered by CloudWatch Event rules, which are currently not operational due to an outage in AWS us-east-1. However, our failover workflow has been implemented which will continue to trigger these scans at reduced capacity until AWS CloudWatch is fully restored in us-east-1. During this time, you may find that scan reports are delayed. For example, if your scan is scheduled to run hourly, it may run 10-15 minutes after the hour at staggered intervals instead. CSPM Real-Time Events are typically triggered via CloudWatch Event rules deployed in the user environment. You may see delayed or missing events in us-east-1 during the outage window as these rules are unlikely to be triggered. The CSPM Real-Time Events service is still operational and available to receive the events that are sent. AWS CloudTrail is also experiencing a partial outage which may result in delayed or missing delivery of events from source AWS accounts to Aqua Wave.
- Monitoring · Nov 25, 17:08 UTC
We are continuing to monitor increased error rates from AWS in us-east-1. At this time, Aqua Wave's core APIs are not impacted, but background scanning continues to remain affected due to the use of AWS CloudWatch APIs.
- Monitoring · Nov 25, 15:49 UTC
We are continuing to monitor for any further issues.
- Monitoring · Nov 25, 15:40 UTC
We are continuing to monitor elevated AWS error rates affecting CloudWatch, Cognito, Kinesis, and other services (see the AWS status page for more information: https://status.aws.amazon.com/). This is currently impacting Aqua Wave in the following ways: - Sign in via SAML/SSO may be degraded. Standard sign in is not impacted. - Scheduled background scans may be delayed - Scans may produce "new risk" or "unknown" results for impacted services
- Monitoring · Nov 25, 15:22 UTC
The AWS outage is also impacting Cognito, which may affect the ability to sign in using SSO. If you require a break-glass user and do not currently have one, please contact our support.
- Monitoring · Nov 25, 15:02 UTC
Aqua Wave is currently monitoring elevated error rates from some AWS APIs. These errors are preventing Aqua Wave from running complete scans, so we have temporarily disabled scheduled scans.
More from Aqua Security
Full historyAqua Supply Chain Vulnerability Scanning Service Disruption Noticelasted 2h 12mAqua Supply Chain Vulnerability Scanning Service Disruption Noticelasted 0mPartial Service Degradationlasted 10h 51mAqua Image Vulnerability Scaning Service Disruptionlasted 0mLogin to the system is currently unavailablelasted 2h 21mUI Login Issue in us-east-1 – CSPM Outage Impactlasted 19m