Who is down right now

18 down now · 221 this week

Aqua Security · Cloud and hostingNov 25, 2020, 15:02 UTC 5 years ago

AWS Outage Impacting Wave Services

MajorNot disclosedUpdated 2h ago
Nov 25, 15:02 UTCNov 25, 23:32 UTC
Duration
8h 31m
Impact
Major
Root cause
Not disclosed
Aqua Security, 90 days
2 incidents
Affected
Cloud Account ScanningReal-Time Events ServiceRemediation ServiceCSPM - Cloud Account ScanningCSPM - Real-Time Events ServiceCSPM - Remediation Service
Status page

Final update

AWS services appear to be recovering and at this time, all core Aqua Wave functionality has returned to normal operation.

Timeline

  1. Resolved · Nov 25, 23:32 UTC
    AWS services appear to be recovering and at this time, all core Aqua Wave functionality has returned to normal operation.
  2. Monitoring · Nov 25, 20:06 UTC
    Aqua Wave is continuing to monitor an ongoing incident in AWS. The impact is currently limited to CSPM background scan schedules and the triggering of real-time events from user accounts in us-east-1. We will continue to provide updates when we have new information to share, but at this point, will not publish any new updates unless we receive a status change from AWS or different issues are detected.
  3. Monitoring · Nov 25, 18:47 UTC
    We are continuing to monitor elevated AWS error rates across numerous AWS services. All Aqua Wave APIs are currently functioning as expected. CSPM background scans continue to run at a reduced interval. Technical details: CSPM background scans are normally triggered by CloudWatch Event rules, which are currently not operational due to an outage in AWS us-east-1. However, our failover workflow has been implemented which will continue to trigger these scans at reduced capacity until AWS CloudWatch is fully restored in us-east-1. During this time, you may find that scan reports are delayed. For example, if your scan is scheduled to run hourly, it may run 10-15 minutes after the hour at staggered intervals instead. CSPM Real-Time Events are typically triggered via CloudWatch Event rules deployed in the user environment. You may see delayed or missing events in us-east-1 during the outage window as these rules are unlikely to be triggered. The CSPM Real-Time Events service is still operational and available to receive the events that are sent. AWS CloudTrail is also experiencing a partial outage which may result in delayed or missing delivery of events from source AWS accounts to Aqua Wave.
  4. Monitoring · Nov 25, 17:08 UTC
    We are continuing to monitor increased error rates from AWS in us-east-1. At this time, Aqua Wave's core APIs are not impacted, but background scanning continues to remain affected due to the use of AWS CloudWatch APIs.
  5. Monitoring · Nov 25, 15:49 UTC
    We are continuing to monitor for any further issues.
  6. Monitoring · Nov 25, 15:40 UTC
    We are continuing to monitor elevated AWS error rates affecting CloudWatch, Cognito, Kinesis, and other services (see the AWS status page for more information: https://status.aws.amazon.com/). This is currently impacting Aqua Wave in the following ways: - Sign in via SAML/SSO may be degraded. Standard sign in is not impacted. - Scheduled background scans may be delayed - Scans may produce "new risk" or "unknown" results for impacted services
  7. Monitoring · Nov 25, 15:22 UTC
    The AWS outage is also impacting Cognito, which may affect the ability to sign in using SSO. If you require a break-glass user and do not currently have one, please contact our support.
  8. Monitoring · Nov 25, 15:02 UTC
    Aqua Wave is currently monitoring elevated error rates from some AWS APIs. These errors are preventing Aqua Wave from running complete scans, so we have temporarily disabled scheduled scans.

More from Aqua Security

Full history