Cluster unavailability and degraded cluster operations in AWS me-central and me-south-1 regions.

This incident has been resolved.

AWS me-central-1 region is impacted with two AZs out. Clusters in that region are currently unavailable. AWS me-south-1 region is also experiencing a single AZ outage, which causes degraded performance of clusters in that region. While no impact has been observed on GCP or Azure in those regions, customers with single-region deployments are advised to have their disaster recovery plans ready should the need to restore from a backup to another region be required. We are closely tracking AWS's status. Impacted customers can track AWS status here https://health.aws.amazon.com/health/status.

me-central-1 region is impacted more with two AZs out and the clusters in that region are unavailable. In addition to this, me-south-1 region is also experiencing an AZ outage, which causes degraded performance of clusters in me-south-1. We are closely tracking AWS's status. Impacted customers can track AWS status here https://health.aws.amazon.com/health/status

AWS continues to work on mitigation efforts, including stabilizing impacted APIs and implementing interim workarounds. Our teams are actively reviewing and validating these mitigations; however, this effort will continue into tomorrow. We will provide our next update at 7:00 AM PST, or sooner if additional information becomes available.

AWS is reporting significant signs of recovery for specific APIs and requests, but several still remain to be mitigated and will likely take several hours more to complete, if not into tomorrow. Our teams have begun testing with this update but have reported we are still impacted at this time. We will continue to monitor the situation closely and will provide one more update this evening by 7PM PST, or sooner if we have additional information to share.

AWS is reporting positive signs of recovery at this time, but are still working on multiple mitigation paths to fully restore services. We continue to keep a close eye on the situation and will provide another update by 6PM PST, or sooner if we have additional information to share.

Current conflict in the region has led to objects impacting the AWS data center creating a fire earlier today. Power was shut down to prevent further damage. AWS is working on multiple mitigation paths to restore services as quickly as possible. We will provide the next update by 4PM PST or sooner if we have more to share.

A power outage at an AWS facility in the me-central-1 (Mid-East) region may cause failures or slower than usual progress on cluster operations. AWS is aware of the issue and is working to restore power. We are monitoring the situation, and will provide updates as we receive them from AWS.