Project Actions Failing Across Multiple Regions

This incident has been resolved.

The mitigation has returned project activity to normal performance. Project actions that failed during the outage are being requeued to run. We will continue to monitor to ensure project functionality.

We have identified a problem with a specific job in us-east-1, we have mitigated the job and are observing the results.

We are testing mitigation options to attempt to restore project action performance.

We have determined that there are four main project actions impacted in the identified regions: Pause/unpause failures, Restore/PITR failures, PITR configuration failures and Backup problems. We are pursuing root cause and mitigation options in parallel.

Project actions in us-east-1, us-east-2, sa-east-1, ca-central-1 are experiencing significant increases in error rates. We are exploring the best options for restoring performance.

Certain project actions are failing. We are actively investigating to determine the regions impacted and source of the issue.