Elevated Management API error rates across all regions

This incident has been resolved.

We've seen error rates return to normal levels, and all regions are now holding steady. We will continue to monitor to make sure things remain stable.

We are continuing to see some periodic spikes in error rates; however, total rate of errors are decreasing across most regions. We're continuing efforts to get all systems back to green. When error rates spike, management API functions, which would mean dashboard actions, project creation, configuration changes, and other functions remain impacted. Existing projects are unaffected and connectivity via the Data APIs, Postgres connections, and Supavisor are unaffected.

The team is continuing to deploy mitigations, and we are seeing error rates improve, but are not back to baseline yet. All management API functions, which would mean dashboard actions, project creation, configuration changes, and other functions remain impacted. Existing projects are unaffected and connectivity via the Data APIs, Postgres connections, and Supavisor are unaffected.

We have identified the source of the increased error rates, and are working on a fix now. All management API functions, which would mean dashboard actions, project creation, configuration changes, and other functions remain impacted. Existing projects are unaffected and connectivity via the Data APIs, Postgres connections, and Supavisor are unaffected.

We've expanded this to all management API functions, which would mean dashboard actions, project creation, configuration changes, and other functions would also be impacted. Error rates are currently improving, but the team is investigating the root cause to ensure continued stability. Existing projects are unaffected and connectivity via the Data APIs, Postgres connections, and Supavisor are unaffected.

We are currently investigating the cause, but users may see issues creating projects across all regions. Existing projects are not affected and remain up and stable.