Elevated errors across multiple services in IAD1

This incident has been resolved.

We implemented a fix and are monitoring the results. Backlogs for Workflows and Queues will be processed within the next few hours.

New workflow runs and queue messages are being processed now. Processing the backlog of existing workflow runs and queued messages is still being investigated.

We are continuing to work on a fix for this issue.

New messages to Vercel Workflows and Vercel Queues are being queued, but message processing is paused. Queued messages will be processed when service is restored.

Traffic to the IAD1 region has been re-routed to nearby regions. Functions configured in IAD1 will be invoked in a different region if failover regions are configured, but if no other regions are configured, the function will still be invoked in IAD1.

Some Vercel functions that run in the IAD1 region are experiencing elevated invocation failures. We are investigating the issue and will share more information as it becomes available.