Increase in Workflow runs stuck in pending, failed

This incident has been resolved.

We are continuing to work on recovering affected workflow runs that may be stuck in a pending state or have reached a failed state. New workflow runs are not impacted.

We've identified an issue where some workflow runs created between June 30 at 22:24 UTC and July 1 at 16:08 UTC may be stuck in a pending state or have reached a failed state. New workflow runs are not impacted. We are working on a fix for runs stuck in a pending state and will provide additional updates as they become available.