Incident with CodeQL, Webhooks, Notifications, and Slack Integration

This incident has been resolved.

CodeQL has fully recovered. We're continuing to work on recovery for the remaining impacted services.

Webhooks have fully recovered. Continuing to work on recovery for the other services.

Webhooks is operating normally.

We've established that most delays are related to a queuing service and are working to scale out. Early signals from the scale-out are showing signs of recovery for some services. We'll provide an update when services are fully recovered.

Webhooks is experiencing degraded performance. We are continuing to investigate.

We're continuing to investigate issues with CodeQL actions workflows. We're additionally seeing delays for notifications, webhooks, and the Slack integration.

CodeQL actions are currently experiencing delays, which may result in those actions being stuck in a pending state or having failed due to a timeout.

We are investigating reports of degraded performance for CodeQL