← Back to GitHub History
major✓ Resolved

Incident with Webhooks and Actions

GitHub•
Status
resolved
Duration
1h 50m
Updates
8
Coverage
0 articles

INCIDENT TIMELINE

✓ Resolved
This incident has been resolved.
resolved
On November 9, 2026, from 09:23 to 11:40 UTC, one of our job queue clusters was degraded, delaying webhook deliveries, Actions workflow runs, and Dependabot and dependency graph updates. Until mitigation at 10:37 UTC, an average of 26% of webhook deliveries were delayed, peaking at 30%, and the backlog cleared by 11:40 UTC. Some deliveries were delayed by up to 3.5 hours, and delays for a small share began as early as 01:00 UTC. Actions skips events older than 30 minutes, so some workflow runs did not start. <br /><br />The incident was caused by a degraded host that had slowed one of the cluster's Kafka brokers since November 7. The broker stopped reporting the metrics our alerts rely on, so we did not identify it until it stalled the cluster on November 9. <br /><br />We mitigated the incident by removing the broker from service. We have since added an alert that pages us within 30 minutes when a host slows a broker this way, so we can remove the broker before it affects customers.
monitoring
The degradation affecting Webhooks has been mitigated. We are monitoring to ensure stability.
investigating
Webhooks is experiencing degraded performance. We are continuing to investigate.
investigating
Webhooks is experiencing degraded availability. We are continuing to investigate.
investigating
The degradation affecting Actions has been mitigated. We are monitoring to ensure stability.
investigating
We've applied a mitigation and are monitoring for recovery.
investigating
We're investigating delays in Actions runs start times and webhook deliveries.
investigating
We are investigating reports of degraded performance for Actions and Webhooks

📊 TECHNICAL DETAILS

Internal ID
40376b51-486e-4520-9972-249d7f321107
External ID
xxb3l2bm76dr
🕐 Started
🔄 Last Updated
✓ Resolved
MAJOR
Impact Level
1h 50m
Total Duration
8
Status Updates
Affected Components
WebhooksActions
REQUEST COVERAGE
No article has been written for this incident yet. When 100 people request coverage, we request generation of one.
0 / 100 requests100 more needed