← Back to GitHub History
major✓ Resolved

Latency issues across a number of services

GitHub•
Status
resolved
Duration
1h 45m
Updates
10
Coverage
0 articles

INCIDENT TIMELINE

✓ Resolved
This incident has been resolved.
resolved
On July 23, 2026, between 07:08 and 09:39 UTC, several services experienced delays: 8% of actions workflow runs experienced an average run start delay of 10 minutes, 5% of webhook deliveries exceeded SLO, and code scanning, repos, notifications, issues and pull requests experienced increased latency over the life of the incident. <br /><br />The root cause of the incident was a node of our background job processing system which did not recover after entering scheduled host maintenance. The incident was mitigated by identifying the problematic shard and restoring its correct state, after which queue backlogs drained and services recovered. <br /><br />To speed mitigation, we have added monitors for nodes in this unhealthy state after maintenance operations. To prevent future recurrence, we are adapting our lifecycle automation to verify host rejoin after a scheduled reboot.
monitoring
The degradation has been mitigated. We are monitoring to ensure stability.
investigating
Webhooks is operating normally.
investigating
The degradation affecting Pull Requests has been mitigated. We are monitoring to ensure stability.
investigating
We identified the source of latency affecting multiple services and applied a fix. Issues and Actions are recovering, and remaining affected services are seeing improvement as processing backlogs clear. We are actively monitoring recovery across all services.
investigating
The degradation affecting Actions has been mitigated. We are monitoring to ensure stability.
investigating
The degradation affecting Issues has been mitigated. We are monitoring to ensure stability.
investigating
We're currently investigating latency across multiple services. This can show as Actions jobs taking longer to start, Issues search serving stale results, and other listed services being similarly impacted.
investigating
Pull Requests is experiencing degraded performance. We are continuing to investigate.
investigating
We are investigating reports of degraded availability for Actions, Issues and Webhooks

📊 TECHNICAL DETAILS

Internal ID
41f2b84c-1c52-4897-8dfa-6fdfbbb40d32
External ID
zq3c1jst2vkq
🕐 Started
🔄 Last Updated
✓ Resolved
MAJOR
Impact Level
1h 45m
Total Duration
10
Status Updates
Affected Components
WebhooksIssuesPull RequestsActions
REQUEST COVERAGE
No article has been written for this incident yet. When 100 people request coverage, we request generation of one.
0 / 100 requests100 more needed