← Back to GitHub History
minor✓ Resolved

Delays in commit processing

GitHub•
Status
resolved
Duration
1h 1m
Updates
4
Coverage
0 articles

INCIDENT TIMELINE

✓ Resolved
This incident has been resolved.
resolved
On September 1, 2026, between approximately 14:01 and 16:01 UTC, updates in response to pushes were delayed, temporarily showing stale diffs. The median time to refresh a diff after a push rose from the normal level of about 3 seconds to over 2 minutes at the peak, and more than 140,000 customer accounts had at least one delayed refresh during the most affected 75 minutes. Pushing commits and opening pull requests continued to work normally. The incident was caused by a sharp, concentrated surge in push volume that saturated worker pools and job queueing infrastructure. Autoscaling did not increase capacity as intended, so the backlog did not clear on its own. <br /><br />The incident was mitigated by manually scaling the affected worker pools and increasing push-processing capacity. This allowed the system to process the backlog, after which refresh times returned to normal. To reduce the likelihood and impact of similar incidents, we are adding quotas and throttling earlier in the push path so a single concentrated source of load cannot saturate shared capacity, improving worker-pool autoscaling so capacity is added automatically, and improving monitors for background job processing so on-call is paged before customers experience delayed pull request updates.
investigating
Time to update pull request diffs have improved to normal thresholds.
investigating
Diffs in the PR view may be stale for several minutes. We are investigating and scaling up resources.
investigating
We are investigating reports of degraded performance for Pull Requests

📊 TECHNICAL DETAILS

Internal ID
cad4971c-5d37-49a1-b328-b2eae160a3f7
External ID
jk2dy5h9yp3m
🕐 Started
🔄 Last Updated
✓ Resolved
MINOR
Impact Level
1h 1m
Total Duration
4
Status Updates
Affected Components
Pull Requests
REQUEST COVERAGE
No article has been written for this incident yet. When 100 people request coverage, we request generation of one.
0 / 100 requests100 more needed