← Back to GitHub History
critical✓ Resolved

Incident with Pull Requests

GitHub•
Status
resolved
Duration
46m
Updates
6
Coverage
0 articles

INCIDENT TIMELINE

✓ Resolved
This incident has been resolved.
resolved
Between July 24, 19:17 UTC and July 24, 20:02 UTC, users were unable to create pull requests due to a database schema change. In total, 113,930 pull request creation attempts were impacted across 50,904 users, with an average error rate of 1.75% and a maximum error rate of 2.25% for all requests to Pull Requests service. Existing pull requests and other GitHub functionality were not affected. The issue was resolved by reverting the change to the affected database, upon which pull request creation immediately resumed.<br /><br />The root cause was related to a backfill workflow into the Vitess keyspace hosting Pull Request data. The backfill Vitess command encountered errors and increased VReplication lag, and the workflow was canceled at 19:17 UTC. The cancellation executed a misunderstood Vitess codepath that dropped the backing table to the target keyspace, leaving a non-existent reference that resulted in errors creating Pull Requests. The mitigation was executing a command to drop the vschema reference to the dropped table, allowing Pull Request creation to resume.<br /><br />We are adding stronger pre-flight validation to our tooling to prevent similar issues and expanding lower-environment support to provide better test coverage end-to-end before promoting them to production. We're also fixing our backfill migration tooling to protect from this specific codepath.
monitoring
The degradation affecting Pull Requests has been mitigated. We are monitoring to ensure stability.
investigating
We have applied a mitigation and are monitoring for recovery
investigating
Pull Requests is experiencing degraded availability. We are continuing to investigate.
investigating
We are investigating errors creating pull requests
investigating
We are investigating reports of degraded performance for Pull Requests

📊 TECHNICAL DETAILS

Internal ID
9cec216d-a6f6-4523-ad37-b04a6a37db86
External ID
jxd617hfwfq8
🕐 Started
🔄 Last Updated
✓ Resolved
CRITICAL
Impact Level
46m
Total Duration
6
Status Updates
Affected Components
Pull Requests
REQUEST COVERAGE
No article has been written for this incident yet. When 100 people request coverage, we request generation of one.
0 / 100 requests100 more needed