← Back to Fly.io History
major✓ Resolved

Upstash unavailability in FRA region

Fly.io•
Status
resolved
Duration
56m
Updates
4
Coverage
0 articles

INCIDENT TIMELINE

✓ Resolved
This incident has been resolved.
resolved
**Impact** From about 12:00 to 13:05 UTC, some clients got connection timeouts to their Redis databases. Affected were databases hosted in fra or gig, and clients whose connections were routed through fra or gig, even if their database is hosted in another region. **What happened** During a planned rolling upgrade, replicas in fra and gig did not finish draining and were left out of service. They were returned to Fly routing before they were ready to accept connections. Connections routed to them then timed out. **What we're changing** • We will be adding tooling to return a replica to service safely after maintenance. • We will be adding safety checks to our upgrade process. • We will make the upgrade procedure more resilient to errors like this one.
resolved
This incident has been resolved.
monitoring
A fix has been implemented and we are monitoring the results.
investigating
We are investigating an issue affecting Upstash services in the fra region. Customers may experience connection failures or service unavailability. We are working with Upstash to identify the cause and restore service. We’ll provide an update as soon as we have more information.

📊 TECHNICAL DETAILS

Internal ID
0b1c69c2-84c5-43cf-8198-0b9073ea8b20
External ID
01M376272WGYT2F3BJ9ZK1WDYB
🕐 First reported
🔄 Last Updated
✓ Resolved
MAJOR
Impact Level
56m
Total Duration
4
Status Updates
REQUEST COVERAGE
No article has been written for this incident yet. When 100 people request coverage, we request generation of one.
0 / 100 requests100 more needed