← Back to Fly.io History
major✓ Resolved

Delayed Metrics

Fly.io•
Status
resolved
Duration
1d 21h
Updates
11
Coverage
0 articles

INCIDENT TIMELINE

✓ Resolved
This incident has been resolved.
identified
Almost all metrics have caught up aside from a small handful in `sin` and `syd`. We're continuing to monitor and expect these to complete in the next few hours.
identified
Backlogged metrics are still being processed. We're bringing extra processing capacity online to speed up the process.
identified
Backlogged metrics are still being processed and ingestion delays persist for some customers
identified
Backlogged metrics are still being processed and ingestion delays persist for some customers, but we’re continuing to see gradual improvement as the backlog continues to drain.
identified
Backlogged metrics data is still being processed, continue to monitor the situation while the system catches up.
identified
Metric ingestion is still delayed for some customers. We’re seeing gradual improvement and continue to monitor the situation while the system catches up.
identified
We are still in the process of increasing the metrics cluster throughput to catch up with metric backlog.
identified
We are continuing to see delayed metrics due to resource contention on a subset of metrics ingestion hosts, and we are working to rebalance ingestion traffic and reduce the backlog.
identified
We are continuing to see delayed metric exports from a number of hosts. Users will see delayed or missing metrics for machines on impacted hosts at this time.
identified
We've identified an issue on multiple hosts causing delayed metric ingestion into our hosted fly-metrics.net dashboards. We are working on a fix.
investigating
We are investigating issues with customer facing metrics in the fly-metrics.net dashboard. Users may see delayed or missing metrics at this time.

📊 TECHNICAL DETAILS

Internal ID
f22ac0e0-e7b7-4775-aa23-3c1957bd50ca
External ID
pd277p5pf2tf
🕐 First reported
🔄 Last Updated
✓ Resolved
MAJOR
Impact Level
1d 21h
Total Duration
11
Status Updates
Affected Components
Metrics
REQUEST COVERAGE
No article has been written for this incident yet. When 100 people request coverage, we request generation of one.
0 / 100 requests100 more needed