← Back to LaunchDarkly History
critical✓ Resolved
Web application unavailable and flag delivery evaluations have elevated failure rate
Status
resolved
Duration
1h 9m
Updates
5
Coverage
0 articles
INCIDENT TIMELINE
✓ Resolved
This incident has been resolved.
resolved
All service availability has been fully restored.
Customers using server-side SDKs might need to restart their applications to re-enable the SDKs' connection to LaunchDarkly. Customers using the Relay Proxy may also need to restart their Relay Proxy; in this case, the Relay Proxy should be restarted before restarting their applications that connect to it.
To determine if an application needs to be restarted, customers can look at their application logs. Affected SDKs will emit (depending on language) logs containing one of "giving up permanently", "Invalid SDK key", "unauthorized", or "not authorized", or "401". Services which emitted logs such as these before 9:13AM PT may be unable to receive flag updates and will be unable to send events to LaunchDarkly until they are restarted.
monitoring
Production service impact has been restored. However, customers might need to restart their applications to fully reconnect to LaunchDarkly's services. Our engineering team is working on guidance that can be provided to customers.
identified
The web application availability has recovered. Flag delivery and event ingestion continue to be impacted and are actively recovering.
Our engineering team continues to work on remediation.
identified
We have identified the root cause and are implementing a fix.
investigating
The LaunchDarkly web application (app.launchdarkly.com) was unavailable and is actively recovering at the moment. The Flag Delivery network was also impacted, causing an elevated failure rate with retrieving flag values.
Customers using client-side or server-side SDKs should continue to see the last known flag values if a local cache exists, or fall back to in-code values.
Customers using our Relay Proxy should continue to see last known flag values if a local cache exists.
Customers using our Edge SDKs should continue to see last known flag values.
Some event data loss may have occurred.
📊 TECHNICAL DETAILS
Internal ID
c4ddac06-afec-4d4b-b9cd-51b694851ead
External ID
chywwz01ptb0
🕐 Started
🔄 Last Updated
✓ Resolved
CRITICAL
Impact Level
1h 9m
Total Duration
5
Status Updates
Affected Components
Emails and notificationsFeature workflowsAudit logExperiment managementPast experiment iterationsAuthenticationContext processingRollout managementConfig managementFrontendFlag targetingData ExportIntegrations and webhooksMetricsRollout results processingQuery APIGraphsFlag usage metricsExperiment results processingExpiring Contexts TargetsWeb app (app.launchdarkly.com)Segment managementRollout results analysisObservability Data Ingest (Sessions, Errors)Playgrounds and offline evaluationsExperiment results analysisProgressive rolloutsOpenTelemetry (Logs, Traces, Metrics)AI insightsRelease pipelinesWarehouse native experimentationAccount management
REQUEST COVERAGE
No article has been written for this incident yet. When 100 people request coverage, we request generation of one.
0 / 100 requests100 more needed