Service Degradation Affecting Sign In, SSO, MFA

This incident has been resolved.

# Incident Postmortem - Cloud Services Degraded **Date of Incident:** 2025-11-18 **Time of Incident \(UTC\):** 5:03pm UTC - 6:05pm UTC **Service\(s\) Affected:** SSO, sign in, sign up, CLI, web interface, access to vault content and other items, admin console, MFA **Impact Duration:** ~60 mins ## Summary On November 18, 2025, at 5:03 PM UTC, 1Password experienced degraded and temporarily unavailable cloud services for customers in the US region. The issue was caused by database resource exhaustion, causing operations to fail and connections to be rejected. This was not a security incident and no customer data was impacted. The issue was resolved by resizing the database to restore normal performance and ensure additional capacity for future growth. ## Impact on Customers * **Single Sign-on \(SSO\), Multi-factor Authentication \(MFA\):** Users with SSO or MFA enabled experienced delays, and in some cases failures to log in. * **Browser Extension:** Users who needed to authenticate via the web interface were unable to unlock their vaults. * **Web Interface, Administration**: Customers were unable to log in, sign-ups failed, syncing between devices was not functioning, access to vault and other items were unavailable and the admin console was not reachable. * **API Access:** CLI users and API requests received timeout errors and slow responses. * **Number of Affected Customers \(approximate\):** All customers utilizing cloud interfaces and APIs in the affected region for the duration of the incident. * **Geographic Regions Affected \(if applicable\):** US/Global. ## What Happened? * **Timeline of Events \(UTC\):** * 2025-11-18 4:59pm: Automated monitoring detects increased errors * 2025-11-18 5:03pm: Team began investigating * 2025-11-18 5:09pm: Servers restarted, service is still degraded * 2025-11-18 5:23pm: Public status page updated to Investigating and services Degraded * 2025-11-18 5:25pm: Servers scaled down to reduce database load * 2025-11-18 5:41pm: Database instance size upgrade started * 2025-11-18 5:44pm: Potentially problematic cron job disabled * 2025-11-18 5:56pm: Services slowly started to scale up * 2025-11-18 5:57pm: Services started to come back as the database instance resize completes * 2025-11-18 6:05pm: Incident marked as Identified * 2025-11-18 6:05pm: Team continues to monitor, performance has returned to normal levels * 2025-11-18 6:23pm: Incident marked as Monitoring and services Operational * 2025-11-18 7:16pm: Incident marked as resolved * **Root Cause Analysis:** The refactor of a feature increased the impact of a poorly performing query that had previously gone undetected. The result was the exponential increase in resource consumption for the main database. Once resources were fully exhausted, the service rejected connections and all requests failed. * **Contributing Factors \(if any\):** * Non-performant queries * Database under-provisioned ## How Was It Resolved? * **Mitigation Steps:** * Background services were halted to reduce load on the database. * Application servers were scaled down to further reduce load. * **Resolution Steps:** Increasing the database instance size resolved the issue. * **Verification of Resolution:** Monitoring metrics were closely observed to ensure error rates returned to normal and database performance had stabilized. ## What We Are Doing to Prevent Future Incidents * **Improving monitoring:** We are updating our monitoring systems to better detect database issues like this before impacting customers. * **Improve database performance:** We are refactoring the responsible query to improve performance and reduce load, and tuning the background service to prevent resource contention. ## Next Steps and Communication No action is required from our customers at this time. We are committed to providing a reliable and stable service, and we are taking the necessary steps to learn from this event and prevent it from happening again. Thank you for your understanding. Sincerely, The 1Password Team

This incident has been resolved.

We implemented a mitigation and service is restored. We will continue to closely monitor performance to ensure the issue is fully resolved.

We identified the issue, the engineering team is working on a mitigation and we're starting to see recovery.

We are continuing to investigate this issue.

We are continuing to investigate this issue.

We are continuing to investigate this issue.

We are actively investigating an issue causing significant service degradation across 1Password impacting sign in, SSO, MFA, and accessing items.