Resolved Service Degradation

This incident has been resolved.

Okta is currently investigating an issue preventing some customers from proceeding past the API enablement step when configuring AWS in Okta. We are actively investigating and will update this message with more information as soon as we have it. Root Cause Analysis: On February 27th at approximately 8:15 am PST, Okta received a customer report of an error in US Cell 1 when attempting to add an additional AWS account. Upon investigation, Okta found that customers in all cells would experience errors if attempting to add child AWS accounts or modify existing AWS integration configurations. In order to prevent further issues to customer configurations, Okta immediately made all AWS app instance configurations read-only. This change was rolled out to all cells by 1:41 pm PST until the root cause of the authentication issues could be identified and resolved. Further investigation determined that the errors were the result of a recent upgrade to the AWS SDK which conflicted with Okta's existing logic for resolving STS endpoints. In the majority of configurations, Okta's logic would previously default to the Global STS endpoint, however after the upgrade the same resolution process began always returning the lexicographically first endpoint, which was the STS endpoint for the Asia Pacific region (Hong Kong). Because the Version 1 STS tokens that Okta generates are only valid in regions which are enabled by default, and Asia Pacific is not enabled by default, Okta's subsequent AssumeRole calls to that region were failing. Though the SDK upgrade was tested in some environments, the affected configuration workflow had not been incorporated into any automated or manual end-to-end testing flows; this service interruption highlighted a gap in our regression test plan within the AWS application integration. Once Okta determined the cause of the errors, a fix was implemented to route all STS calls to the global endpoint. This fix was deployed to US Cell 1 at 8:49 pm PST and then to all cells. By February 28 at 9:01 pm PST, the update was completed and service was fully restored. Okta is currently working to update how endpoints and regions are passed to the AWS SDK to increase efficiency and reduce latency. As part of our on-going focus on reliability, Okta is also actively working to identify and audit test cases where application integrations and core infrastructure rely on common elements to prevent issues like this from occurring in the future.