Elevated latencies for network change propagation

This incident has been resolved.

Between 7:08 AM and 10:48 AM PST, we experienced elevated latencies for network change propagation in the EU-WEST-1 Region. This was due to a delay in propagation of configuration updates on a sub-system which supports external and cross account connectivity. Customers experienced delays when attempting to connect to newly launched instances using their assigned public IPs, as well as delays in establishing new VPC Peering connections. Other AWS Services were also impacted by this issue and were impacted by delayed or stuck workflows such as pulling container images. Engineers were automatically engaged at 6:29 AM, prior to customer impact beginning. At 7:20 AM, root cause was identified. By 9:20 AM we began observing early signs of recovery. Our mitigation efforts completed at 10:43 AM and full recovery was observed at 10:48 AM. We recommend customers retry any failed requests. The issue has been resolved and all services are operating normally.

We are seeing significant signs of recovery. We continue to work toward full resolution.

We continue to work toward fully mitigating the issue that is resulting in elevated latencies for network change propagation delays in the EU-WEST-1 Region. We have a high degree of confidence that our current mitigation efforts will fully mitigate the issue. Our current estimation is that these mitigations will complete within the next 60 to 120 minutes. We recommend customers continue to retry failed requests. We will provide additional information as recovery continues to progress.

We are seeing early signs of progressive recovery. We continue to work toward full recovery and will continue to provide updates as we work toward full mitigation. We recommend customers continue to retry failed requests. As recovery progresses, additional requests will succeed.

We can confirm elevated latencies for network change propagation in the EU-WEST-1 Region. This causes delays when attempting to connect to newly launched instances using their assigned public IPs, as well as delays in establishing new VPC Peering connections. Other AWS Services are also impacted by this issue and may be impacted by delayed or stuck workflows such as pulling container images. Engineers were automatically engaged and we are actively working on mitigating this issue. We plan to provide an update within the next 60 minutes.

We are investigating elevated latencies for network change propagation in the EU-WEST-1 Region. This is resulting in impact to other services, such as timeouts when pulling images. We will provide you with further information shortly.