Elevated Errors and Query Timeouts for D1 Databases and SQLite Durable Objects

This incident has been resolved.

We are continuing to work on a fix for this issue.

After monitoring today, we still see intermittent elevated errors and query timeouts. Multiple, in-parallel code changes are being implemented or rolling out as additional remediation.

A code change was rolled out and we're monitoring throughout NAMER business hours for reduction in errors and timeouts. Additionally in parallel, we are rolling out more changes that should reduce the impact of these small fraction of errors and timeouts.

One implemented fix is being rolled out across Cloudflare's network but has not yet reached locations running Durable Objects or D1. The release will progress over several more hours after which we will monitor throughout 2025-01-27 for the desired reduction in errors and timeouts.

We are continuing to work on a fix for this issue. Observed errors target only a small number of Durable Objects and D1 databases (D1 errors are less than 0.1% of total requests).

Investigation identified the issue is a recurrence of https://www.cloudflarestatus.com/incidents/kzvk0c2s5fy7, whose remediation did not fully address the elevated errors and query timeouts. We are actively implementing additional fixes to address the errors and timeouts.

We are continuing to investigate this issue.

A small number of users are reporting severe increases in query latency when querying D1 databases and SQLite-backed Durable Objects. This is resulting in timeouts and/or hard errors for those queries. We are investigating the cause and potential fixes and will provide updates with our findings.