Trello - degraded performance

This incident has been resolved.

Our team is still working to fix minor data inconsistencies on cards that were moved between boards during the incident. The following are still being addressed: -Missing card labels -Attachment restrictions are not being enforced on moved cards -Checklist assignee information on some cards. -Incorrect due date reminders sent to some card assignees Your cards are safe to edit. If you make changes to a card before our automated fix reaches it, your edits will be preserved, and the automated restore for that card will be skipped. No data will be lost in either scenario. We will provide another update once the above items are fully resolved. If you need assistance in the meantime, please reach out to our support team.

The team continues to remediate potential outstanding data issues that occurred on Boards and Cards during the incident

Users should be experiencing restored functionality to their boards and cards. Our team has resolved 'card not found' errors, and continues to remediate potential outstanding data issues that occurred on Boards and Cards during the incident: - Missing card labels - Attachment restrictions not being enforced on moved cards - Checklist assignees that are not a member of the current board may incorrectly linger - Card assignees that are not a member of the current board may have incorrectly received due date reminders Please note that while cards are safe to edit, they may be excluded from certain restoration work if they are edited. We will provide another update when above issues are resolved. In the meantime, please reach out to our support team if you need further assistance.

The team continues to remediate potential outstanding data issues that occurred on Boards and Cards during the incident (details are in yesterday's update). Users should be experiencing restored functionality to their boards and cards. Please note that while cards are safe to edit, they may be excluded from certain restoration work if they are edited. We greatly appreciate your patience as we work through these issues. Please reach out to our support team if you need further assistance.

Users should be experiencing restored functionality to their boards and cards. The team continues to remediate potential outstanding data issues that occurred on Boards and Cards during the incident. Below is a list of remaining issues that may still be occurring, but we are currently working to resolve via processing on our end. These issues were caused by card moves between boards during the incident. - Missing card labels - Attachment restrictions not being enforced on moved cards - Checklist assignees that are not a member of the current board may incorrectly linger - Card assignees that are not a member of the current board may have incorrectly received due date reminders Please note that cards are safe to edit, but they may be excluded from certain restoration work if they are edited.

We greatly appreciate your patience as we work through these issues. We will provide another update when above issues are resolved. In the meantime, please reach out to our support team if you need further assistance.

Users should be experiencing restored functionality to their boards and cards. We are running scripts to replay actions that occurred during the incident. We will update once these are complete.

Index rebuild is complete and functionality should be restored for users. We are aware that cards moved by users during the incident may be in an incomplete state and we are investigating solutions. We have no indication of data loss during the incident. We will continue to monitor and will provide the next updates in 4 hours or sooner.

We are continuing to monitor the index rebuild process across all affected shards and will provide updates as each phase completes. We will provide next update in eight hours or sooner as we make progress.

We want to provide more transparency regarding this incident given its duration. We were optimizing one of our largest databases, specifically the one storing comments and card activity. At the end of this process, a bug caused some database partitions to go live without their indexes. This means queries to those partitions are timing out, which is why approximately 40 percent of boards are experiencing failures. Your data is safe. Any comments added during this incident are being saved and will reappear once the process is complete. We are currently rebuilding the missing indexes. We have allocated significant compute resources to this task to move as quickly as possible, but the data volume is enormous. We estimate it will take approximately 20 more hours to reach full resolution. Most impacted features include, but are not limited to: - Comments (viewing, adding, notifications) - Activity views on cards - Attachments, Power-Ups, board exports - Home page / 'Updates' view - Search - Some cards showing 'Card Not Found' - Moving/copying cards or lists Please avoid moving or copying cards between boards for now. If your board is working normally, you can continue your work, as most boards remain unaffected. We know Trello is where your work lives and we are committed to getting everything back to normal. We will provide next update in eight hours or sooner as we make progress.

Our teams continue to work diligently on the mitigation. During this time, we recommend that you do not move or copy cards or lists between boards. For users whose boards are affected by this issue, you may create and delete cards and edit card descriptions as usual, but you will not be able to see comments or the detailed activity log of the cards. We estimate that the rebuild will take another 24 hours to complete. We will update this page as we have more information.

Trello continues to experience slowdowns and failures that are affecting comment operations, including creating and deleting comments. Our teams continue to work on the issue resolution, while it is taking longer than expected. We will provide more updates within the next 8 hours or sooner once we have further information to share.

Trello is experiencing slowdowns and failures affecting comment operations, including creating and deleting comments. Users may also experience delays when loading activity views on cards, viewing attachments, using Power-Ups, exporting boards and 'Updates' on home page. Our team is actively working on the resolution and will likely take up to 7 hours to restore the services back to full functionality. We will provide further update at this time, or earlier if we have additional information to share prior to this resolution.

Trello is experiencing slowdowns and failures affecting comment operations, including creating and deleting comments. Users may also experience delays when loading activity views on cards, viewing attachments, using Power-Ups, or exporting boards. Some users are encountering errors when viewing specific cards. The team is working hard to restore full functionality and will provide another update within 3h.

Work by the Trello engineering team continues to restore functionality to comments and activity views. Recovery of the affected systems is taking longer than anticipated. Next update will be posted within 4h or on a significant change. Thank you for your patience.

Work by the Trello engineering team continues to restore functionality to comments and activity views. Next update will be posted within 4h or on a significant change. Thank you for your patience.

The team has identified the root cause and is hard at working resolving this issue. Users will see problems creating/updating/deleting comments and loading the activity view for cards at this time. The rest of Trello's features should be working as normal. We will provide more updates within 2h.

Users may be experiencing slowdowns across various functions of Trello including creating/deleting comments, editing cards, or creating new attachments. The team is actively working to fix the issue and will provide more updates within the hour.