Contact webhook events are delayed in delivery

This incident has been resolved.

We have resolved the underlying issue and service has been resumed.

We have processed more than 66% of the backlog at this point. Within the next hour, the backlog will be fully processed, and this incident will be marked as resolved.

We are still monitoring the progress on processing the backlog. The impact has been reduced from a peak 3-hour delay in delivering a contact webhook to 2h 20 min. The delay is being reduced further as we process the backlog. Within 2-3 hours, the backlog will be completely processed.

A fix has been deployed to the underlying infrastructure and we can now confirm we're processing the backlog with the correct volume. Delayed webhooks are now being delivered. We will post another update in 30 minutes to share more about the progress on processing the backlog.

We have identified the issue and are working on mitigating its effects. The volume of contact webhooks processed over time was affected by an infrastructure change, and now we have an increasing backlog of contact webhooks to process. The team is working on increasing the speed of processing the backlog. No contact webhooks were lost.

We are currently investigating increased latency in contact webhook delivery. We have noticed a delay of approximately 50 minutes in sending webhooks related to contact events. The team is investigating and mitigating this processing delay issue. Contact management via dashbaord and its API are not affected.