Delays in Workflows Starting
Incident Report for CircleCI
Postmortem

Monday, November 23, 2020 update on November 18 incidents

Beginning on the morning of November 18 and throughout the day our platform experienced a series of issues that impacted our ability to run customer workflows, and in some cases, made the UI unavailable. It is important to note that we do not believe any customer jobs were lost as a result of this issue. All jobs were eventually run, albeit with a significant delay in some instances.

We value our customers tremendously and apologize for any disruption you may have experienced in your work. We plan to publish a full incident report when we have completed our incident review later this week. We are confident in the root cause of these incidents. As a first step, we have made monitoring changes to ensure they do not happen again, and will share more about the additional steps we are taking to address these issues. Thank you for your support and patience.

Posted Nov 23, 2020 - 21:08 UTC

Resolved
After monitoring we haven't had any further instances of any issues, we are moving this to resolved.
Posted Nov 18, 2020 - 20:34 UTC
Update
We are continuing to monitor to ensure no further issues arise.
Posted Nov 18, 2020 - 19:46 UTC
Update
Our systems have recovered. We will continue to monitor this for an extended period of time.
Posted Nov 18, 2020 - 17:55 UTC
Monitoring
Our systems are recovering and any delayed workflows will run as expected. We will continue to monitor the recovery.
Posted Nov 18, 2020 - 17:25 UTC
Investigating
We are currently investigating an issue causing delays in workflows starting.
Posted Nov 18, 2020 - 17:07 UTC
This incident affected: Pipelines & Workflows.