Jobs Queueing

Incident Report for CircleCI

Resolved

Build start times for all executor types have returned to normal. We are continuing to monitor the situation for reoccurrence.
Posted Mar 27, 2020 - 21:48 UTC

Monitoring

We are continuing to monitor the queue for macOS builds.
Posted Mar 27, 2020 - 21:43 UTC

Update

The Incident has been resolved for the Docker and Machine executors. We are monitoring the queue for macOS builds.
Posted Mar 27, 2020 - 21:38 UTC

Update

macOS builds have begun running however we are seeing degraded performance as the queue is processing.

We are seeing degraded performance with Google Cloud Platform Google Compute Engine.
Posted Mar 27, 2020 - 21:02 UTC

Update

We are continuing to work on a fix for this issue.
Posted Mar 27, 2020 - 20:48 UTC

Update

We are continuing to work on a fix for this issue.
Posted Mar 27, 2020 - 20:03 UTC

Update

We are continuing to work towards a fix for the inability to start macOS builds.
Posted Mar 27, 2020 - 19:34 UTC

Update

We are continuing to work on a fix for this issue.
Posted Mar 27, 2020 - 19:07 UTC

Update

We are continuing to work on a fix for this issue.
Posted Mar 27, 2020 - 18:38 UTC

Update

We are continuing to work on a fix for this issue.
Posted Mar 27, 2020 - 18:12 UTC

Update

We are continuing to work on a fix for this issue.
Posted Mar 27, 2020 - 17:40 UTC

Update

We have disabled a replica sync process that appeared to be placing extra load on Mongo.
Posted Mar 27, 2020 - 17:13 UTC

Update

macOS builds are running however builds may queue over the next few hours as we process existing builds.
Posted Mar 27, 2020 - 16:52 UTC

Update

We are continuing to work on a fix for this issue.
Posted Mar 27, 2020 - 16:47 UTC

Update

We are continuing to work on a fix for this issue.
Posted Mar 27, 2020 - 16:40 UTC

Identified

The issue has been identified and a fix is being implemented.
Posted Mar 27, 2020 - 16:24 UTC

Investigating

We are investigating the issue, the impact is that some jobs are being impacted across all build types.
Posted Mar 27, 2020 - 16:21 UTC
This incident affected: Docker Jobs, Machine Jobs, macOS Jobs, Pipelines & Workflows and CircleCI Dependencies (Google Cloud Platform Google Compute Engine).