Long provisioning times for VM and remote docker jobs

Incident Report for CircleCI

Resolved

We are marking the incident as resolved but realize that macOS builds will take a few hours to process the current backlog, as such that component will remain as degraded.
Posted Sep 17, 2018 - 22:23 UTC

Update

We are continuing to monitor for any further issues.
Posted Sep 17, 2018 - 22:08 UTC

Monitoring

We are seeing active provisioning of new VMs and the backlog is draining. We will continue to monitor the situation and update again in 30 minutes.
Posted Sep 17, 2018 - 21:55 UTC

Identified

We are scaling capacity to meet current recovery demands, some queueing is anticipated. We will update in 30 minutes.
Posted Sep 17, 2018 - 21:44 UTC

Monitoring

All jobs are executing normally. We will continue to monitor the stability and performance of the platform, and provide an update within 40 minutes.
Posted Sep 17, 2018 - 21:09 UTC

Update

VM, macOS 2.0 and remote docker jobs are not executing. Our engineers are working to mitigate the issue. We will provide an update within 20 minutes.
Posted Sep 17, 2018 - 20:53 UTC

Update

VM and remote docker jobs are not executing. Our engineers are working to mitigate the issue. We will provide an update within 20 minutes.
Posted Sep 17, 2018 - 20:33 UTC

Identified

A regression has been identified and we are rolling back to a known good version. We will provide an update within 20 minutes.
Posted Sep 17, 2018 - 20:11 UTC

Investigating

Some VM and remote docker jobs are not being executed in a timely manner. We are investigating.
Posted Sep 17, 2018 - 20:09 UTC
This incident affected: Docker Jobs and macOS Jobs.