OS X builds are queueing

Incident Report for CircleCI

Postmortem

At 1429 UTC our OS X data center vendor experienced a network outage which caused our OS X fleet to stop taking jobs. This reduced our OS X fleet capacity tremendously and we immediately started to get a backlog of jobs.

At 1448 UTC the network was restored and we immediately started to drain the job backlog and we cleared the backlog at 1530 UTC.

We considered the incident as resolved at 1649 UTC.

Posted Mar 07, 2017 - 20:06 UTC

Resolved

This incident has been resolved.

Posted Mar 07, 2017 - 16:49 UTC

Monitoring

The queue is now empty and builds are being dispatched correctly. We're continuing to monitor the situation.

Posted Mar 07, 2017 - 16:12 UTC

Identified

Builds are being dispatched again, but we're still investigating the underlying cause

Posted Mar 07, 2017 - 15:30 UTC

Investigating

OS X builds are queueing. We are investigating the reason for that and will update in 30 mins.

Posted Mar 07, 2017 - 14:55 UTC