Real-time health of CircleCI's own infrastructure and services.
Summary: On October 26th all of our OS X customers had issues with downtime. We understand that you depend on us to run your business, we apologize for the outage and have provided a summary of events for those interested in the details. We are working with our provider to minimize disruptions tied to our hosting provider. If you incur an overage on minutes in the next 30 days because of the lengthy queue time, please reach out to your customer success manager for prompt attention and resolution.
Early in the morning of 26 October CircleCI’s Support Team started to gather reports and investigate a sharp increase in network timeouts and build failures within our OS X fleet relating to network issues.
During the discovery and investigation phase of our Incident process the build queue for the OS X fleet started to climb and our build success rate dropped from around 60% to 25%. This caused our team to increase the priority with our data center vendor about networking speed improvements.
We got the vendor on the phone to have their networking team look into the issue and they could not initially find anything wrong with any of the network connections. During this conversation the build queue started to climb as a result in the normal job load increase for the day and we switched to Partial Outage.
At 08:00am PDT the network team at our vendor tried to give us more bandwidth and, in their own words, experienced an "unexpected result" that caused all of our OS X fleet to go offline.
The network was re-established quickly but the delay in communication and the time it took for us to reset and check our fleet generated a huge increase in the backlog which, coupled with the already slow network, created a backlog large enough to overwhelm the build rate for the remainder of the day.
We looked into multiple issues during the time we were contending with all of the above which included a reduced number of containers available and a slow container recycle time. During this investigation we discovered that a code change to the OS X builder master was causing containers to not get placed back into the available pool. That change was reverted and the OS X builder master instances were rolled.
That still didn't allow our available container count to return to normal so we continued to investigate, we then discovered that on some of the fleet the local firewall (which is used to isolate containers from each other) was offline, which should not have been a major item, but was causing not only it's containers to be offline but preventing the containers that were being rebooted after a job from being marked as available. After a manual reset of those firewall containers we were able to return to our normal level of available containers and the backlog was finally drained.
We still need to explore why the firewall was causing this issue and will reach out to the Platform team for definitive answers. Once identified, we will be able to create a plan to reduce this type of incident from happening in the future. After the backlog was drained the data center vendor then performed the bandwidth upgrade and it went of without a hitch and our OS X fleet now has increased bandwidth which we are monitoring to see if we need more.