We have worked thru the remaining backlog and all of our systems are operational. Thank you for your patience as we dealt with this. Please note that you may need to push a new commit for any workflow that was not started.
Posted Mar 15, 2019 - 22:00 UTC
Update
We are continuing to monitor as we drain the remainder of the backlog. Most requests are processing with minimal delay. Some continued network issues are impacting overall capacity. We will update within 30 mins.
Posted Mar 15, 2019 - 21:37 UTC
Monitoring
We are now processing the backlog of workflows. We are monitoring closely to ensure that all operations are happening as expected. Will update again within 20 minutes as we continue to make progress.
Posted Mar 15, 2019 - 21:06 UTC
Update
Workflows are once again running and we are monitoring closely as our demand scales. We will update within 20 mins with progress against our backlog of work.
Posted Mar 15, 2019 - 20:57 UTC
Update
We continue to work on recovering full workflow execution. We are now receiving inbound hooks and those workflows should run once we are fully back online. We will update within 20 mins with any additional progress.
Posted Mar 15, 2019 - 20:42 UTC
Update
Our UI is back online and we are working to recover other systems. We are actively working to recover workflows and scaling up capacity to support a quick recovery. Will update within the next 20 mins with further information.
Posted Mar 15, 2019 - 20:19 UTC
Update
The earlier networking issue has created an ongoing issue with one of our core systems, resulting in impact across builds and our UI. We are actively working to recover that component and are starting to recover dependent systems. Will update in 20 mins on the status of the recovery.
Posted Mar 15, 2019 - 20:14 UTC
Update
We are continuing to recover from the AWS networking event
Posted Mar 15, 2019 - 20:03 UTC
Update
The issue has been identified and a fix is being implemented.
Posted Mar 15, 2019 - 19:21 UTC
Update
We are continuing to recover from the AWS networking event
Posted Mar 15, 2019 - 19:10 UTC
Identified
We are recovering from an AWS networking event and we are scaling up to meet the load.
Posted Mar 15, 2019 - 18:41 UTC
Update
We are continuing to investigate this issue.
Posted Mar 15, 2019 - 18:25 UTC
Investigating
We are experiencing an interruption in our ability to run Workflows. We are investigating.