Resolved -
We have worked thru the remaining backlog and all of our systems are operational. Thank you for your patience as we dealt with this. Please note that you may need to push a new commit for any workflow that was not started.
Mar 15, 22:00 UTC
Update -
We are continuing to monitor as we drain the remainder of the backlog. Most requests are processing with minimal delay. Some continued network issues are impacting overall capacity. We will update within 30 mins.
Mar 15, 21:37 UTC
Monitoring -
We are now processing the backlog of workflows. We are monitoring closely to ensure that all operations are happening as expected. Will update again within 20 minutes as we continue to make progress.
Mar 15, 21:06 UTC
Update -
Workflows are once again running and we are monitoring closely as our demand scales. We will update within 20 mins with progress against our backlog of work.
Mar 15, 20:57 UTC
Update -
We continue to work on recovering full workflow execution. We are now receiving inbound hooks and those workflows should run once we are fully back online. We will update within 20 mins with any additional progress.
Mar 15, 20:42 UTC
Update -
Our UI is back online and we are working to recover other systems. We are actively working to recover workflows and scaling up capacity to support a quick recovery. Will update within the next 20 mins with further information.
Mar 15, 20:19 UTC
Update -
The earlier networking issue has created an ongoing issue with one of our core systems, resulting in impact across builds and our UI. We are actively working to recover that component and are starting to recover dependent systems. Will update in 20 mins on the status of the recovery.
Mar 15, 20:14 UTC
Update -
We are continuing to recover from the AWS networking event
Mar 15, 20:03 UTC
Update -
The issue has been identified and a fix is being implemented.
Mar 15, 19:21 UTC
Update -
We are continuing to recover from the AWS networking event
Mar 15, 19:10 UTC
Identified -
We are recovering from an AWS networking event and we are scaling up to meet the load.
Mar 15, 18:41 UTC
Update -
We are continuing to investigate this issue.
Mar 15, 18:25 UTC
Investigating -
We are experiencing an interruption in our ability to run Workflows. We are investigating.
Mar 15, 18:25 UTC