Postmortem -
Read details
Mar 24, 01:38 UTC
Resolved -
This incident has been resolved. Docker, Mac, Machine jobs should be running correctly.
Mar 15, 01:59 UTC
Monitoring -
Machine jobs are now running, we expect the backlog to clear over the next hour. We will continue to monitor.
Mar 15, 01:34 UTC
Update -
We are seeing Docker and Mac jobs are now becoming operational. We are seeing Machine jobs are starting to run, but we are still monitoring for capacity issues.
Mar 15, 01:27 UTC
Update -
We've identified a problem in our internal networking systems that triggered the issue and made changes to configuration and deployments to address the issue.
Docker jobs with contexts are starting to run successfully. We will continue to monitor for further issues with Contexts.
We are still working to add capacity for machine jobs.
Mar 15, 01:03 UTC
Update -
UI access remains stable. We are still tuning capacity and resources to process backlogs. We appreciate your patience and understanding.
Mar 15, 00:14 UTC
Update -
UI and API access has mostly recovered. We are still working through capacity issues for Machine jobs and Contexts.
Mar 14, 23:39 UTC
Update -
We are continuing to add capacity to process the backlog of jobs. We appreciate your patience.
Mar 14, 22:59 UTC
Identified -
We are seeing jobs running again, and are adding additional capacity.
Mar 14, 22:28 UTC
Update -
We are seeing intermittent successes in the UI as some components have been recovered. We are continuing to work on getting jobs moving and will update shortly with status on jobs.
Mar 14, 22:13 UTC
Update -
Our debugging efforts have led to partial recovery of some internal services. We are seeing intermittent success on user actions and continue to work on restoration. We'll report back in under 20 mins as we see the impact.
Mar 14, 21:46 UTC
Update -
We are continuing to see degradation on our services including jobs not starting and UI impacts. We are currently investigating networking issues and will update further within 30 mins.
Mar 14, 21:09 UTC
Update -
We are continuing to investigate this issue.
Mar 14, 20:46 UTC
Update -
We are continuing to investigate this issue.
Mar 14, 20:19 UTC
Update -
We are continuing to investigate this issue. Thank you for your patience while we work toward a resolution.
Mar 14, 19:52 UTC
Update -
We are continuing to investigate this issue.
Mar 14, 19:20 UTC
Update -
We are continuing our investigation. Currently jobs are delayed in starting and not completing.
Mar 14, 18:58 UTC
Investigating -
We are seeing a delay in starting jobs. We are currently investigating and will update shortly.
Mar 14, 18:19 UTC