Resolved -
This incident has been resolved.
Sep 11, 20:52 UTC
Monitoring -
Machine, MacOS and Windows jobs are now within normal operating parameters. We will continue to monitor.
Sep 11, 20:36 UTC
Update -
Machine and Windows jobs are now within normal operating parameters. Wait times continue to be longer than the expected normal for MacOS jobs as we work through the backlog of jobs. We thank you for your patience while our engineers continue to work towards mitigation.
Sep 11, 18:55 UTC
Update -
We are continuing to optimize our infrastructure and resource usage due to a slower than expected recovery times. While the wait times have improved they continue to be longer than the expected normal. We thank you for your patience while our engineers continue to work towards mitigation.
Sep 11, 18:19 UTC
Identified -
We are beginning to see early signs of recovery with Linux and Windows jobs processing. As the system works through the backlog, you should start to see reduced wait times. Our engineers continue to monitor our resources closely to ensure a steady improvement in our systems. macOS jobs may see a longer recovery period.
We thank you for your patience during this time as we work towards a complete mitigation.
Sep 11, 17:27 UTC
Update -
We are continuing to investigate and mitigate the high resource pressure on our platform. The mitigating efforts may lead to some steps within the running jobs having an empty output or being skipped. We thank you for your patience while our engineers work to bring the platform to a healthy state
Sep 11, 16:44 UTC
Update -
We are continuing to alleviate the resource pressure our platform is currently experiencing. We thank you for your patience while we work to mitigate the issue causing high wait times on multiple job types
Sep 11, 16:15 UTC
Update -
Our engineers continue to investigate the issue and are working on changes to our infrastructure to relieve the pressure caused by high resource usage. We thank you for your patience while we work to reduce these high queue times.
Sep 11, 15:31 UTC
Update -
We are continuing to investigate this issue and are noticing high pressure on other job types at this time.
Sep 11, 14:54 UTC
Update -
Our engineers are continuing to investigate the issue causing high utilization of our resources and higher wait times for machine and other job types. We thank you for your patience as we work to mitigate this issue.
Sep 11, 14:51 UTC
Investigating -
Our engineers are currently investigating an issue causing longer wait times for machine jobs running on the platform. We will provide further updates as more information becomes available.
Sep 11, 14:29 UTC