The incident has been resolved. One note, We are still using a slower storage class instead of the standard disk instead of SSD to work around the GCE outage, so the VMs are slower than normal time. We'll revert this change once the GCE outage resolved.
Posted Jun 22, 2021 - 04:45 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jun 22, 2021 - 04:11 UTC
Update
We are continuing to work on a fix for this issue.
Posted Jun 22, 2021 - 03:53 UTC
Identified
We have still seen the issue around VM provisioning, and are implementing a fix for this issue.
Posted Jun 22, 2021 - 02:47 UTC
Update
We are currently using a slower storage class to work around the GCE outage, and we expect any jobs involving lots of disk IO (docker pulls, large artifact uploads/downloads) will take a little longer than normal until we can go back to using our normal SSD.
Posted Jun 22, 2021 - 01:38 UTC
Update
We're attempting to implement a workaround for the current GCE outage to the machine executors.
Posted Jun 22, 2021 - 00:35 UTC
Monitoring
we're currently monitoring the google outage and will provide an update once the google outage is resolved.
Posted Jun 21, 2021 - 23:32 UTC
Update
We are continuing to work on a fix for this issue.
Posted Jun 21, 2021 - 22:24 UTC
Identified
We are continuing to investigate the issue. The issue is being caused by a GCE(Google Compute Engine) outage. https://status.cloud.google.com/
Posted Jun 21, 2021 - 20:59 UTC
Update
We are continuing to investigate this issue.
Posted Jun 21, 2021 - 20:15 UTC
Investigating
We are seeing delays in provisioning jobs using the 2xlarge machine executor.