We have implemented a fix and we're seeing normal operation. Thank you for your patience throughout this incident.
Posted Mar 22, 2022 - 16:13 UTC
Monitoring
We have implemented a fix and are currently monitoring the result. Customers can re-run previously failed workflows.
Posted Mar 22, 2022 - 16:01 UTC
Identified
We have identified the cause of the failing permission checks and we are currently working on a fix. Customers should start to see successful workflows, and can re-run previously failing workflows. We will update once we have more information to share.
Posted Mar 22, 2022 - 15:58 UTC
Update
We are still experiencing issues with our permissions service causing permission checks to fail. This is causing some pipelines to fail, and could also result in the inability to view contexts in the UI. We are continuing to investigate.
Posted Mar 22, 2022 - 15:38 UTC
Update
We are seeing a number of pipelines failing due to permission checks failing. We are continuing to investigate the cause of the incident.
Posted Mar 22, 2022 - 15:10 UTC
Investigating
We are experiencing delays with pipelines starting across all executor types. Some pipelines may fail. We are investigating and will update with more information as we have it.