Resolved -
We have resolved the issues affecting job triggering, workflow starts, and API queries. Our systems have been stabilized and are operating normally.
What was impacted: Job triggering, workflow starts, API queries, and pipeline page loading experienced disruptions for some customers. This affected all resource classes and executors.
Resolution: We implemented mitigation measures to address high volume workflow queries impacting our internal systems and increased system capacity. All new jobs and workflows are now starting normally, pipeline pages are loading, and API queries are functioning as expected.
What to expect: If you have jobs that became stuck during this incident, please rerun them. If you continue to experience issues after rerunning, please contact our support team. Some customers may still see jobs stuck in a cancelling state. Engineering is aware and addressing to mitigate risk.
We will continue monitoring our systems and conducting a thorough review to identify additional preventive measures.
Dec 3, 23:48 UTC
Update -
We have deployed changes to mitigate the high volume of workflow queries impacting our systems. Pipeline pages that were previously failing to load are now loading successfully, and we are seeing significant reduction in API errors.
What's impacted: Some customers continue to experience jobs stuck in a not-running state from earlier in the incident. New job triggering and workflow starts are now functioning normally.
What's happening: We have implemented mitigation measures and increased system capacity. We are continuing to investigate the remaining stuck jobs for affected customers.
What to expect: If you experienced issues loading pipeline pages or querying workflow data via the API, these should now be resolved. New jobs and workflows should trigger normally. If you have jobs that appeared stuck earlier, please try rerunning them while we continue to investigate the reports of jobs that do remain stuck for a small number of customers. The data for those workflows should be available and queryable.
Next update: We will provide an update within 30 minutes. Thank you for your patience while our engineers work through this incident.
Dec 3, 22:59 UTC
Update -
We are currently experiencing issues affecting job triggering and workflow starts across all resource classes. Jobs may appear stuck in a not-running state, and some customers may encounter 500 errors when making API calls to check job or workflow status.
What's impacted: Job triggering, workflow starts, and API queries for job and workflow status are experiencing disruptions. This affects all resource classes and executors. Some users may also experience issues loading the pipeline page.
What to expect: We are actively working to stabilize our systems and restore normal operations. We will provide updates as we make progress toward resolution.
We thank you for your patience while we work through these issues - we will update with our progress within 30 minutes or earlier.
Dec 3, 21:58 UTC
Investigating -
We are currently investigating reports of jobs not starting. We apologize for the inconvenience.
Dec 3, 21:28 UTC