Summary: between Sunday, December 4th, and Tuesday, December 7th, we saw an increased number of builds that were using CircleCI to mine cryptocurrencies. These builds were triggered by PRs on a multitude of open source repos. These pull requests were invisible on GitHub's UI, and this raised the question of a potential security breach on CircleCI or GitHub as the commits were visible in the users’ repositories when pulling the PR refspecs. After reaching out to GitHub's Security team, we discovered that the PRs were invisible in the UI as they had been marked as spammy, but they could still be pushed to and would generate web hooks.
We also discovered that we were always building the pull requests from forks by default, and there was no way to disable that. On Tuesday, December 7th, we disabled building fork PRs for open source repos unless the Permissive Fork PR Building setting is turned on.
What happened?
On Sunday, December 4th, the CircleCI support team started receiving reports of suspicious builds being run on customer projects. Those reports were treated as regular support requests, as we believed that they could have been produced by the misconfiguration of build settings.
On Monday, December 5th, we received an increased amount of similar requests which made us escalate the priority of the issue. During the initial investigation, we determined that the builds were triggered from forks of open-source projects, and that those projects had the Permissive building of fork pull requests feature disabled. We also confirmed that the builds were being triggered by pushes to GitHub.
On Tuesday, December 6th, we realised that the pull requests that the malicious builds supposedly belonged to couldn’t be viewed on GitHub’s UI. GitHub’s Security department confirmed that those pull requests were hidden as they were marked as spammy. However, pushes to hidden pull requests were still possible and GitHub was sending push webhooks to us as a result.
Later in the day, after inspecting how we handle builds for pull requests, we realised that the PRs from forks of open-source repos were being built regardless of the value of the Permissive building of fork pull requests feature. The secrets from the parent repos were only imported in forked PR builds if the Permissive fork PR building was enabled, but when it was disabled, the builds for fork PR pushes to open source projects were still triggered. This kind of fork PR push was being used to mine cryptocurrency during the builds.
At 00:30 UTC on Wednesday, December 7th, we deployed a fix that disabled fork PR building unless the Permissive building of fork pull requests feature is enabled. This stopped further mining PR builds from appearing on customer projects.
What we are doing about it?
We have split the Permissive building of fork pull requests setting into two settings: a) Run builds of pull requests from forks, and b) Pass secrets to forked pull requests. This gives the owners of both open-source and private projects more fine-grained control over the way builds are run for pull requests coming from forks. Both settings are disabled by default.
If you would like to continue building fork pull requests for your open-source project, please enable the Run builds of pull requests from forks feature in your project’s settings. Please review our doc on building pull requests from forks before enabling building fork PRs, and before enabling the Pass secrets to forked pull requests setting.
We are also working with GitHub to understand the handling of spam PRs better, and to establish further measures that would prevent malicious builds from being triggered on the platform.
Posted Dec 07, 2016 - 20:00 UTC