Doing some final backend tests. We're nearly ready to flip the switch.
Posted Oct 30, 2013 - 20:23 UTC
Update
We are expecting to go live again very soon. Working out the final details.
Posted Oct 30, 2013 - 17:44 UTC
Update
We are still waiting on upstream providers to ensure we have secured customer data, before resuming service. We are also preparing our post-mortem and a guide to getting back up and running once we resume our build servers.
Posted Oct 30, 2013 - 16:21 UTC
Update
We are ready to bring CircleCI back up. Before proceeding, we are waiting on a number of upstream vendors to validate that user data is protected.
Posted Oct 30, 2013 - 14:53 UTC
Update
We are working through your support emails as quickly as possible. We apologize for the delay in getting back to you, as there are a lot of support requests at the moment. We are also working on a post-mortem which explains the incident and our response in detail, which should answer all your questions.
Posted Oct 30, 2013 - 12:38 UTC
Update
We sent out alerts about potentially compromised SSH keys. If you received an email, we advise you to revoke that key and check the server to which it had access for signs of unusual access.
Posted Oct 30, 2013 - 11:17 UTC
Identified
We are currently rebuilding the cluster, and working out the kinks after cycling all our credentials. We expect to be back today (Pacific time).
Posted Oct 30, 2013 - 08:27 UTC
Investigating
Our support system is down, so your support requests may not have gotten to us. Please email us directly at support@circleci.com if you need to contact us.
Posted Oct 30, 2013 - 07:23 UTC
Update
We have contacted all upstream providers, and asked them to revoke all keys we have access to. We are close to completing our security response, and will move to providing more detail and recovering the service.
Posted Oct 30, 2013 - 06:19 UTC
Identified
We have notified all users and recommended appropriate action:
"We are contacting you to inform you of an ongoing security incident affecting CircleCI customers, as a result of the compromise of our database (http://security.mongohq.com/notice).
We are taking aggressive action to protect your data and systems. At this time, we have suspended all CircleCI account access, and all builds & workers have been suspended. In addition we have revoked all access to Heroku and GitHub OAuth tokens and API keys uploaded to CircleCI.
We do not yet know the scope and impact of the intrusion and are therefore treating this event as if all data has been compromised. While we have no evidence that these credentials have been compromised, we urge you to revoke the following:
SSH keys that were uploaded to CircleCI
API tokens added to CircleCI as environment variables
secrets stored in GitHub repositories
We will be keeping you informed at https://status.circleci.com and will update you at regular intervals as the situation progresses.
We deeply regret that this has happened and are working around the clock to resolve this incident and protect your data and systems."
Posted Oct 30, 2013 - 05:25 UTC
Update
We are still investigating the issue. The full team is engaged and we are working with upstream providers to diagnose and respond to the issue, and protect all of our users. We will keep you informed.
Posted Oct 30, 2013 - 04:17 UTC
Update
We are currently investigating an ongoing issue with our database service. At this time, we have suspended all account access to our service. All builds & workers have been suspended. We will have another update in the next 30 minutes.
Posted Oct 30, 2013 - 03:20 UTC
Investigating
CircleCI is experiencing technical problems. We're investigating and should have an update within 30 minutes.