Heartbleed Bug Response

Updated 4/9 7:30pm

What is Heartbleed?

CVE-2014-0346, known as “Heartbleed”, is a bug in OpenSSL v1.0.1 through 1.0.1f that allows a remote attacker to access private memory on the target server. It has existed for almost 2 years. More info can be found here: http://heartbleed.com/

With this vulnerability, an attacker can:

• Get your private key for your domain’s ssl cert
• Decrypt all current and past SSL traffic to/from all affected machines

If this sounds bad, it is. Most sites on the Internet are affected.

Are you affected?

Probably. If your web server or load balancer is running on linux and you’ve updated your packages anytime in the last 2 years, you are more-than-likely affected.

To check your OpenSSL version, run openssl version -a

Check out http://filippo.io/Heartbleed/ to test your servers for the vulnerability.

How We Responded

We learned of CVE-2014-0346 at around 4:50pm on 4/7 and immediately began our response. We completed the most important fix (patching OpenSSL) within about an hour, and have been working over the past 24 hours on related issues.

Here is a timeline of what we’ve done since the vulnerability was announced:

• 4/7 - 3:01pm - Ubuntu Security Announcements email

  Subscribe to this list here

• 4/7 - 4:50pm - Began updating our load balancers with the fix. All servers patched by 6pm.

  We’re running nginx on Ubuntu 12.04. Updating is as simple as:

  apt-get update
  apt-get upgrade
  openssl version -a # should show that it was built on April 7, 2014
  service nginx restart

  The above didn’t work for us on the first try because our servers were talking to a mirror that hadn’t updated to the latest packages (after all, they were only a couple hours old). Changing the domain in each line in /etc/apt/sources.list to archive.ubuntu.com and then running apt-get update again solved this.

• 4/7 - 11pm - rollbar.com and api.rollbar.com SSL certs were rekeyed

  We use DigiCert for our SSL certs. The process was quick and easy.

• 4/7 - 11:10pm - Previous rollbar.com and api.rollbar.com SSL certs revoked

  In order to prevent a possible man-in-the-middle attack we had Digicert revoke our old certs.

• 4/7 - 11:30pm - ratchet.io and submit.ratchet.io rekey requested

  We still support our old domain, ratchet.io which use NetworkSolutions SSL certs

• 4/8 - 11:50am - All rollbar.com cookies were invalidated, forcing users to re-auth

  Since an attacker could have accessed our customers’ cookies, we changed the secret key that we use to encrypt cookies. This invalidated all logged-in users’ sessions.

• 4/8 - 12:30pm - 2:25pm - All third-party tokens and keys were regenerated and deployed

  We use services like Stripe, Mailgun, Heroku - All required new keys to be generated.

• 4/8 - 3:30pm - ratchet.io and submit.ratchet.io certs were rekeyed and deployed

• 4/8 - 5:30pm - Published this blog post and added in-app notifications to change passwords and cycle access tokens

Update 4/9

Thanks to this post on security.stackexchange, we additionally patched our application and compute servers (everything that can make outgoing HTTPS requests). This was started at 3:45pm and completed at 4:45pm. The attack surface here is much lower, as it requires creating a Rollbar account and setting up a webhook that points to the attacker’s malicious server. We audited our logs and confirmed that there has been no such suspicious activity.

Recommended actions for Rollbar Customers

• Change your password

• Cycle any access tokens you have used (create and start using a new one, then disable or delete the old one).

  • For projects, go to the project dashboard, then Settings -> Project Access Tokens. Most customers will need to do this.
  • For accounts, go to Account Settings -> Account Access Tokens. Most customers will not need to do this.

Note for Heroku Users

If you’re using Rollbar through Heroku, we’ve already started the process of cycling your access tokens. We’ve created new tokens and updated them in your Heroku config. You should update the token in any other locations (i.e. development environments, and anywhere it might be hardcoded) and then disable/delete the old tokens.

Closing notes

This was painful, but we’re thankful to the security researchers who discovered and responsibly disclosed this issue, and to the security teams at Ubuntu and elsewhere who quickly released patched packages.

If you have any questions, please don’t hesitate to contact us at support@rollbar.com