Blog |

Heartbleed Bug Response

Heartbleed Bug Response

Updated 4/9 7:30pm

What is Heartbleed?

CVE-2014-0346, known as “Heartbleed”, is a bug in OpenSSL v1.0.1 through 1.0.1f that allows a remote
attacker to access private memory on the target server. It has existed for almost 2 years. More info
can be found here: http://heartbleed.com/

With this vulnerability, an attacker can:

  • Get your private key for your domain’s ssl cert
  • Decrypt all current and past SSL traffic to/from all affected machines

If this sounds bad, it is. Most sites on the Internet are affected.

Are you affected?

Probably. If your web server or load balancer is running on linux and you’ve updated your packages
anytime in the last 2 years, you are more-than-likely affected.

To check your OpenSSL version, run openssl version -a

Check out http://filippo.io/Heartbleed/ to test your servers for the vulnerability.

How We Responded

We learned of CVE-2014-0346 at around 4:50pm on 4/7 and immediately began our response. We completed
the most important fix (patching OpenSSL) within about an hour, and have been working over the past
24 hours on related issues.

Here is a timeline of what we’ve done since the vulnerability was announced:

  • 4/7 - 3:01pm - Ubuntu Security Announcements email

    Subscribe to this list here

  • 4/7 - 4:50pm - Began updating our load balancers with the fix. All servers patched by 6pm.

    We’re running nginx on Ubuntu 12.04. Updating is as simple as:

    apt-get update
    apt-get upgrade
    openssl version -a # should show that it was built on April 7, 2014
    service nginx restart

    The above didn’t work for us on the first try because our servers were talking to a mirror that
    hadn’t updated to the latest packages (after all, they were only a couple hours old). Changing the
    domain in each line in /etc/apt/sources.list to archive.ubuntu.com and then running
    apt-get update again solved this.

  • 4/7 - 11pm - rollbar.com and api.rollbar.com SSL certs were rekeyed

    We use DigiCert for our SSL certs. The process was quick and easy.

  • 4/7 - 11:10pm - Previous rollbar.com and api.rollbar.com SSL certs revoked

    In order to prevent a possible man-in-the-middle attack we had Digicert revoke our old certs.

  • 4/7 - 11:30pm - ratchet.io and submit.ratchet.io rekey requested

    We still support our old domain, ratchet.io which use NetworkSolutions SSL certs

  • 4/8 - 11:50am - All rollbar.com cookies were invalidated, forcing users to re-auth

    Since an attacker could have accessed our customers’ cookies, we changed the secret key that we use
    to encrypt cookies. This invalidated all logged-in users’ sessions.

  • 4/8 - 12:30pm - 2:25pm - All third-party tokens and keys were regenerated and deployed

    We use services like Stripe, Mailgun, Heroku - All required new keys to be generated.

  • 4/8 - 3:30pm - ratchet.io and submit.ratchet.io certs were rekeyed and deployed
  • 4/8 - 5:30pm - Published this blog post and added in-app notifications to change passwords and cycle access tokens

Update 4/9

Thanks to
this post on security.stackexchange,
we additionally patched our application and compute servers (everything that can make outgoing HTTPS
requests). This was started at 3:45pm and completed at 4:45pm. The attack surface here is much
lower, as it requires creating a Rollbar account and setting up a webhook that points to the
attacker’s malicious server. We audited our logs and confirmed that there has been no such
suspicious activity.

Recommended actions for Rollbar Customers

  • Change your password
  • Cycle any access tokens you have used (create and start using a new one, then disable or delete
    the old one).

    • For projects, go to the project dashboard, then Settings -> Project Access Tokens.
      Most customers will need to do this.
    • For accounts, go to Account Settings -> Account Access Tokens. Most customers will not need
      to do this.

Note for Heroku Users

If you’re using Rollbar through Heroku, we’ve already started the process of cycling your access
tokens. We’ve created new tokens and updated them in your Heroku config. You should update the token
in any other locations (i.e. development environments, and anywhere it might be hardcoded) and then
disable/delete the old tokens.

Closing notes

This was painful, but we’re thankful to the security researchers who discovered and responsibly
disclosed this issue, and to the security teams at Ubuntu and elsewhere who quickly released patched
packages.

If you have any questions, please don’t hesitate to contact us at support@rollbar.com

Related Posts

See all posts

"Rollbar allows us to go from alerting to impact analysis and resolution in a matter of minutes. Without it we would be flying blind."

Start continuously improving your code today.

Get Started Shape