Post-mortem from last night's outage

Written By Brian Rue January 11th, 2013

tl;dr: from about 9:30pm to 12:30am last night, our website was unreachable and we weren’t sending out any notifications. Our API stayed up nearly the whole time thanks to an automatic failover.

We had our first major outage last night. We want to apologize to all of our customers for this outage, and we’re going to continue to work to make the service stable, reliable, and performant.

What follows is a timeline of events, and a summary of what went wrong, what went right, and what we’re doing to address what went wrong.

Read more

Join Our Community

Get the latest updates, tutorials and more, delivered to your inbox, once a month.

Happy Halloween

Written By Cory Virok October 31st, 2012
Read more

Real-time Search for Exceptions and Errors

Written By Brian Rue and Cory Virok October 24th, 2012

We’re happy today to announce the release of real-time search. You can now search your exceptions, errors, and log messages by title:

For exceptions, the title contains the exception class and message. For errors and log messages, it contains the entire message. It’s a full-text search that works best on whole words; we also do a few tricks with camelCase and underscore_separated terms.

The search index is kept up-to-date in real-time as new items are added to the system (that’s the “real-time” part). Typically the delay is ~2 seconds from receiving the input at our API to being in the index and searchable.

Current customers can try it out now; let us know if you run into any issues. What else would you like to see indexed?

If you don’t have an account yet, sign up here for early access.

Read more

Using a Request Factory in Pyramid to write a little less code

Written By Brian Rue September 7th, 2012

At, we’ve been using Pyramid as our web framework and have been pretty happy with it. It’s lightweight and mostly stays out of our way.

Pyramid doesn’t have a global request object that you can just import[1], so it makes you pass around request wherever you need it. That results in a lot of library code that looks like this:

# lib/
def flash_success(request, body, title=''):
    request.session.flash({'body': body, 'title': title'})

and a lot of view code that looks like this:

# views/
def login(request):
    # (do the login...)
    helpers.flash_success(request, "You're now logged in.")
    # (redirect...)

That is, there ends up being a lot of function calls that pass request as their first argument. Wouldn’t it be nicer if we could attach these functions as methods on request itself? That would save a few characters every time we call them, and let us stop thinking about whether request is the first or last argument. Pyramid facilitates this by letting us provide our own Request Factory:

Read more

Writing a simple deploy script with Fabric and @roles

Written By Brian Rue August 16th, 2012

I first heard about Fabric a couple years ago while at Lolapps and liked the idea of:

  • writing deployment and sysadmin scripts in a language other than Bash
  • that language being Python, which we used everywhere else

but we already had a huge swath of shell scripts that worked well (and truth be told, Bash isn’t really that bad). But now that we have at clean slate for Rollbar, Fabric it is.

I wanted a simple deployment script that would do the following:

  1. check to make sure it’s running as the user “deploy” (since that’s the user that has ssh keys set up and owns the code on the remote machines)
  2. for each webserver:
  3. git pull
  4. pip install -r requirements.txt
  5. in series, restart each web process
  6. make an HTTP POST to our deploys api to record that the deploy completed successfully

Here’s my first attempt:

Read more

Join Our Community

Get the latest updates, tutorials and more, delivered to your inbox, once a month.

Join Our Community

Get the latest updates, tutorials and more, delivered to your inbox, once a month.