Error alert notifications + how to use them for better monitoring

Rollbar offers an impressive array of error alerting and notification options for you to choose from, which is awesome! But sometimes we hear from our users that they aren't quite sure how to leverage their notifications to get the best results. What do notifications here at Rollbar do? How do they work? How should you interpret them? And perhaps most importantly, what are the best practices? Let's explore the answers to all these questions today.

Rollbar supports many different messaging and incident management tools, so you and your team can get notified about errors and important events. These notifications can be anything from a message in a Slack channel to an alert in PagerDuty. There are some common notification types that we offer, no matter what tool you're using to be notified.

So now that you know what types of notifications you can get, how about customizing them? You can filter your notifications to adjust the circumstances under which you receive a notification. For instance, you can control what team the notification is sent to, or even which individual users should be notified. You may wish to only receive a notification when the source is a certain notifier, or the title of the item matches a specific string. You can find all of the myriad filters available to you for notifications (and further documentation of how to configure these filters) in the docs.

Once you've set up the specific conditions under which a notification is triggered, you might want to customize the message that gets sent to you and your team. That's where the different notification variables Rollbar offers come in handy. You can have your notification alert an entire channel in Slack to a new item in Rollbar, for instance. Check out the docs to see exactly what variables are available, and then come back here for some tips and tricks on how to use filters and variables to get the most out of Rollbar notifications.

Routing your notifications effectively

Here at Rollbar, we sometimes hear from users that they aren't sure exactly how to route their notifications so that the right people see the right ones. The best way to do this is to use our filters! You can customize your notifications so that each one goes to a specific person or team, based on specific criteria. For instance, let's say you have an Item Reopened rule, and you want to alert your entire engineering team right away in Slack when an item is reopened, but only if the item's level is Error, and it occurred in production. Just use notification filters in your Slack notification rules, and have it post to your #engineering channel, and voila! All your engineers will see when an item is reopened that matches that criteria.

In essence, the best way to ensure the right people see the right notifications at the right time is to set filters on the environment, filename, title, or other criteria as needed, and send it to the specific emails, teams, or channels that need to be alerted in a timely manner. The more specific you can get with your matching criteria, the less notification clutter you'll get. You can set up multiple rules for the same type of notification, so you can customize exactly who sees what notification to reduce the clutter for everyone on your team. Want the Python developers on your team to get all the Python errors and nothing else? Filters are your friend! The best way to get started with filters is to check out the docs on notification filters.

There are two main types of tools that Rollbar users have questions about when it comes to properly setting up their notifications, and maximizing the information included in the notifications: messaging apps, and incident management tools. Let's take a look at messaging apps first.

Notifications for messaging apps

When setting up notifications for messaging apps such as Slack or HipChat , you want to make sure the message is succinct but helpful and informative. A basic Slack message is [{{"{{project_slug"}}}}] {{"{{username"}}}} deployed revision {{"{{revision"}}}} to {{"{{environment"}}}} {{"{{link"}}}}, which gives a good amount of information in one sentence for a deploy notification. Rollbar supports variables in notifications using {{VARIABLE_NAME}} syntax. Different variable values are available depending on the type of event that triggers the notification, which you can check out in the docs here. Any data value sent in the JSON payload of an item or occurrence may be used as a variable, including custom data. Examples of usage are {{"{{body.request.url"}}}} and {{"{{body.server.host"}}}}. If your JSON payload includes the custom values {{"{{ handler: { key: process-job, id:100"}}}} then you can use the variables {{"{{body.handler.key"}}}} and {{"{{body.handler.id"}}}} in your notifications. To view the full set of available values, look at the "Params" values of an occurrence in your project. All params besides the ones listed in the above notification variables must be prefaced with "body".

Sometimes, our users want to be able to alert everyone in a channel using common Slack syntax such as @channel, @group, @here, or @everyone. You can do this with notification variables in Rollbar as well! Make sure to use the syntax <!channel>, <!group>, <!here>, or <!everyone>. So, for the earlier example, if you wanted to add @here to the beginning of the message, you would simply change the message to <!here> [{{"{{project_slug"}}}}] {{"{{username"}}}} deployed revision {{"{{revision"}}}} to {{"{{environment"}}}} {{"{{link"}}}}.

Here at Rollbar, we have six different notification rules for Slack:

As you can see, we set a high occurrence rate notification to alert us once there's been at least 100 occurrences in 5 minutes of an error in production, which ensures that we are notifed when there is a error occurring enough that it's immediately concerning. We also like to be alerted if an item that we thought we resolved is reopened, if an item is reactivated or resolved, if a deploy has happened, and especially if there is a new error. Here's how some of Rollbar's notifications look like in Slack:

Notifications for incident management tools

The type of information you want in a notification from incident management tools like PagerDuty, VictorOps or Datadog. is probably slightly different than for a messaging app. Typically, the alerts you're getting for an incident management tool deal with the possibility of an urgent problem that your team needs to address. You're not going to want to be alerted with these tools if a simple deploy is reported. You'll want to know what type of notification was triggered and why, as quickly as possible. Therefore, the basic incident management message we start with is [{{"{{project_slug"}}}}] {{"{{environment"}}}} - {{"{{trigger_description"}}}} {{"{{level"}}}}: {{"{{title"}}}}. All of the incident management tools we are currently integrated with, aside from Datadog, offer the same amount of customization for your notifications. Datadog has some extra options, which are detailed in the docs.

Notifications for email

Aside from messaging apps and incident management tools, Rollbar also offers email notifications, and automatically sends out a daily summary of each project's activity via email. This daily summary includes how many occurrences of each type of error occurred in the past day, broken down by error level and environment. You can specify the time it is sent out daily, as well as whether to send it only if there is new data. The summary can be a very helpful overview of what errors are occurring and where, without needing to log into the UI. We recommend using the configurable email notifications for alerting your team to items that aren't vital to be seen immediately.

Here at Rollbar, we have 11 email notifications configured:

This allows us to get notified via email for every occurrence with certain titles, if an item is resolved, reactivated, or reopened, and when a deploy happens. We also have two different high notification occurrence rates to be notified on, one of which goes solely to one person, so you can see there are quite a few configuration options for who gets alerted and under what circumstances. Here's how one of these emails looks in our inboxes and in the email itself:

Now that you've gotten an overview of Rollbar notification types, give them a spin for your project and you can be sure you'll never miss an important error again. If you need help setting up notifications for your project, don't hestitate to contact us at [email protected]. We love hearing from you and we're happy to help!

If you haven’t already, sign up for a 14-day free trial of Rollbar and let us help you take control of your web and mobile application errors (and notifcations).