Rollbar experienced a technical incident on Jan 8, 2024 as a result of conducting database maintenance. The technical incident began at 11:44 PST and was resolved by 14:30 PST.
A data exposure occurred in Rollbar due to an internal system error, which was triggered during a database maintenance operation. The root cause was related to caching.
Impact
Data that Rollbar received between 11:44:54 PST and 12:23:30 PST was affected. No data received before or after this time range was affected.
There are two categories of affected data:
- Group A: Occurrences mismatched in the cache, general purpose. Approximately 3.4 million occurrences total.
- Group B: Occurrences that additionally became mismatched on disk. 37,037 occurrences total.
Group A:
- These data were accessible only via the Rollbar Web Application and API, only between 11:44:54 PST and approx 12:49 PST when the web application and API were disabled.
- Each affected occurrence could have been exposed only to approximately 1 other Rollbar customer. No data was publicly exposed. No data was sent via notifications.
- A small subset of these occurrences could have been viewed. Based on web activity data, we estimate that approximately 0.035% of the data that could have been exposed, was actually viewed.
- The data potentially viewed would include the data in the Rollbar occurrence.
- This data has been removed.
Group B:
- These occurrences created new items in the incorrect project. These items could have been accessible via the web app or API.
- This data has been removed.
- Out of the 37,037 occurrences potentially affected, 106 known exposures occurred in total.
Impact to your account
If you are a Rollbar customer who was affected by this incident, we have already sent you a notice via email containing the specific information known with respect to your account.
Next steps
All affected data has already been removed and is no longer available via the Rollbar service.
All affected customers have been notified.
A confidential RFO is available to customers under NDA, upon request.
Rollbar staff is assessing the following follow-up actions to prevent an incident like this from occurring again:
- Implementing new protocols to require better verification before the deployment of new database shards.
- Updating logic in our pipeline workers to ensure that no occurrence data can cross account boundaries.
- Improving our methods for caching occurrences to prevent mismatched data from being written.
- Updating our web application logic to serve as a last line of defense for preventing cross-account occurrence data from appearing.
- Further actions will be taken once a full internal postmortem is conducted.
If you have additional questions, you can reach us at:
[email protected]
[email protected]
