Rollbar Security Notice and Response – Sept 8, 2023

TL;DR:

• A third party gained unauthorized access to our data warehouse. Rollbar project access tokens were exposed.
• We have already expired all read/write scope tokens - If you use the Rollbar API via a read/write token (like reading the Metrics API or updating item statuses), you must rotate those.
• We will expire all post_server_item tokens on Oct 10, 2023 12:00 am UTC, if you use Rollbar via a server-side language, like Ruby, PHP, Python, Node.js, Go, Java, or .NET, you will need to rotate those tokens.
• We've released a new tool to make this easier in our project settings.

What Happened

On September 6, 2023, at 8:28 AM PDT, we identified an irregularity in our data warehouse query logs. We immediately performed an initial forensic analysis, determining what had occurred and assessing the impact.

Our preliminary analysis established that a cloud platform service account that only had access to our data warehouse was used by an unauthorized party. Our investigation indicates this access occurred from August 9, 2023, to August 11, 2023.

When we became aware of this access, we disabled the service account and began analyzing what actions had been taken by the unauthorized party. The party first tried to launch compute resources, and after that failed for lack of permission, they accessed the data warehouse. They ran searches that suggested they were interested in Bitcoin wallets or other cloud credentials.

We will also engage a third-party forensic consultant to assist us in verifying these findings, and that work is ongoing. Our analysis of the incident continues, but we are contacting you now because our initial forensic research indicates the unauthorized party accessed data about your account, including:

• Rollbar usernames and user email addresses
• Account names
• Project and environment names
• Project access tokens
• Project service link configuration

Impact to Customers

The affected project access tokens need to be rotated. We are taking the following approach:

• All 'read' and 'write' scope project access tokens are being invalidated immediately. You will need to rotate these tokens if you are using the Rollbar API via a read/write token (like reading the Metrics API or updating item statuses). It's crucial that you generate new tokens promptly to restore these services.
• All 'post_server_item' scope project access tokens will be invalidated on Oct 10, 2023 12:00 am UTC. If you use Rollbar via a server-side language, like Ruby, PHP, Python, Node.js, Go, Java, or .NET, or to record deployments or upload source maps, you must rotate those tokens. Failure to update these tokens will result in your application no longer sending your error data to Rollbar (recording deployments or uploading source maps).
• All 'post_client_item' scope project access tokens (i.e., for client-side javascript and mobile) are public by design, so they are not being invalidated.

We strongly encourage you to revisit your account and examine any unusual activity that may suggest malicious use of your data. Our team is actively monitoring the situation and will provide account-by-account updates if any more information becomes available.

Help to Resolve the Issue

We understand that regenerating and updating tokens can introduce complexity into your workflow. We have built a new Expired Tokens feature to support you through this process. This will reduce the time you have to spend in the UI by allowing you to regenerate your tokens by project and account. For detailed instructions, please visit our access token documentation. Should you require additional guidance or assistance, you can email us at [email protected] or initiate a chat directly in the app. We're committed to providing the timely and practical support you need to navigate this situation.

Next Steps:

1. Regenerate “read” & “write” tokens: We have expired all ‘read’ and ‘write’ tokens. Generate new tokens with our “Generate all tokens” button on your access token page to restore data retrieval via our API.
2. Update “post_server_item” tokens: You have 30 days to update these tokens in your SDK to avoid interrupting our service.
3. Audit your settings: Review your project settings to ensure they align with your security requirements.
4. Monitor your account: Keep an eye on your account for any unusual activity that may indicate malicious use of your data.
5. Contact customer support: For any questions or technical support, contact [email protected] or initiate a chat in the app.

FAQ

Why were the “read” and “write” tokens expired immediately?

We took immediate action to minimize any potential risks associated with the compromised data. Invalidating “read” and “write” tokens was a crucial first step in ensuring the security of your account.

What happens if I don’t update my ‘post_server_items” in 30 days?

Failure to update your code or configuration SDK with the "post_server_item” scoped tokens within this timeframe will result in your applications no longer being able to send data to Rollbar until they are updated

Have any of my accounts been directly compromised due to this breach?

We haven’t seen evidence to suggest that any accounts have been accessed using the compromised data.

What steps are you taking to prevent future incidents?

We are investing in our core product and platform and implementing new initiatives to reduce outages, bolster security measures, and optimize system performance to serve your organization better.

How can I regenerate my expired tokens?

To simplify the process, we added the ability to regenerate all expired tokens on our project token access page under project settings.