Improve Error Detection and Speed Up Resolution in Selenium Tests

Reducing errors in production is critical for all engineering teams. The earlier the error is detected, the cheaper it is to fix and the lower the impact it has on your development team and customer experience. The shift-left movement has pushed more and more of this test automation work on development teams even though the way to debug a failed test and find the root cause is still inefficient and time consuming for all developers.

There is a better way to detect, identify and resolve errors in your applications when executing your automated testing suite.

Developer Pain with Automated Tests

• Automated Test Report does not show the full error stack trace.
• Missing detail to understand and fix the root cause.
• Debug process is time consuming to re-run test script and debug locally.
• Reproducing errors is often inconsistent or impossible.
• Environment may lack exact data points and variables to accurately reproduce the error.

What do we have to do today?

Let’s look at a practical example of what most development teams do today. Imagine that we’ve run our automated test and some failure occurred. If you have a QA team your first question is; what failed? The QA team will point you to the test results of the failed test case or test suite. The test failure happens in most cases because of an assert that failed in the test case without any other information.

Example Assert:
Assert.assertNotEquals(Actual, Expected);

Example Test Failure Report

We can see the exact step that failed with a screenshot but we will still need to re-run the test to get to the root cause of the failure in the application itself.

The missing piece here is information from the application under test to show the real error or failure in the application under test when it occurs.

Let’s look at how we can improve this with Rollbar

First we want to ensure that we have Rollbar instrumented in the application (View Rollbar SDK’s), if you already have Rollbar instrumented you are moments away from a debugging-life changing experience.

Next, we want to notify Rollbar when a selenium test is executing so we can link the test failure (when it occurs) to the Rollbar errors detected at the time of the test execution.

Here is a basic example of injecting the Selenium SessionId into the Rollbar SDK during the test execution. In this example we will fetch the selenium sessionid and inject it into the web application using the ExecuteScript function.

The important line of code is this one:

_driver.ExecuteScript("Rollbar.configure({ payload: { 'seleniumsessionid' : '" + _driver.SessionId + "' } })");

Example Test:

[Test]
public void InvalidLoginAuth()
{
    //Use driver to open Page
    _driver.Navigate().GoToUrl("https://test.app.com/login");
    _driver.ExecuteScript("Rollbar.configure({ payload: { 'seleniumsessionid' : '" + _driver.SessionId + "' } })");

    IWebElement emailInput = _driver.FindElement(By.Id("email"));
    IWebElement passwordInput = _driver.FindElement(By.Id("password"));
    IWebElement loginButton = _driver.FindElement(By.Id("btnLoginAuth"));

    emailInput.SendKeys("[email protected]");
    passwordInput.SendKeys("test1234");
    loginButton.Click();

    var lastRollbaruuid = _driver.ExecuteScript("return _lastRollbaruuid;");

    _driver.ExecuteScript("browserstack_executor: {\"action\": \"setSessionStatus\", \"arguments\": {\"status\":\"failed\", \"reason\": \" View Error in Rollbar https://rollbar.com/occurrence/uuid/?uuid=" + lastRollbaruuid + " \"}}");
    Assert.Fail();
}

That is it! Your test executions can now continue normally without any modifications to your selenium scripts.

The best test automation failure results you have ever had!

If we take the example above of an assert that failed we will now have the full error stack trace in Rollbar linked to the selenium test case that failed.

Inside my test failure report I can now view the Rollbar Error Occurrence URL to directly access the Error information in Rollbar.

Error Report from Test Platform

Detailed Error in Rollbar

We now have a much clearer understanding of the exact reason that this specific test failed without having to re-run the test, or try to debug it locally. Rollbar gives you the full stack trace, line of code that caused the error, and much, much more.

Rollbar can also link back to the test execution report in most popular cloud vendors like Browserstack, Perfect and others.

Example Service Links Setup

Benefits>Benefits and Results

• Reduce your time to debug and surface errors from test automation by 50%.
• Direct link between the test execution report and error in Rollbar.
• Works with your existing test automation scripts.
• Detect and Fix errors before the test execution completes using Rollbar’s real time error detection capabilities.

To learn more about how your team can speed up test execution and debug times contact us today.