While automating critical applications like banking apps, it’s important to implement error handling and validation checks to ensure that the application is secure and reliable for the bank’s customers. Additionally, clear error messages and notifications should be provided to assist users in troubleshooting the issue.

Unforeseen circumstances or anomalies can occur during software development, and it is crucial to handle them efficiently to maintain program stability. To achieve this, exception handling can be implemented in the code.

Since these are an inherent part of any software, an exception test is a critical and mandatory requirement. By writing exception tests, you can ensure that your application behaves as expected without impacting customers when faced with such exceptional scenarios. This is also critical for ensuring business continuity because it reduces risk and costs and, in some cases, helps to meet compliance and regulatory testing requirements.

TestNG is the framework that is most frequently used to run exception tests.

This blog post will explore some of the common exceptions encountered during automation testing, demonstrate how to handle them by writing exception tests in TestNG, and verify that your code performs as intended.

Do you use a Mac and want to run the test in Internet Explorer? This article explores how to test Internet Explorer for Mac.

What are Exceptions?

Any event that interrupts the normal flow of a program’s execution is called an Exception. These are typically caused by errors or unusual conditions, such as invalid input, a file that cannot be found, or divide-by-zero or element locator/interaction-related issues while working with Selenium.

When an exception occurs, the program will typically halt execution or simply crash or produce incorrect results.

Exception handling allows programs to recover from such errors, provide more informative error messages to users, and ensure that critical resources are correctly released even when an error occurs.

Let us take this example flow to understand how exceptions and exception handling work.

1. Consider you are working to automate a web page. For this, you would need to use the object of a WebDriver to interact with the page.

2. As a practice, the following steps would be to perform desired actions on the webpage and then close the browser and terminate the driver instance.

3. But imagine a scenario where for some reason, the browser gets closed between execution, or you do so manually. Then, the code tries to continue and interact with some web element that is supposed to be present on the webpage.

4. Such a scenario would lead to an exception as the browser is already closed. The output, in this case, would show you a SessionNotFound or WebDriver exception as the browser session is terminated.

5. To handle such exceptions, we can use exception handling and maybe relaunch the browser in such a scenario to continue execution and get desired results.

Commonly encountered Exceptions

Here are a few examples of exceptions that we may encounter when writing automation scripts with Selenium for web automation or when writing code in general.

NullPointerException

When trying to refer to some object or variable that has no reference or a null reference.

ArrayIndexOutOfBoundsException

Occurs while working with lists or arrays when a program tries to access an element that does not exist and is beyond array length.

IllegalArgumentException

When an illegal or corrupt argument is passed to a function.

IOException

Occurs due to issues on input/output flows, like accessing a file or opening/closing it.

NoSuchElementException

Occurs when the element the code is trying to access is not present in the DOM.

ElementNotVisibleException

Occurs when the element is present on DOM but is not actually visible on the webpage for interaction.

ElementNotInteractableException

Occurs when we try to interact with a disabled or inactive web element.

StaleElementReferenceException

Occurs when we try to access some element with an older reference that is no longer valid due to some actions.

TimeoutException

Occurs when the required action takes too long to be completed, like loading a page, waiting for some web element, etc.

NoSuchWindowException

Occurs when working with multiple windows when the required window is not available.

InvalidSelectorException

Occurs when the web element selector is not valid. This can be due to syntax errors or invalid selector values.

WebDriverException

Generic exception when working with WebDriver. This can occur due to WebDriver or browser-related problems in the automation script.

Exception tests in TestNG

An exception test is a test case that confirms a particular exception is raised whenever a certain code block is executed. Exception testing is an important part of software testing, as it helps to validate that the application behaves as expected when faced with exceptional scenarios. It also helps to improve code and product quality by identifying such issues. This, in turn, also increases test coverage and provides better results.

To write such a test, the TestNG testing framework is used.

TestNG is an open-source automation testing framework for Java built on the capabilities of JUnit and NUnit. It helps to write test cases in a well-structured, easy-to-maintain way using TestNG annotations, attributes, and other features.

This tutorial dive deep into web testing to help you understand its life cycle, elements, angles, the role of automation, and more.

expectedExceptions

Using the expectedExceptions attribute of the @Test annotation, we can write exception tests in TestNG. This attribute accepts a single exception class or a list of exception classes that are expected to be thrown from the test case. The test will succeed if any of the predefined exceptions are raised. The test will fail if a different exception or no exception is thrown. Using the expectedExceptionsMessageRegExp attribute, we can also confirm the exception message.

For example, a TestNG exception test for the scenario mentioned above would look like this below

@Test(expectedExceptions = ElementNotInteractableException.class)
    public void testButtonNotInteractable() {
     //code to fill transfer details and
     //try to click on submit button
    }

It verifies that an ElementNotInteractableException is thrown when a user tries to click on the transfer button. If no exception or any other exception is thrown, it will fail.

Advantages of using Exception tests in TestNG

There are several advantages of using exception tests in TestNG, a few of them are:

• Improved quality

Exception tests help verify that the code behaves as expected when faced with unexpected scenarios and can handle them gracefully. This helps to improve application quality and stability.

• Early issue detection

With exception testing, ROI improves significantly as issues can be detected early on. This saves time, money, and effort to fix any such issue if they reach production.

• Increased confidence

Having exception tests in place also gives testers more confidence while testing any new changes as it ensures nothing on existing functionality breaks and all unwanted situations are catered to.

• Easy to maintain and write cases

TestNG makes writing the cases very easy and well-structured using annotations. We can simply write one by providing the expectedExceptions attribute of the @Test annotation.

• Better insight into the code

Exception tests provide an understanding of how the application would behave in exceptional scenarios. This can be used to verify assumptions around expected behavior and thus improve overall understanding.

How to write Exception tests in TestNG?

TestNG helps to validate the exceptions thrown by the test using the expectedExceptions attribute within @Test annotation.

expectedExceptions can take a single exception or a list of exceptions to validate from. If the exception raised from the test case is one of those mentioned above, then the test case passes; else, it fails.

Verify exception class

There are two ways to define a single exception to be validated. Both these are just different ways to represent the same thing.

• Pass the exception class name directly.

   @Test(expectedExceptions = ElementNotInteractableException.class)

• Pass the exception class name within curly brackets.

  @Test(expectedExceptions = {ElementNotInteractableException.class})

There might be scenarios where the same code could lead to different exceptions under different circumstances. In such a case, we can also pass the list of all such expected exceptions by enclosing them within curly brackets.

   @Test(expectedExceptions = {ElementNotInteractableException.class, NoSuchElementException.class})

In this, the test case passes if the raised exception is one amongst the list and fails if it is not present in the list or no exception is thrown at all from the automation code.

Upload your app for testing within seconds using the LambdaTest cloud and perform mobile app testing right away. Find bugs early on, improve performance, quality, user experience and make the most of mobile appl testing on LambdaTest.

Verify exception message

TestNG with @Test annotation also provides support to assert the exception message using the expectedExceptionsMessageRegExp attribute. We can have this message in the code from where the exception is thrown.

Asserting the exception message enables us to confirm that the exception is raised for the correct reasons, which we expected.

To understand this better, consider insufficient funds in the source account part of the bank application example we took in the introduction. In this case, an InsufficientFundsException can be raised for two reasons.

• The source has no funds, or

• The source has insufficient funds to complete the transfer.

In both cases, the exception raised would be the same, and the exception test would pass. Asserting exception messages enables us to be more specific and helps in verifying the corner cases that the message raised is as expected or not. Thus confirming what exactly leads to failure in this case, the message could specify the same — no funds or insufficient funds, although the exception class remains the same.

 @Test(expectedExceptions = { InsufficientFundsException.class }, expectedExceptionsMessageRegExp = "Source has fewer than requested funds")
    public void testFewerFunds() {}


    @Test(expectedExceptions = { InsufficientFundsException.class }, expectedExceptionsMessageRegExp = "Source has zero funds")
    public void testZeroFunds() {}

We can also verify the exception messages with the help of Regular Expressions using .*., depending upon the position of the regular expression we can use it to make pattern matching such as starts-with, contains, and ends-with while verifying the exception message.

 @Test(expectedExceptions = { InsufficientFundsException.class }, expectedExceptionsMessageRegExp = "Source has .*")

This means the exception message should have the word Source followed by certain other words.

Let us now move forward and see how the implementation of exception tests in TestNG works by taking examples.

Run your Selenium Automation Testing scripts on the LambdaTest cloud grid. Test on 3000+ desktop & mobile environments. Try it for free!

Demonstration: Exception tests in TestNG

We will be creating a Maven project. It will use Selenium WebDriver and TestNG dependencies to automate web page flows and test case execution for exception tests, respectively. It is highly recommended to use the latest versions of the dependencies for the best results.

After creating the project and adding said dependencies, the pom.xml should look like this.

<project xmlns="http://maven.apache.org/POM/4.0.0"
           xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
           xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
           <modelVersion>4.0.0</modelVersion>
           <groupId>ExceptionTest</groupId>
           <artifactId>ExceptionTest</artifactId>
           <version>0.0.1-SNAPSHOT</version>
           <build>
                   <sourceDirectory>src</sourceDirectory>
                   <plugins>
                           <plugin>
                                   <artifactId>maven-compiler-plugin</artifactId>
                                   <version>3.8.1</version>
                                   <configuration>
                                           <release>16</release>
                                   </configuration>
                           </plugin>
                   </plugins>
           </build>
           <dependencies>
                   <dependency>
                           <groupId>org.seleniumhq.selenium</groupId>
                           <artifactId>selenium-java</artifactId>
                           <version>4.8.0</version>
                   </dependency>
                   <dependency>
                           <groupId>org.testng</groupId>
                           <artifactId>testng</artifactId>
                           <version>7.7.1</version>
                           <scope>test</scope>
                   </dependency>
           </dependencies>
    </project>

Next, add a Java test file and name it ExceptionTest.java. We will add different test cases to this to understand exception tests with different examples.

In this blog on exception tests in TestNG, we will mostly take examples of exceptions faced during Selenium automation with TestNG. Here, we are using the LambdaTest Cloud Grid to run our tests.

LambdaTest is a digital experience testing cloud, a highly scalable, reliable, and easy-to-use platform that provides support for over 3000+ browsers and OS to perform TestNG testing on Selenium Cloud Grid.

To kick start Selenium automation testing with TestNG, refer to our documentation TestNG with Selenium.

Automate Cypress testing and perform browser automation testing with LambdaTest. Our cloud infrastructure has 3000+ desktop & mobile environments. Try for free!

To stay up-to-date on tutorials related to Selenium testing, Cypress testing, and more, subscribe to the LambdaTest YouTube Channel.

Using the RemoteWebDriver configuration to access a webpage on the cloud platform and after adding different test cases to demonstrate exception testing, the final ExceptionTest.java would look like the below.

 package test;


    import java.net.*;
    import java.util.HashMap;
    import java.util.concurrent.TimeUnit;


    import org.openqa.selenium.*;
    import org.openqa.selenium.chrome.ChromeOptions;
    import org.openqa.selenium.remote.RemoteWebDriver;
    import org.testng.annotations.*;


    public class ExceptionTest {


       public RemoteWebDriver driver = null;


       String username = System.getenv("LT_USERNAME") == null ? "<lambdatest_username>" : System.getenv("LT_USERNAME");
       String accessKey = System.getenv("LT_ACCESS_KEY") == null ? "<lambdatest_accesskey>" : System.getenv("LT_ACCESS_KEY");


       @BeforeTest
       public void setup() {
           try {
               ChromeOptions chromeOptions = new ChromeOptions();
               chromeOptions.setPlatformName("Windows 10");
               chromeOptions.setBrowserVersion("110.0");


               HashMap<String, Object> ltOptions = new HashMap<String, Object>();
               ltOptions.put("build", "Exception Test in TestNG");
               chromeOptions.setCapability("LT:Options", ltOptions);


               driver = new RemoteWebDriver(
                       new URL("https://" + username + ":" + accessKey + "@hub.lambdatest.com/wd/hub"), chromeOptions);
           } catch (MalformedURLException e) {
               e.printStackTrace();
           }


           driver.get("https://www.lambdatest.com/selenium-playground/");
       }


       @AfterTest
       public void tearDown() {
           driver.quit();
       }


       // This case will pass as same exception is thrown
       // due to invalid web element selector
       @Test(expectedExceptions = TimeoutException.class)
       public void testSingleException_passed() {
           driver.manage().timeouts().pageLoadTimeout(10, TimeUnit.MILLISECONDS);
           driver.get("https://ecommerce-playground.lambdatest.io/");
       }


       // This case will fail as web element selector is valid
       // hence no exception is raised
       @Test(expectedExceptions = TimeoutException.class)
       public void testSingleException_failed() {
           driver.get("https://ecommerce-playground.lambdatest.io/");
       }


       // This case will pass as exception thrown
       // is one of the expected list i.e. NoSuchWindowException
       @Test(expectedExceptions = { NoSuchElementException.class, NoSuchWindowException.class })
       public void testMultipleException_NoSuchWindowException_passed() {
           driver.switchTo().window("new_window");
       }


       // This case will pass as exception thrown
       // is one of the expected list i.e. NoSuchElementException
       @Test(expectedExceptions = { NoSuchElementException.class, NoSuchWindowException.class })
       public void testMultipleException_NoSuchElementException_passed() {
           driver.findElement(By.xpath("//*[@class='st_heading_1']"));
       }


       // This case will fail as exception raised is not on the list
       // i.e. TimeoutException
       @Test(expectedExceptions = { NoSuchElementException.class, NoSuchWindowException.class })
       public void testMultipleException_failed_1() {
           driver.manage().timeouts().pageLoadTimeout(10, TimeUnit.MILLISECONDS);
           driver.get("https://ecommerce-playground.lambdatest.io/");
       }


       // This case will fail as none of the expected exception is raised
       @Test(expectedExceptions = { NoSuchElementException.class, NoSuchWindowException.class })
       public void testMultipleException_failed_2() {
           driver.findElement(By.xpath("//*[@class='st_heading']"));
       }


       // This case will pass as expected exception message is same.
       @Test(expectedExceptions = NoSuchElementException.class, expectedExceptionsMessageRegExp = "(?s).*no such element.*")
       public void testExceptionMessage_passed() {
           driver.findElement(By.xpath("//*[@class='st_heading_1']"));
       }


       // This case will fail as expected exception message is not same.
       @Test(expectedExceptions = NoSuchWindowException.class, expectedExceptionsMessageRegExp = "(?s).*no such element.*")
       public void testExceptionMessage_failed() {
           driver.switchTo().window("new_window");
       }
    }

Let’s start to understand the code step by step now.

Code Walkthrough:

We will first do a walkthrough for the common code used to set up and launch the browser and close it at the end of the test execution. This will be followed by different test case examples to understand what we have learned so far in this blog about exception tests in TestNG.

Step 1: Create an instance of RemoteWebDriver. This is required as we will be executing the code on the cloud Selenium Grid.

Step 2: Since we are using the LambdaTest platform, we need to provide the username and access key for our user to connect with the cloud grid. You can find these details under LambdaTest Profile Section after your account is created.

You can configure the username and access key as environment variables on your system and fetch them in the automation script.

To configure these following commands can be used

For macOS and Linux:

 export LT_USERNAME=LT_USERNAME
    export LT_ACCESS_KEY=LT_ACCESS_KEY

For Windows:

set LT_USERNAME=LT_USERNAME
    set LT_ACCESS_KEY=LT_ACCESS_KEY

Step 3: Create a method named setup() and annotate it with @BeforeTest TestNG annotation to be able to execute it before each test case for driver setup and opening the webpage

In this function, we start by creating an object of ChromeOptions class, which is used to define browser properties like browser version, OS version, etc.

This is followed by steps to define some variables for LambdaTest execution. These variables are used to set values like build, name, or any other browser property, which will help to identify the run on the dashboard or modify behavior. This is done with the help of a HashMap variable, which is then passed to chromeOptions.

An additional benefit of using LambdaTest is that all of these browser properties can be easily retrieved using the LambdaTest Automation Capabilities Generator. You only need to select the required operating system and browser combination and versions, and the rest is done for you by this utility, and you have the code ready to use. This reduces the possibility of error and provides more valid combinations and data.

Next, using all these properties, we create an instance of RemoteWebDriver for connecting to the LambdaTest cloud grid to start execution. Your LambdaTest username and the access key will be used in this step.

Finally, navigate to the required URL.

Step 4: Add a new method as tearDown() and annotate it with @AfterTest annotation. This function will be used to close the browser and quit the driver session after each test case.

This completes the basic code setup. Let’s start to see some working examples now.

Black Box testing? Don’t worry; we will be covering what is Black box testing, merits, demerits, types & techniques.

Exception test for TimeoutException

Add the first test case and name it testSingleException_passed(). In this test case, we will try to simulate TimeoutException by setting the driver timeout to a very low value. As a result, when we try to navigate to a new URL, the driver throws an exception as the page load is not completed. To make the test case pass, the value for the expectedExceptions attribute is set as TimeoutException.class.

As you can see, we have the pageLoadTimeout() to 10 milliseconds and then try to get a new URL. This is expected to throw a Timeout exception as the website would not be loaded in this time frame.

Now try to execute the test case and see if it fails or passes.

You can notice that this case is passed since the expected exception matches the one thrown by the code.

To understand this better, add another test case, testSingleException_failed(). This case is exactly similar to the previous one with just one difference: the pageLoadTimeout() is not updated, unlike earlier, and hence the webpage is loaded successfully in the expected time. Hence, no exception is raised from the test case.

Here, you can see we are still expecting a TimeoutException. This should lead to test failure.

From the test execution result above, you can see that the test case failed as expected to get an exception, but none was thrown.

Test your Puppeteer test scripts online. Deploy a quality build faster by running automated checks across over 3000+ browsers and OS combinations with the LambdaTest cloud. Try for free!

Exception test for NoSuchElementException and NoSuchWindowException

In this section, we take examples where multiple exceptions are passed as a list to the expectedExceptions attribute. We will take two passed test case scenarios to understand that the same test case with the same list passes when any of the given exceptions are raised from the code.

Add a test case testMultipleException_NoSuchWindowException_passed(). This test will showcase how we can pass multiple exceptions as a list and test cases getting passed when the thrown exception is one amongst the list.

In this test case, you can see we are expecting to get either NoSuchElementException or NoSuchWindowException when trying to switch the window.

Executing this test would result in success as the window name given is invalid and would make the code throw NoSuchWindowException, which is on the expectedExceptions list.

Similarly, we can add another test case as testMultipleException_NoSuchElementException_passed(), keep the expectedExceptions list the same, and just update the code to raise NoSuchElementException due to an invalid web element locator and the test case is passed this time as well.

While working with multiple exception tests in TestNG, the test case could fail due to two reasons related to expected exceptions.

• No exception is thrown at all from the test case.

• The exception raised is not on the expected list.

Let’s take a look at the implementation of these.

In this test case, the expectedExceptions list is still the same as the previous one, but the code is updated to throw a Timeout exception.

Executing this would lead to failure because the thrown exception is not one of the given lists, and the output would look like the one below.

Another reason for failure in this implementation could be no exception thrown at all. For this, we keep the list the same and update the code to fetch a valid web element from the webpage.

Executing this would give results like the one below.

You can see from the highlighted error message that the test case failed as it was expecting an exception from the list, but none was thrown.

Which are the most wanted automation testing tools in 2023 that have climbed the top of the ladder so far? Let’s take a look on top test automation tools.

Exception test with Exception Messages

In this section, we take examples of test cases to understand how exception messages can be asserted in exception tests in TestNG.

In this case, we have added the code which should throw a NoSuchElementException with an exception message containing keywords no such element and the same is set to be verified as the expected message for the exception using the expectedExceptionsMessageRegExp attribute of @Test annotation. Since this exception message is dynamic depending on the web element locator passed, we use regex to pass the expected message.

On executing the above case, you can see that the test output is passed since the expected and actual exception message is the same.

To demonstrate the failure due to TestNG assertion in the message, we reuse the same test case but update the execution code inside it to make it raise a different exception. A different exception would lead to another message and hence showcase test case failure in this example.

In the above code, we have updated the expected exception to be a valid one as there is no such window with the name new window, but the exception message is not updated. Let’s run it to see the output.

From the output error, it is evident that it failed due to the difference in the expected and actual exception messages.

Execution

Since we are using the LambdaTest cloud grid for the execution, results can be viewed on the LambdaTest Dashboard as well

You can also view details of all the test cases executed by navigating to the Build section under Automation.

If you want to test your knowledge around TestNG concepts, its configuration, annotations, and more, you can take TestNG certification by LambdaTest. LambdaTest TestNG certification is an industry-recognized certification that validates an individual’s proficiency in using the TestNG testing framework for automated testing.

Run your Jest automation tests in massive parallel across multiple browser and OS combinations with LambdaTest.

Conclusion

That concludes our blog on exception tests in TestNG. Throughout this post, we have gained knowledge about exception tests in TestNG, their purpose, and how to implement them with practical examples. By applying these concepts in your automation testing with TestNG in Selenium, you can achieve optimal results when encountering unforeseen circumstances. It’s time for you to dive in and start testing some exceptions!