Ruchira ShuklaA test automation framework is a set of components that facilitates the execution of tests along with reporting the results of the test execution. However, the discovery of the right test automation framework can be super-challenging since there are so many options at your perusal. Picture this — When performing Selenium using Java, you have to choose from a gruelling list of 10 Java testing frameworks.
As far as test automation frameworks in Selenium Java are concerned, JUnit and TestNG are preferred in comparison to the other frameworks in the list. Some of the pertinent questions are:
How does JUnit stack up against TestNG in the TestNG vs JUnit comparison?
Is it possible to run JUnit tests with TestNG (and vice-versa)?
What annotations are present in JUnit and TestNG?
A side-by-side comparison of TestNG vs JUnit will give a clear indication of how the two prominent test automation frameworks stack up against each other. Unearthing the difference between JUnit and TestNG frameworks in Selenium WebDriver will help in deciding the best-suited framework for automation tests.
In this blog, we do a thorough TestNG vs JUnit comparison — learnings of which will help in arriving at a wise decision for choosing the ideal test automation framework for your project. In this TestNG vs JUnit comparison, we have used JUnit 5 (the latest version of the JUnit framework) for showcasing example implementation.
Introduction to JUnit &TestNG Test Automation Frameworks
Before we get started with the TestNG vs JUnit comparison, let’s look at some of the basic essentials of the two frameworks. JUnit is an open-source unit testing framework for Java that was first introduced in 1997. At the time of writing this blog, JUnit 5 is the latest version of the JUnit framework. As the Selenium framework supports Java; many QA engineers prefer using the JUnit framework for web automation testing. In case you are getting started with JUnit, you could check out our earlier blog that helps in setting up JUnit Environment for the first test.
TestNG is also a popular open-source testing framework for Java. TestNG was created in 2007 with a goal to cover a wider range of test categories — Unit Testing, Functional Testing, End-To-End Testing, Integration Testing, and more. As of writing this blog, the latest version of TestNG is 7.3.0. You can check out our introductory blog on how to create a TestNG project in Eclipse for realizing web automation testing with TestNG.
It is necessary to know the difference between JUnit and TestNG so that you can make an informed decision when it comes to choosing the ideal test automation framework. This is the primary reason why we came up with a head-on TestNG vs JUnit comparison.
Did you know? JSON is a common format for storing data, but it can be hard to read. Prettify JSON will indent the JSON file, make sure stuff is lined up in a similar manner, and will put closing parenthesis in the same space every time.
TestNG vs JUnit — Comparison of the best test automation frameworks
Both TestNG and JUnit are amongst the top automation frameworks in Java. Though they are hugely popular, the fact is that both frameworks have their own share of pros and cons. We would keep this discussion for a later blog since the TestNG vs JUnit can be a good reference to choose the framework for your automation project.
For the TestNG vs JUnit comparison, we have made use of the JUnit 5 (latest version of JUnit framework) instead of JUnit 4 framework. If you are still using the JUnit 4 framework with Selenium WebDriver, you can still execute JUnit 4 tests with JUnit 5 framework.
Shown below is the detailed difference between JUnit and TestNG frameworks in Selenium WebDriver:
Watch this video to learn how TestNG has become one of the most robust test automation frameworks and all you need to know to get started with TestNG in Selenium.
Test Suites in JUnit and TestNG
For starters, a test suite is a collection of test cases that lets you run the test cases simultaneously. It’s a logical grouping that can be treated as a single pack while execution.
JUnit 5 was re-designed to overcome the limitations of the previous JUnit versions (including JUnit 4). The concept of test suites was introduced in JUnit 5 since Junit 4 (and earlier versions of JUnit) did not provide the feature to create test suites.
Test suites in JUnit 5 are realized using the @ RunWith and @ Suite classes
[@RunWith](http://twitter.com/RunWith)(Suite.class)
[@Suite](http://twitter.com/Suite).SuiteClasses({
JUnitTestSuiteDemo1.class,
JUnitTestSuitDemo2.class
})
In TestNG, the test suite is defined in an xml file (e.g. testng.xml) The suite tag is defined in the file as shown below:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "[http://testng.org/testng-1.0.dtd](http://testng.org/testng-1.0.dtd)" >
<suite name="DemoTestSuite">
<test name="DemoTest" >
<classes>
<class name="demo.demo.TestSuiteDemo/>
</classes>
</test>
</suite>
With TestNG certification, you can challenge your skills in performing automated testing with TestNG and take your career to the next level.
Here’s a short glimpse of the TestNG certification from LambdaTest:
Annotations in JUnit and TestNG
Annotations in the test automation framework provide additional information about the class or the test method. For a quick recap, check out our detailed guide on TestNG annotations for Selenium automation.
Support for annotation can be considered as one of the vital points of TestNG vs JUnit comparison. Both JUnit and TestNG are annotation-based frameworks.
Most of the annotations in TestNG and JUnit offer the same functionality with a slight change in the naming nomenclature. The difference between JUnit and TestNG from an annotation standpoint is that TestNG has some additional annotations in comparison to JUnit.
Here is the TestNG vs JUnit comparison as far as annotations are concerned:
| SCRIPTION | TESTNG | JUNIT 5 |
|---|---|---|
| Marks the method as test method | @ Test | @ Test |
| The annotated method is executed before the first test method of the class | @ BeforeClass | @ BeforeAll |
| The annotated method is executed after all the test methods of the current class have been executed. | @ AfterClass | @ AfterAll |
| The annotated method is executed before each test method | @ BeforeMethod | @ BeforeEach |
| The annotated method is executed after each test method | @ AfterMethod | @AfterEach |
| The annotated method is executed before the suit. | @ BeforeSuite | NA |
| The annotated method is executed after suit. | @ AfterSuite | NA |
| The annotated method is executed before the test. | @ BeforeTest | NA |
| The annotated method is executed before the test. | @ AfterTest | NA |
| The annotated method is executed before the first test method of any of these groups. | @ BeforeGroups | NA |
| The annotated method is executed after the first test method of any of these groups. | @ AfterGroups | NA |
| Ignore Test | @ Test(Enable=false) | @ Disabled (In JUnit4 it is @ignore) |
| Expected exception | @ Test(expectedException=Arithmetic | |
| Exception.class) | @test(expected=Arithmetic | |
| Exception.class) | ||
| Timeout | @ Test(timeout = 1000) | @ Timeout |
Check out our earlier blog on JUnit Annotations in Selenium WebDriver to get detailed insights on annotations in the said framework.
Watch this video to learn about the TestNG Annotations and how they help provide better structure and readability to the code.
TestNG vs JUnit: Test case Management in TestNG and JUnit frameworks
Managing test case execution is a lot easier in TestNG in comparison to JUnit. In TestNG, QA engineers can group tests, ignore tests, parameterize tests, and write dependent tests in an efficient manner.
Let’s look at each of those points in this section of TestNG vs JUnit comparison:
- Grouping Test Cases
In TestNG, you can group the test cases by simply providing parameters in the @ Test annotation as shown below:
@Test(groups={“groupname1”,<”group2">..,<”groupN”>})
You can check out our detailed blog on how to group test cases in TestNG for more information on grouping tests in TestNG. The particular group(s) can be executed by providing the name(s) in the XML file under and tags as shown below:
<suite name="MyTestSuite">
<groups>
<run>
<include name="Dashboard"/>
</run>
</groups>
<test name="LoginTest">
<classes>
<class name="CheckUserLogin"/>
</classes>
</test>
<test name="UserHomePageTest">
<classes>
<class name="CheckUserHomePage"/>
</classes>
</test>
</suite>
Here, the tests under group “Dashboard” will be executed. JUnit does not provide a straightforward way for grouping the test cases.
- Ignore Test
There are scenarios where you might need to run a select set of tests from a huge test suite. The test cases that need not be executed have to be ignored. Ignoring tests in JUnit and TestNG is specifically useful when only a particular feature has to be tested.
TestNG and JUnit automation frameworks provide the feature where tests can be ignored. In TestNG, we can provide a parameter in @ Test annotation as shown below:
[@Test](http://twitter.com/Test)(enabled=false)
public void TestIgnoreDemo()
{
/* Implementation goes here */
}
In Junit 5, the @ ignore annotation is used for ignoring test cases:
[@Ignore](http://twitter.com/Ignore)
public void TestIgnoreDemo()
{
/* Implementation goes here */
}
- Parameterization
Parameterization in the test suite lets you run a certain test against different input values. Improved code reusability and readability are the major benefits of parameterization in automation testing. Parameterization can turn out to be a make or break deal in the TestNG vs JUnit comparison.
In TestNG, passing parameters to a testcase is straightforward. It uses @ Parameter annotation with the parameter to the given test method. The @ Parameters annotation should be used in case you want to pass more than one parameter to the method.
public class Demo {
[@Test](http://twitter.com/Test)
[@Parameters](http://twitter.com/Parameters)({"browser"})
public void demoTest(String browser)
{
System.out.println("The browser Is : " + browser);
}
}
The value of variable ‘browser’ is declared in the xml file (e.g. testng.xml) as shown below:
<suite name="ParameterizedDemo">
<test name="test1">
<parameter name="browser" value="chrome"/>
<parameter name="browser" value="IE"/>
<classes>
<class name="Demo" />
</classes>
</test>
</suite>
That’s not all. Data Providers in TestNG can also be used for realizing parameterized tests in TestNG.
In JUnit, parameter(s) can be passed using the @ParameterizedTest annotation:
[@ParamerizedTest](http://twitter.com/ParamerizedTest)
[@ValueSource](http://twitter.com/ValueSource)(strings = { "Chrome", "IE", "Safari" })
void DemoParameterize(String browser) {
void setBrowser("The browser is:"+browser));
}
On executing the above code, the test DemoParameterize will be triggered separately with the input parameters as Chrome, IE, and Safari. You can check out our detailed blog on Parameterization in JUnit with Selenium Driver to gather insights on how to use parameterized tests in JUnit.
- Dependent Test
Autonomous test design is considered as one of the best practices for Selenium test automation. However, there are cases where dependency between automation tests cannot be avoided at any cost. Test dependency refers to the ability of one test being dependent on another test method.
In TestNG, Dependent methods can be defined with the help of @ DependsOnMethods annotation.
public class DependentDemo {
[@Test](http://twitter.com/Test) (dependsOnMethods = { "Launcher" })
public void login() {
System.out.println("Login to the portal");
}
[@Test](http://twitter.com/Test)
public void Launcher() {
System.out.println("Launch the application");
}
}
Here, the test method login() is dependent on the Launcher() method. Here is the execution output:
JUnit does not support method dependency, hence TestNG scores a brownie point in the test dependency TestNG vs JUnit comparison 🙂
This JUnit certification establishes testing standards for those who wish to advance their careers in Selenium automation testing with JUnit.
Here’s a short glimpse of the JUnit certification from LambdaTest:
- Exception Handling
Exception handling in web automation testing refers to the ability of a test method to throw an exception when it encounters an error during the execution phase. Both JUnit and TestNG frameworks have support for exception handling.
In TestNG, exceptions are handled using the expectedException parameter in @ Test annotation
[@Test](http://twitter.com/Test)(expectedExceptions = ArithmeticException.class)
public void DivideByZero() {
int i = 10/0;
}In JUnit 5, the assertThrows API is used for throwing an exception
[@Test](http://twitter.com/Test)
public void whenDividerIsZero_thenDivideByZeroExceptionIsThrown() {
Calculator calculator = new Calculator();
assertThrows(DivideByZeroException.class, () -> calculator.divide(10, 0));
}Timeouts
Timeout in tests means that the tests should fail if they take longer than expected time for executing a test. This is a critical feature in web automation testing as an alarm is raised if the tests are not executed within a stipulated time period.
In TestNG, the timeout parameter which specifies the maximum execution time is added to the @ Test annotation.
[@Test](http://twitter.com/Test)(timeOut = 1000)
public void timeoutDemo() {
while (true);
}
Here is how timeouts in Selenium are handled in the JUnit 5 framework:
[@Test](http://twitter.com/Test)
public void timeoutDemo() throws InterruptedException {
Assertions.assertTimeout(Duration.ofMillis(1000), () -> Thread.sleep(10000));
}
- Customization of Test Names
There is no way to customize test names in the TestNG framework. In JUnit 5, we can provide a customized name to the tests with the help of @ DisplayName annotation. The same is shown below:
[@ParameterizedTest](http://twitter.com/ParameterizedTest)
[@ValueSource](http://twitter.com/ValueSource)(strings = { "Hello", "World" })
[@DisplayName](http://twitter.com/DisplayName)("Test Method to check that the inputs are not nullable")
void givenString_TestNullOrNot(String word) {
assertNotNull(word);
}
If you are looking for customized test names, JUnit should be the go-to test automation framework.
- Nesting of Test Cases
The @ Nested annotation in JUnit Jupiter can be used to mark nested classes that have to be included in test case execution. @ Nested tests give the QA engineer more capabilities to express the relationship among several groups of tests.
public class MyApp
{
[@BeforeAll](http://twitter.com/BeforeAll)
static void launchMyApp()
{
//the code to launch MyApp
}
[@Test](http://twitter.com/Test)
[@DisplayName](http://twitter.com/DisplayName)(“Testcase to create new user”)
void signUp()
{
//the code to create new user
}
[@Nested](http://twitter.com/Nested)
[@DisplayName](http://twitter.com/DisplayName)(value=“Nested class for Login to MyApp”)
class login
{
void verifyUser()
{
//the code to validate the user
}
void verifyNotNull()
{
//code to validate that username and password fields are not Null
}
}
[@AfterAll](http://twitter.com/AfterAll)
static void closeMyApp()
{
//the code to close MyApp
//releasing all used objects goes here
}
}
Grouping tests in TestNG is a common way for realizing nested tests in TestNG. As far as nested tests are concerned, TestNG scores high in the TestNG vs JUnit battle.
Did you know? Minify JSON is a JavaScript library that removes whitespace and comments from blocks of JSON-like content, while preserving its syntax.
Parallel Test Execution in TestNG and JUnit frameworks
Parallel testing in Selenium is one of the preferred ways to perform tests in parallel across different input combinations. Using a cloud-based Selenium Grid instead of a local Selenium Grid helps to achieve more from parallelism offered by the respective framework.
Both TestNG and JUnit frameworks support parallel test execution. In testNG, testers can assign the thread count in xml file (e.g. testng.xml) and run the tests parallely as shown below:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "[http://testng.org/testng-1.0.dtd](http://testng.org/testng-1.0.dtd)" >
<suite name="DemoTestSuite">
<test name="DemoTest" parallel="methods" thread-count="2">
<classes>
<class name="demo.demo.TestSuiteDemo/>
</classes>
</test>
</suite>
Our blog on parallel testing in JUnit is a good resource to gain insights on how to practically use parallelism in JUnit for Selenium automation testing. In testNG, parallel execution can be achieved at method, test, class, and instance levels.
Parallel test execution in JUnit was introduced in JUnit 5, however it is still in the experimental mode. Here are some ways in which parallel testing can be achieved with JUnit 5:
junit.jupiter.parallel.process = true
junit.jupiter.execution.parallelism = true
junit.jupiter.execution.parallel.enabled = true
junit.jupiter. execution = “parallel”
The value for junit.jupiter.execution.parallel.enabled configuration can be set by providing necessary parameters in the Maven Surefire plugin. Alternatively, you can also set the parameter in unit-platform.properties or by providing system properties to JVM.
This is how the difference between JUnitand TestNG pans out when evaluated on parameters like test execution, parallelism, parameterization, and more.
Do check out Android Emulator Online- Test your web and mobile apps on Android Emulators online. Ensure your apps are compatible across latest and legacy Android operating systems, devices, and browsers. Start Testing Now !!
Parallel Test Execution using TestNG on LambdaTest Grid
The true potential offered by parallel testing in JUnit and TestNG can be harnessed by running the tests on a cloud-based Selenium Grid (instead of a local Selenium Grid). There are numerous benefits offered by Selenium testing on the cloud.
For demonstration of parallel testing, we will use the cloud-based Selenium Grid by LambdaTest. LambdaTest offers both Selenium Automation testing with JUnit and Selenium Automation testing with TestNG frameworks. The browser capabilities are generated using the Lambda Test capability generator. The combination of username and access key is used to access the LambdaTest Grid.
Here are the two test cases used for the demonstration of parallel testing with TestNG:
TestCase — 1
Go to Selenium Playground Simple Form Demo
Enter the message in the text field
Click on the Show Message button
Verify the message on the right panel
TestCase — 2
Go to the Selenium Playground Simple Form Demo
Enter the value of a in the first textbox.
Enter the value of b in the second textbox.
Click on the Get Total button.
Verify the sum on the right panel.
Implementation (using TestNG Framework)
package Test;
import static org.testng.Assert.assertTrue;
import java.net.MalformedURLException;
import java.net.URL;
import org.openqa.selenium.By;
import org.openqa.selenium.remote.DesiredCapabilities;
import org.openqa.selenium.remote.RemoteWebDriver;
import org.testng.annotations.AfterClass;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;
public class Lambdatest {
public RemoteWebDriver driver = null;
public static String username = "user-name";
public static String accessKey = "access-key";
@BeforeClass
public void setup() throws Exception {
DesiredCapabilities capabilities = new DesiredCapabilities();
capabilities.setCapability("platform", "Windows 10");
capabilities.setCapability("browserName", "Chrome");
capabilities.setCapability("version", "88.0");
capabilities.setCapability("resolution","1024x768");
capabilities.setCapability("build", "Lambdatest");
capabilities.setCapability("name", "Lambdatest");
capabilities.setCapability("network", true); // To enable network logs
capabilities.setCapability("visual", true); // To enable step by step screenshot
capabilities.setCapability("console", true);
capabilities.setCapability("user", username);
capabilities.setCapability("accesskey", accessKey);
try {
driver= new RemoteWebDriver(new URL("https://"+username+":"+accessKey+"@hub.lambdatest.com/wd/hub"), capabilities);
} catch (MalformedURLException e) {
System.out.println("Invalid grid URL");
}
}
@Test
public void testAddition() throws Exception {
try {
driver.get("");
driver.findElement(By.id("sum1")).clear();
driver.findElement(By.id("sum1")).sendKeys("100");
driver.findElement(By.id("sum2")).clear();
driver.findElement(By.id("sum2")).sendKeys("200");
driver.findElement(By.className("btn btn-dark selenium_btn")).click();
String strResult=driver.findElement(By.id("addmessage")).getText();
assertTrue(strResult.equals("300"),"Pass");
} catch (Exception e) {
System.out.println(e.getMessage());
}
}
@Test
public void testUserMessage() throws Exception {
try {
String strMessage="Hello! Welcome to the Lambda Test";
String actualMsg="";
driver.get("");
driver.findElement(By.id("user-message")).clear();
driver.findElement(By.id("user-message")).sendKeys(strMessage);
driver.findElement(By.id("showInput")).click();
actualMsg=driver.findElement(By.id("message")).getText();
assertTrue(actualMsg.equals(strMessage),"Pass") ;
} catch (Exception e) {
System.out.println(e.getMessage());
}
}
@AfterClass
public void tearDown() throws Exception{
driver.quit();
}
}
Here is the execution snapshot from LambdaTest:
LambdaTest also provides the video recording of the test case execution
We have not covered parallel testing with JUnit 5 here since, the feature is still in the experimental mode. In case you are using JUnit 4, do check out how to perform parallel testing with JUnit 4 and Selenium.
Conclusion
While JUnit 4 has several limitations in test case management, JUnit 5 tried to overcome those and also added new features in the process. Though there is a difference between JUnit and TestNG as far as annotations are concerned, the point is that both the frameworks are equally capable for usage in Selenium automation testing.
We have to wait and watch how JUnit 5 (and the subsequent versions of JUnit) evolve over a period of time. Overall, there is a thin line of difference between JUnit and TestNG and the choice for the test automation framework purely depends on the requirements of the project.
In the TestNG vs JUnit battle, my overall vote would go for TestNG since parallel testing in Selenium can be achieved with ease using the TestNG framework.
Irrespective of whether you choose JUnit or TestNG, the true potential of the corresponding framework can only be exploited by running the tests on a cloud-based Selenium Grid like LambdaTest. The major reason being parallel tests can be executed on different browser and OS combinations at blazing fast speeds.
Which framework do you prefer in the TestNG vs JUnit battle? Do leave your reasons in the comments section of this blog…
Happy Testing 🙂
