An exhaustive test-suite comprises of many test cases that test different features on various combinations of browsers, platforms, and devices Though it is recommended not to skip tests, there are cases where you may want to stop test suite after n test failures, (n is the failure threshold value) number of the test fails while performing test automation.

The option comes handy in scenarios where there are inter-dependent test cases and failure of one test case causes its dependent test cases to fail. This also avoids the execution of unnecessary Selenium test automation cases that are bound to fail (due to failure of dependent tests).

In case you are new to pytest framework checkout this video to know more about pytest framework & its installation.

In this Python tutorial, I’ll explore how to stop test suite after n test failures in pytest using the @pytest.mark.incremental decorator and maxfail command-line option.

Using @pytest.mark.incremental Decorator To Stop Test Suite After N Test Failures

The @pytest.mark.incremental decorator is used for skip test in Python with pytest. It is used for marking all the tests that are expected to fail so that the dependent tests are not attempted for execution. It is also used to stop test suite after n test failures in pytest For instance, there are two tests in a class i.e. Test_1 & Test_2 and Test_2 is dependent on Test_1. If Test_1 fails, Test_2 need not be executed unnecessarily as it is bound to fail and its trackback adds no insight.

PyTest provides hook implementations that work together to stop incremental-marked tests in a class. Here is the conftest.py file to introduce the incremental marker used to stop test suite after n test failures.

conftest.py (to skip test in Python with pytest)

import pytest
from selenium import webdriver
import urllib3
import warnings

def pytest_runtest_makereport(item, call):
    if "incremental" in item.keywords:
        if call.excinfo is not None:
            parent = item.parent
            parent._previousfailed = item

def pytest_runtest_setup(item):
    previousfailed = getattr(item.parent, "_previousfailed", None)
    if previousfailed is not None:
        pytest.xfail("previous test failed (%s)" % previousfailed.name)

The @pytest.mark.incremental decorator is used in cases where there is a dependency between tests and if a particular test (in a class) fails, the subsequent tests are marked as expected to fail (or marked as xfail) in order to skip the test in Python with pytest.

This PyTest Tutorial for beginners and professionals will help you learn how to use PyTest framework with Selenium and Python for performing Selenium automation testing.

Leverage the power of LambdaTest to execute your Jest automation tests at scale, simultaneously on various operating system and browser configurations. Learn more about LambdaTest's capabilities.

How To Skip Test In Python With pytest Using @pytest.mark.incremental?

To demonstrate the usage of @pytest.mark.incremental to skip the test in Python with pytest, we take an automated browser testing example that contains four test scenarios. Class Test_Scenario_1 consists of three Selenium test automation cases and Test_Scenario_2 consists of one test case.

Fixture with scope as the class is used to insatiate the Chrome browser instance. Implementation to skip test in Python with pytest, is also included in conftest.py.

Conftest.py

import pytest
from selenium import webdriver
import urllib3
import warnings

def pytest_runtest_makereport(item, call):
    if "incremental" in item.keywords:
        if call.excinfo is not None:
            parent = item.parent
            parent._previousfailed = item

def pytest_runtest_setup(item):
    previousfailed = getattr(item.parent, "_previousfailed", None)
    if previousfailed is not None:
        pytest.xfail("previous test failed (%s)" % previousfailed.name)

@pytest.fixture(scope="class")
def driver_init(request):
    web_driver = webdriver.Chrome()
    request.cls.driver = web_driver
    yield
    web_driver.close()
    # web_driver.quit()

test_step.py

Facebook
Twitter
LinkedIn
How To Stop Test Suite after N Test Failures in Pytest?

Himanshu Sheth
Posted On: May 26, 2020

view count61133 Views

Read time7 Min Read

Home>Blog>How To Stop Test Suite after N Test Failures in Pytest?
This article is a part of our Content Hub. For more in-depth resources, check out our content hub on Selenium pytest Tutorial.

An exhaustive test-suite comprises of many test cases that test different features on various combinations of browsers, platforms, and devices Though it is recommended not to skip tests, there are cases where you may want to stop test suite after n test failures, (n is the failure threshold value) number of the test fails while performing test automation.

The option comes handy in scenarios where there are inter-dependent test cases and failure of one test case causes its dependent test cases to fail. This also avoids the execution of unnecessary Selenium test automation cases that are bound to fail (due to failure of dependent tests).

In case you are new to pytest framework checkout this video to know more about pytest framework & its installation.

Youtube Thubnail
In this Python tutorial, I’ll explore how to stop test suite after n test failures in pytest using the @pytest.mark.incremental decorator and maxfail command-line option.

TABLE OF CONTENT

Using @pytest.mark.incremental decorator To Stop Test Suite after N Test Failures
How To Skip Test In Python With pytest Using @pytest.mark.incremental?
Using maxfail Command Line Option To Stop Test Suite after N Test Failures
How To Skip Test In Python With pytest Using maxfail?
Using @pytest.mark.incremental Decorator To Stop Test Suite After N Test Failures
The @pytest.mark.incremental decorator is used for skip test in Python with pytest. It is used for marking all the tests that are expected to fail so that the dependent tests are not attempted for execution. It is also used to stop test suite after n test failures in pytest For instance, there are two tests in a class i.e. Test_1 & Test_2 and Test_2 is dependent on Test_1. If Test_1 fails, Test_2 need not be executed unnecessarily as it is bound to fail and its trackback adds no insight.

PyTest provides hook implementations that work together to stop incremental-marked tests in a class. Here is the conftest.py file to introduce the incremental marker used to stop test suite after n test failures.

conftest.py (to skip test in Python with pytest)

import pytest
from selenium import webdriver
import urllib3
import warnings

def pytest_runtest_makereport(item, call):
    if "incremental" in item.keywords:
        if call.excinfo is not None:
            parent = item.parent
            parent._previousfailed = item

def pytest_runtest_setup(item):
    previousfailed = getattr(item.parent, "_previousfailed", None)
    if previousfailed is not None:
        pytest.xfail("previous test failed (%s)" % previousfailed.name)
view rawconftest.py hosted with ❤ by GitHub
The @pytest.mark.incremental decorator is used in cases where there is a dependency between tests and if a particular test (in a class) fails, the subsequent tests are marked as expected to fail (or marked as xfail) in order to skip the test in Python with pytest.

This PyTest Tutorial for beginners and professionals will help you learn how to use PyTest framework with Selenium and Python for performing Selenium automation testing.


How To Skip Test In Python With pytest Using @pytest.mark.incremental?
To demonstrate the usage of @pytest.mark.incremental to skip the test in Python with pytest, we take an automated browser testing example that contains four test scenarios. Class Test_Scenario_1 consists of three Selenium test automation cases and Test_Scenario_2 consists of one test case.

Fixture with scope as the class is used to insatiate the Chrome browser instance. Implementation to skip test in Python with pytest, is also included in conftest.py.

Conftest.py

import pytest
from selenium import webdriver
import urllib3
import warnings

def pytest_runtest_makereport(item, call):
    if "incremental" in item.keywords:
        if call.excinfo is not None:
            parent = item.parent
            parent._previousfailed = item

def pytest_runtest_setup(item):
    previousfailed = getattr(item.parent, "_previousfailed", None)
    if previousfailed is not None:
        pytest.xfail("previous test failed (%s)" % previousfailed.name)

@pytest.fixture(scope="class")
def driver_init(request):
    web_driver = webdriver.Chrome()
    request.cls.driver = web_driver
    yield
    web_driver.close()
    # web_driver.quit()
view rawConftest.py hosted with ❤ by GitHub
test_step.py

#Using @pytest.mark.incremental decorator to stop test suite after n test failure using Selenium test automation in Python with pytest 
import pytest
import pytest_html
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.keys import Keys
import time
from time import sleep
import sys
import urllib3
import warnings
# @pytest.mark.incremental Stop Test Suite after N Test Failures in Pytest
@pytest.mark.usefixtures("driver_init")
@pytest.mark.incremental
class Test_Scenario_1:
    def test_1(self):
        self.driver.get('https://www.lambdatest.com/blog/')
        self.driver.maximize_window()

        expected_title = "LambdaTest | A Cross Browser Testing Blog"
        assert expected_title ==  self.driver.title
        time.sleep(5)

    def test_2(self):
        self.driver.get('https://www.google.com/')
        self.driver.maximize_window()
        title = "Google"
        assert title == self.driver.title

        search_text = "LambdaTest"
        search_box = self.driver.find_element_by_xpath("//input[@name='q']")
        search_box.send_keys(search_text)

        time.sleep(5)
        search_box.submit()

        time.sleep(5)

        # Click on the LambdaTest HomePage Link
        # This test will fail as the titles will not match
        title = "Cross Browser Testing Tools | Free Automated Website Testing | LambdaTest_1"
        lt_link = self.driver.find_element_by_xpath("//h3[.='LambdaTest: Cross Browser Testing Tools | Free Automated ...']")
        lt_link.click()

        time.sleep(10)
        assert title == self.driver.title   
        time.sleep(2)

    # As test_2 fails, test_3 will xfail i.e. skipped and all subsequent tests in the same class
    # will be marked as xfailed
    def test_3(self):
        self.driver.get('https://www.lambdatest.com/')
        self.driver.maximize_window()

        expected_title = "Cross Browser Testing Tools | Free Automated Website Testing | LambdaTest"
        assert expected_title ==  self.driver.title
        time.sleep(5)

@pytest.mark.usefixtures("driver_init")
@pytest.mark.incremental
class Test_Scenario_2:
    def test_4(self):
        self.driver.get('https://lambdatest.github.io/sample-todo-app/')
        self.driver.maximize_window()

        self.driver.find_element_by_name("li1").click()
        self.driver.find_element_by_name("li2").click()

        title = "Sample page - lambdatest.com"
        assert title ==  self.driver.title

        sample_text = "Happy Testing at LambdaTest"
        email_text_field =  self.driver.find_element_by_id("sampletodotext")
        email_text_field.send_keys(sample_text)
        time.sleep(5)

        self.driver.find_element_by_id("addbutton").click()
        time.sleep(5)

        assert self.driver.find_element_by_xpath("//span[.='Happy Testing at LambdaTest']").text == sample_text

As seen in the implementation, incremental testing is applied to class Test_Scenario_2 and Test_Scenario_2 via the @pytest.mark.incremental decorator.

......................................
......................................
from time import sleep
import sys
import urllib3
import warnings
......................................
......................................

@pytest.mark.usefixtures("driver_init")
@pytest.mark.incremental
class Test_Scenario_1:
 ..................
 ..................
 def test_2(self):
        self.driver.get('https://www.google.com/')
        self.driver.maximize_window()
        title = "Google"
        assert title == self.driver.title

        search_text = "LambdaTest"
        search_box = self.driver.find_element_by_xpath("//input[@name='q']")
        search_box.send_keys(search_text)

        time.sleep(5)
        search_box.submit()

        time.sleep(5)

        # Click on the LambdaTest HomePage Link
        # This test will fail as the titles will not match
        title = "Cross Browser Testing Tools | Free Automated Website Testing | LambdaTest_1"
 ..................
 ..................

@pytest.mark.usefixtures("driver_init")
@pytest.mark.incremental
class Test_Scenario_2:
 ..................
 ..................

In test case test_2(), Google search for LambdaTest is performed. Once the search results are ready, a click on the first search result is done and the page title is compared with the expected title. For testing, we have deliberately introduced an error in the code and the Selenium test automation case fails as the titles do not match.

class Test_Scenario_1:
 ..................
 ..................
 def test_2(self):
        self.driver.get('https://www.google.com/')
        .................
   ..................
        search_text = "LambdaTest"
        search_box = self.driver.find_element_by_xpath("//input[@name='q']")
        search_box.send_keys(search_text)

        time.sleep(5)
        search_box.submit()

        time.sleep(5)

        # Click on the LambdaTest HomePage Link
        # This test will fail as the titles will not match
title = "Cross Browser Testing Tools | Free Automated Website Testing | LambdaTest_1"
 ..................
 ..................

 # As test_2 fails, test_3 will xfail i.e. skipped and all subsequent tests in the same class
 # will be marked as xfailed
      def test_3(self):
   ..................
   ..................
@pytest.mark.usefixtures("driver_init")
@pytest.mark.incremental
class Test_Scenario_2:
 def test_4(self):
   ..................
   ..................

Test case test_3() which follows after test_2() will be skipped from execution and marked as xfail as the previous test [i.e. test_2()] in the same class has failed. Selenium test automation case test_4() which is a part of class Test_Scenario_2() is also marked for incremental testing.

Here is the command for execution:

py.test -rx --verbose --capture=no

Shown below is the execution snapshot:

Try browser automation testing on the most powerful cloud infrastructure. Leverage LambdaTest automation testing for faster, reliable and scalable experience on cloud.

Here is the final status of the tests performed in the example:

Test_2 executes and fails whereas test_3 execution is skipped due to failure of test_2. Hence, it is marked as xfail. Though the test_4 () passes here, even if it were to fail, the test would be marked as fail (and not xfail) as there is no dependency on the test from the other class.

In case, incremental testing is removed from the above implementation, all the tests would be executed. Only test_2() will fail whereas the remaining three tests would pass as the @pytest.mark.incremental decorator is removed from the implementation.

Pytest Tutorial: Executing Multiple Test Cases From Single File

This certification is for professionals looking to develop advanced, hands-on expertise in Selenium automation testing with Python and take their career to the next level.

Here’s a short glimpse of the Selenium Python 101 certification from LambdaTest:

Experience hassle-free execution of your Selenium Testing scripts by running them on the LambdaTest cloud grid. With support for over 3000 desktop and mobile environments, try LambdaTest for free and take advantage of its comprehensive testing capabilities.

Using maxfail Command Line Option To Stop Test Suite After N Test Failures

PyTest offers a command-line option called maxfail which is used to stop test suite after n test failures. For example, to stop the test suite after n test failures, where the value of n is 1. Once a failure is encountered, all the subsequent tests are skipped.

Though it is not a good practice to develop inter-dependent Selenium test automation cases, in some scenarios, you may have to develop such tests. The maxfail option comes handy in such cases as unnecessary execution of dependent tests can be avoided. Shown below is the syntax of maxfail:

py.test --maxfail=<num>

How To Skip Test In Python With pytest Using maxfail?

To demonstrate the usage of maxfail, we take the automated browser testing example which was shown in the section How To Skip Test In Python With pytest Using @pytest.mark.incremental. The implementation specific to skip the test in Python with pytest will be removed, remaining code remains the same.

conftest.py

import pytest
from selenium import webdriver
import urllib3
import warnings

def pytest_runtest_setup(item):
    previousfailed = getattr(item.parent, "_previousfailed", None)
    if previousfailed is not None:
        pytest.xfail("previous test failed (%s)" % previousfailed.name)

@pytest.fixture(scope="class")
def driver_init(request):
    web_driver = webdriver.Chrome()
    request.cls.driver = web_driver
    yield
    web_driver.close()
    # web_driver.quit()

test_step.py

import pytest
from selenium import webdriver
import urllib3
import warnings

def pytest_runtest_setup(item):
    previousfailed = getattr(item.parent, "_previousfailed", None)
    if previousfailed is not None:
        pytest.xfail("previous test failed (%s)" % previousfailed.name)

@pytest.fixture(scope="class")
def driver_init(request):
    web_driver = webdriver.Chrome()
    request.cls.driver = web_driver
    yield
    web_driver.close()
    # web_driver.quit()

The example shown below is executed where we stop the test suite after n test failures and the maximum number of failures set to 2. Here is the execution command:

py.test -rx --verbose --capture=no --maxfail=2

As incremental testing is not enabled, all the four Selenium test automation cases will be executed and the execution stops once the number of failures becomes 2. Test case test_2() fails as the window title does not match the expected one, rest all the three test cases are executed. Out of 4 test cases, 1 test case fails and 3 test cases pass. Here is the execution snapshot:

All In All

In this Python tutorial, we looked at how to stop test suite after n test failures. Skip tests in Python with pytest can be followed in Selenium test automation scenarios where tests are inter-dependent and execution of dependent tests needs to be skipped in case the primary test case fails.

The @pytest.mark.incremental decorator is used to skip tests in Python with PyTest. The other mechanism to stop test suite after n test failures by using maxfail command-line option.

This brings an end to this Python tutorial. I hope this article was informative, and you now know skip tests in python or how to stop test suite after n test failures. In case you still have any doubts, do reach out to us in the comment section below, and we’d be happy to help you with your problem. Happy Testing!!!?