Tutorial

Once integrated into your project, the Pytest::Pytest target and the pytest_discover_tests() function are available for using.

Using the target

Let’s consider a project that wraps C++ logic with Python bindings. We need to add a CMakeLists.txt configuration file to include Python tests within the same directory.

The Pytest command can easily be implemented using the add_test function:

add_test(
    NAME PythonTest
    COMMAND Pytest::Pytest ${CMAKE_CURRENT_SOURCE_DIR}
)

For the tests to run, the PYTHONPATH environment variable must be updated to locate the built package library. We can use a generator expression to resolve the path of the dependent target directory dynamically:

set_tests_properties(
    TEST PythonTest
    PROPERTY ENVIRONMENT
        PYTHONPATH=$<TARGET_FILE_DIR:MyLibrary>:$ENV{PYTHONPATH}
)

The shared library might also be required during runtime execution, so its location should be provided:

set_tests_properties(
    TEST PythonTest
    APPEND PROPERTY ENVIRONMENT
        LD_LIBRARY_PATH=$<TARGET_FILE_DIR:MyLibrary>:$ENV{LD_LIBRARY_PATH}
)

Warning

The environment variable used to locate shared libraries depends on the platform. LD_LIBRARY_PATH is used on Linux, DYLD_LIBRARY_PATH on macOS, and PATH on Windows.

After building the project, the tests can then be executed using CTest. If all tests are successful, the output will look as follows:

    Start 1: PythonTest
1/1 Test #1: PythonTest .......................   Passed    0.55 sec

However, if only one test is unsuccessful, the entire test suite will be marked as failed.

    Start 1: PythonTest
1/1 Test #1: PythonTest .......................***Failed    0.47 sec

Using the function

A pytest_discover_tests() function is provided to create CMake tests for each Python test collected. Therefore, the configuration added in the previous section could be replaced by the following:

pytest_discover_tests(
    PythonTest
    LIBRARY_PATH_PREPEND
        $<TARGET_FILE_DIR:MyLibrary>
    PYTHON_PATH_PREPEND
        $<TARGET_FILE_DIR:MyLibrary>
    DEPENDS MyLibrary
)

This will create a new PythonTest target, dependent on the MyLibrary target.

The expected environment can be defined simply with the LIBRARY_PATH_PREPEND and PYTHON_PATH_PREPEND options, which both accept multiple values. The environment variable used to locate shared libraries will be automatically chosen according to the platform.

A list of dependent targets can be defined with the DEPENDS option, which accepts multiple values.

After building the project, running CTest will display the tests as follows:

    Start 1: PythonTest.test_greet_world
1/4 Test #1: PythonTest.test_greet_world ...........   Passed    0.47 sec
    Start 2: PythonTest.test_greet_john
2/4 Test #2: PythonTest.test_greet_john ............   Passed    0.47 sec
    Start 3: PythonTest.test_greet_julia
3/4 Test #3: PythonTest.test_greet_julia ...........   Passed    0.47 sec
    Start 4: PythonTest.test_greet_michael
4/4 Test #4: PythonTest.test_greet_michael .........   Passed    0.54 sec

A fully identified test collected by Pytest might look like this:

tests/test_module.py::TestMyClass::test_example

By default, only the class and function name of each Pytest test collected are used to create the CMake tests. You can use the INCLUDE_FILE_PATH option to include the file path within the name:

 pytest_discover_tests(
     PythonTest
     LIBRARY_PATH_PREPEND
         $<TARGET_FILE_DIR:MyLibrary>
     PYTHON_PATH_PREPEND
         $<TARGET_FILE_DIR:MyLibrary>
     INCLUDE_FILE_PATH
     DEPENDS MyLibrary
 )

Pytest usually requires the test class and function to start with a specific prefix, which can be trimmed using the TRIM_FROM_NAME or TRIM_FROM_FULL_NAME options. The value can use a regular expression to match the part of the test name that should be trimmed.

The TRIM_FROM_FULL_NAME option can be used to trim parts of the entire name, while the TRIM_FROM_NAME option will be applied to the class, method and function name of each Pytest test collected for convenience.

 pytest_discover_tests(
     PythonTest
     LIBRARY_PATH_PREPEND
         $<TARGET_FILE_DIR:MyLibrary>
     PYTHON_PATH_PREPEND
         $<TARGET_FILE_DIR:MyLibrary>
     TRIM_FROM_NAME "^(Test|test_)"
     INCLUDE_FILE_PATH
     DEPENDS MyLibrary
 )

After rebuilding the project, running CTest will display the tests as follows:

    Start 1: PythonTest.greet_world
1/4 Test #1: PythonTest.greet_world ...............   Passed    0.47 sec
    Start 2: PythonTest.greet_john
2/4 Test #2: PythonTest.greet_john ................   Passed    0.47 sec
    Start 3: PythonTest.greet_julia
3/4 Test #3: PythonTest.greet_julia ...............   Passed    0.47 sec
    Start 4: PythonTest.subfolder.greet_michael
4/4 Test #4: PythonTest.subfolder.greet_michael ...   Passed    0.54 sec

You can also define custom environment variables and test properties using the ENVIRONMENT and PROPERTIES options, respectively.

It is also possible to regroup all tests under one CTest test, as was the case when using the target. This can be useful during development to ensure that the tests run faster, especially if you use fixtures with a broader scope.

This can be done by setting the BUNDLE_TESTS option to True:

 pytest_discover_tests(
     PythonTest
     LIBRARY_PATH_PREPEND
         $<TARGET_FILE_DIR:MyLibrary>
     PYTHON_PATH_PREPEND
         $<TARGET_FILE_DIR:MyLibrary>
     DEPENDS MyLibrary
     BUNDLE_TESTS True
 )

After rebuilding the project once again, running CTest will display the tests as follows:

    Start 1: PythonTest
1/1 Test #1: PythonTest .......................   Passed    0.51 sec

Note

The BUNDLE_PYTHON_TESTS environment variable can also set this option dynamically.