pytest support

[verhovsky]

py_test is a Bazel rule and pytest is a Python library for writing and running tests

https://docs.pytest.org
https://github.com/pytest-dev/pytest

I've found two third-party Bazel rules for running tests written with pytest bazel_rules_pex and rules_pyz, but both projects are unmaintaned and don't work anymore. Edit 2020-01-15: see https://github.com/ali5h/rules_pip/blob/b4e94d0ddc95367217b7059bcdb926c159d28602/defs.bzl#L139 as well.

pytest has been mentioned a few times on the bazel mailing list https://groups.google.com/forum/?nomobile=true#!searchin/bazel-discuss/pytest%7Csort:date

I'm not sure what the correct way to add support for it would be. Probably another rule but I don't know what it should be called, because the names are so similar. pytest pytest_py_test py_test_pytest all seem pretty funny. Or maybe it could be an option for py_test or a global one in WORKSPACE.

pytest can run tests written with unittest (and has nicer output when a test fails), so theoretically you could just change the py_test rule to run tests with pytest but that's obviously not happening.

[groodt]

It is already possible to use pytest with py_test.

py_test(
    name = "unit_test",
    srcs = ["test.py"],
    main = "test.py",
    python_version = "PY3",
    srcs_version = "PY3",
    deps = TEST_DEPS,
)

test.py

import sys

import pytest


def test_aws_login_without_aws_profile():
    assert True


if __name__ == "__main__":
    sys.exit(pytest.main([__file__]))

[verhovsky]

Thanks for the quick reply. I saw someone mention this trick on the mailing list https://groups.google.com/forum/?nomobile=true#!searchin/bazel-discuss/pytest|sort:date/bazel-discuss/d1RKUmyve_Q/hm_Dc3XYCAAJ

If you forget the sys.exit like the person in that mailing list post, Bazel will never know if the tests in that file fail. It would also just be nice if you didn't need 3-4 cryptic lines of boiler plate for every test file.

[pstradomski]

@brandjon: it's something I have in my fork of rules_python, using a genrule to create a snippet similar to the one in @groodt comment above.

Do you think it makes sense to add it to experimental/ ?

[ali5h]

i added a sample macro for pytest here https://github.com/ali5h/rules_pip/blob/master/defs.bzl#L116

[pstradomski]

i added a sample macro for pytest here https://github.com/ali5h/rules_pip/blob/master/defs.bzl#L116

PTAL at pull request #243. It takes a slightly different approach. In particular, it does not pull dependency on pytest on its own, to give users more freedom (e.g. whether to use requirements.txt, virtualenv or vendor pytest in repository).

[ali5h]

i added that as a sample. there is no need to load the pytest explicitly, updated the sample to show this

[dayfine]

Suppose defining your own BUILD rule is the right solution: semantically it would belong to /contrib (though there isn't one), instead of /experimental, which is more for things that might become official in the future.

This issue specifically is a pytest 'problem', in the sense that the framework that does 'a lot' (and maybe too much in a sense). From the viewpoint of the build system, py_test rule would not make any assumption about the existence of pytest or another custom testing framework.

Personally I think having a 10-line main file as @ali5h did is most straightforward and I probably won't even create any custom BUILD rule:

py_test(
  name = 'my_target',
  srcs = [
    ...
    "//path/to:pytest_main",
  ] ,
  main = "//path/to:pytest_main",
  ...
)

[doubler]

If the test.py has relative import code like from .module import xx, then bazel test would fail.
ModuleNotFoundError: No module named '__main__.module'; '__main__' is not a package

[thundergolfer]

We don't use relative imports anywhere in our Bazel workspace. They're not worth the trouble imo, and I wouldn't be surprised at all if implementation details in Python rules made them not work correctly.

[doubler]

@thundergolfer why do python rules has the imports. https://docs.bazel.build/versions/master/be/python.html

[thundergolfer]

@doubler We used to use that imports attr but I think it's a code-smell. For our use cases we've been able to move to absolute imports and remove all use of imports = [] in our BUILD files.

[jxramos]

@dayfine makes a solid point, py_test strictly operates off return codes and the absence of segfaults or something like that taking place during test execution. That the phonetics are identical to pytest makes the concepts crowd into each other somewhat unavoidably but they are very distinct.

There's also the more generalized approach from what @groodt shares above, that includes the passing of arguments from bazel py_test targets into pytest via the args attribute. I've used that with great success rendering additional debug detail on failures and even to factor out deep dependency graph test cases lumped into a parametrized pytest case such that each parameter value becomes its own py_test target in the BUILD file via a list comp. Where pytest would normally iterate over all test cases (even those that aren't implicated in the the current diff to the repo) bazel is more nimble to only run those test parameters that are implicated by a particular change. That saves a great deal of test execution runtime by leaning on the remote cache for test results.

BUILD

py_test(
    name = "test_foo",
    srcs = [
        "test_foo.py",
    ],
    args = [
        "--junit-xml=test_foo_results.xml",
    ],
    deps = [
        "pytest",
    ],
)

test_foo.py

import pytest
import sys
...

if __name__ == "__main__":
    # if using 'bazel test ...'
    args = sys.argv[1:]
    ret_code = pytest.main([__file__, "--verbose"] + args)

    # Print relevant debug logs or other artifacts for CI job to have richer context
    if ret_code != 0:
        print_debug_stuff(args)

    # Exit with return code
    sys.exit(ret_code)