Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add functional tests for pytest adapter. #4739

Closed
3 tasks
ericsnowcurrently opened this issue Mar 13, 2019 · 0 comments
Closed
3 tasks

Add functional tests for pytest adapter. #4739

ericsnowcurrently opened this issue Mar 13, 2019 · 0 comments
Assignees

Comments

@ericsnowcurrently
Copy link
Member

ericsnowcurrently commented Mar 13, 2019

We recently added a Python script for running pytest discovery. It has plenty of unit tests, but no functional tests. Those are tests that use actual external dependencies (e.g. pytest, filesystem) and don't stub things out. (Technically, system tests are a kind of functional tests, but the distinction isn't important in this case).

Since we want to ensure that we've integrated properly with pytest, we should add functional tests for the adapter script. This will involve the following tasks:

  • add support to pythonFiles/tests/__main__.py for opting in to (or out of) functional tests (e.g. --functional or --no-functional, respectively)
  • add a functional test that runs against a simple test tree
  • add a functional test that runs against a complex test tree

These functional tests will go in pythonFiles/tests/testing_tools/adapter/test_functional.py (new file).

Simple Test Tree

pythonFiles/
    tests/testing_tools/adapter/
        .data/  # new
            tests/  # test root
                test_spam.py
                    #def test_simple():
                    #    assert True

Complex Test Tree

Note: the following is probably not complete; there are other things worth adding.

pythonFiles/
    tests/testing_tools/adapter/
        .data/  # new
            tests/  # test root
                test_doctest.txt
                    * ...
                test_pytest.py
                    * simple module-level test function
                    * module-level doctests
                    * marked test functions
                        + skip, skipif, xfail, filterwarnings, custom  # see https://docs.pytest.org/en/latest/reference.html#marks
                        + one function for each
                        + at least one function with multiple marks
                    * parameterized test functions
                        + no parameters, single case
                        + one parameter, single case
                        + one parameter, multiple cases
                        + multiple parameters, multiple cases
                        + with *args & **kwargs parameters, multiple cases
                        + 4 parameters, 3 decorators
                        + decorator with "marks" arg (skip/xfail/etc)
                    * test function that calls pytest.skip()
                    * test function that calls pytest.fail()
                    * test function that raises Exception
                    * for loop that generates test functions dynamically
                    * suite (a test class)
                        + simple test method
                        + suite-level doctests
                        + marked ("skip") test method
                        + parameterized test method
                        + nested suite
                            + nested suite
                                 + simple test method
                                 + nested suite (no methods)
                        + nested suite (no methods)
                    * another suite
                        + a simple test method
                    * an empty suite
                    * suite using a fixture
                test_unittest.py
                    * test suite inheriting from unittest.TestCase
                        + simple test function
                        + skipped test method (decorator)
                        + skip-if'ed test method (decorator)
                        + test method with expected failure decorator
                        + test method that raises unittest.Skip
                        + test method that calls unittest.skip()
                        + test method that raises Exception
                        + test method that uses one "with self.subtest():"
                        + test method that uses multiple "with self.subtest():" (nested)
                        + for loop that generates test methods dynamically
                        + nested class (no base class)
                        + nested class inheriting from unittest.TestCase
                    * another suite
                        + a simple test method
                    * an empty suite
                test_mixed.py
                     * module-level test function (a la pytest)
                     * marked ("skip") test function
                     * class with no base class (a la pytest)
                         + a simple test method
                     * class inheriting from unittest.TestCase
                         + simple test method
                spam.py  # note: no "test_" prefix
                    * module-level test function
                test_foo.py
                    * module-level test function
                test_42.py
                    * module-level test function
                test_42-43.py  # note the hyphen
                    * module-level test function
                v/
                    __init__.py
                    spam.py
                        * pytest-style suite
                            * simple test function ("test_simple")
                            * simple test function ("test_simple")
                    test_eggs.py
                        * "from .spam import *"
                    test_ham.py
                        * "from .spam import test_simple"
                        * "from .spam import test_simple as test_not_hard"
                    test_spam.py
                        * "from .spam import test_simple"
                        * simple test function ("test_simpler")
                w/
                    # no __init__.py
                    test_spam.py
                        * simple test function
                    test_spam_ex.py
                        * simple test function
                x/y/z/   # each with a __init__.py
                    test_ham.py
                        * a simple test function
                    a/
                        test_spam.py
                            * a simple test function
                    b/
                        test_spam.py
                            * a simple test function

Also check the following prior art for more examples to include:

@ericsnowcurrently ericsnowcurrently added area-testing needs PR debt Covers everything internal: CI, testing, refactoring of the codebase, etc. labels Mar 13, 2019
@DonJayamanne DonJayamanne removed this from the 2019, week 11 - March Sprint 6 milestone Mar 13, 2019
ericsnowcurrently added a commit that referenced this issue Mar 22, 2019
(for #4739)

This will facilitate optionally skipping functional tests (under pythonFiles) for the sake of speed. Functional tests (or suites) will be marked with the "functional" marker:

```python
@pytest.mark.functional
def test_spam():
    ...

@pytest.mark.functional
class TestSpam:
    def test_x(self):
        ...
```
@DonJayamanne DonJayamanne removed the debt Covers everything internal: CI, testing, refactoring of the codebase, etc. label Mar 25, 2019
@ericsnowcurrently ericsnowcurrently self-assigned this Mar 25, 2019
@ghost ghost removed the needs PR label Apr 15, 2019
@lock lock bot locked as resolved and limited conversation to collaborators May 13, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

2 participants