Support usage of other fixtures in xunit style methods: setup() and setup_class()

[rsyring]

I've been following #517 for some time and I've recently looked at #4091 in the hope that it would help me figure out how I can get better support for xunit style class fixtures. Here is what I want to be able to do:

@pytest.fixture(scope="session")
def db():
    return database.Database()

class TestIt(object):
    def setup_class(cls, db):
        # This method fails with: TypeError: setup_class() takes exactly 2 arguments (1 given)
        cls.foo = db.build_expensive_foo()

    def setup(self, db):
        # fails with same error as setup_class
        cls.bar = db.build_expensive_bar()

    @pytest.fixture(autouse=True, scope='class')
    def prep_class(self, db):
       # db is available here, but self is an instance, not the class itself, so this
       # doesn't work as intended.  I could use self.__class__, but feel like I
       # shouldn't need to.
        print('prepare', self, db)
        self.foo = 'bar'

    def test_foo_bar(self):
        assert self.foo
        assert self.bar

I seem to have two, maybe distinct, problems:

1. Despite the work done in Use fixtures to invoke xunit-style fixtures #4091, setup_class and setup do not seem to have access to fixtures (db in this example). I'm not sure if that is intended, an oversight, but the above "feels" like it should work to me.

2. Using an autouse fixture at the class level gives access to the db fixture but the fact that an instance instead of the class is being passed in makes it non-intuitive on how to store state at the class level.

Suggestion

I'd really like to see setup_class() and setup() get support for using fixtures defined elsewhere. We have a lot of tests that use this pattern to cache test data at the class level and it would be helpful to give those methods access to other fixtures.

I understand from the discussions in the issues and PR that pytest is trying to keep a distinction between xunit style and newer fixture style tests, but I'm not sure why that needs to be the case. Unless there is a technical limitation that makes this hard to support, the UX described above feels well balanced IMO and would be ideal for our code bases that are heavily using setup_class and setup extensively.

Being able to use fixtures as proposed also allows a more gradual transition from one style to another.

2. Maybe consider passing the class object instead of the instance object to class level methods using a fixture

Version Info

Using Arch Linux. Versions:

(pytest-xunit)rsyring@nanite:~/tmp$ pytest --version
This is pytest version 4.5.0, imported from /home/rsyring/tmp/virtualenvs/pytest-xunit/lib/python3.7/site-packages/pytest.py

(pytest-xunit)rsyring@nanite:~/tmp$ pip freeze
atomicwrites==1.3.0
attrs==19.1.0
more-itertools==7.0.0
pluggy==0.11.0
py==1.8.0
pytest==4.5.0
six==1.12.0
wcwidth==0.1.7

(pytest-xunit)rsyring@nanite:~/tmp$ python --version
Python 3.7.3

Edit: tested with clean virtualenv and Python 3

[gryznar]

I'll bump the topic. It would be great if fixtures were supported in setup_class, setup_method, teardown_class and teardown_method.

[RonnyPfannschmidt]

in "xunit" style one can jsut declare those a fixture and it works, fixtures will not be supported as args in normal xunit style methods

[rsyring]

in "xunit" style one can jsut declare those a fixture and it works,

As I stated in the initial description, this becomes problematic for setup_class and autouse:

Using an autouse fixture at the class level gives access to the db fixture but the fact that an instance instead of the class is being passed in makes it non-intuitive on how to store state at the class level.

To be specific, if one uses def setup_class(cls, ...) in that method cls is the class itself. When making it a fixture and using autouse, cls is instead an instance of cls, not cls itself. It can be worked around, as stated in my original post, but I think the DX on that is problematic.

fixtures will not be supported as args in normal xunit style methods

Why? Is this just a time issue, i.e. not a priority for the maintainers? Or, is there some technical reason why this would create a lot of additional complexity? FWIW, I personally find the need to add @fixture everywhere a bit busy. I prefer xunit style and try to avoid @fixture if possible. I'd even do the work on this if someone could point me in the right direction. I just don't want to waste my time if it's not going to get merged due to philosophical differences.

[bluetech]

@rsyring Does

    @pytest.fixture(autouse=True, scope='class')
    @classmethod
    def prep_class(cls, db):

not work?

[nicoddemus]

Or, is there some technical reason why this would create a lot of additional complexity?

Mostly this: we run the xunit tests using their own TestCase execution (and TestResult), which don't support parameters/fixtures. I suspect we would need to duplicate some work, increasing the plugin's complexity, with the risk of introducing bugs.

[gryznar]

Moreover using xunit style is much simplier and much more explicit than defining it via fixtures. So support for fixtures in it is very reasonable.

[gryznar]

Or, is there some technical reason why this would create a lot of additional complexity?

Mostly this: we run the xunit tests using their own TestCase execution (and TestResult), which don't support parameters/fixtures. I suspect we would need to duplicate some work, increasing the plugin's complexity, with the risk of introducing bugs.

How about rewritting it and use fixture with autouse and proper scope instead? So for example setup_class will be a fixture with scope 'class' and autouse set to True beneath, forced to run in proper order.

[rsyring]

@bluetech I can't check now, but I will in a bit. It could be that it didn't work b/c I wasn't including @classmethod. I think I cobbled the example together from actual tests, so not sure if that is a mistake in the example or an actual mistake in my testing of this issue.

Mostly this: we run the xunit tests using their own TestCase execution (and TestResult), which don't support parameters/fixtures. I suspect we would need to duplicate some work, increasing the plugin's complexity, with the risk of introducing bugs.

I can appreciate this. If it's just a matter of extracting the shared functionality and using it in both places, and the tests of this functionality are good, is it really that big of a concern? I don't want to downplay the complexity of the work if it is indeed complex, but I don't want to give up on this if it's just a timing/capacity issue as I'm willing to put in the time to do the work and test thoroughly. Are we really talking about unreasonable risk to pytest's stability at this point? If so, so be it. If not, then I'm motivated to get this resolved.

I realize as a maintainer you are not just approving the PR but "owning" the future of that functionality as well. I've been in the Python ecosystem for the last 15 years and I plan to be around in the future just as long. I'm willing to "own" this functionality for the foreseeable future if issues arise that need to be resolved because of it. I don't do a lot of work in visibly public repos, but I did help get Flask-SQLAlchemy unstuck a couple years go. My code is solid and I'm reasonable to work with. :)

Moreover using xunit style is much simpler and much more explicit than defining it via fixtures.

I agree with this from a DX perspective. Having to make those methods fixtures, just to get access to other fixtures, when I'm never going to use them as fixtures themselves feels like a...cludge.

[nicoddemus]

Are we really talking about unreasonable risk to pytest's stability at this point? If so, so be it. If not, then I'm motivated to get this resolved.

I'm sure given the time and effort, this could be made to work, and depending on how it is implemented, it might be low risk for the rest of pytest's functionality.

I realize as a maintainer you are not just approving the PR but "owning" the future of that functionality as well. I've been in the Python ecosystem for the last 15 years and I plan to be around in the future just as long. I'm willing to "own" this functionality for the foreseeable future if issues arise that need to be resolved because of it.

I appreciate that. This makes me much more willing to accept this kind of change.

Given this perspective, I'm personally OK with accepting that change, but not sure if other maintainers feel the same.

Also note that even if we are not willing to add this to the core, I'm pretty sure it is possible to implement this as an external plugin, if you would like to experiment with that first and see how it fares in the wild.

[RonnyPfannschmidt]

we have 2 compatibility concerns to consider

1. some those methods have optional parameters and multiple potential names
2. structurally all we would do is consider them "fixtures"

based on that im inclined to prefer just marking/declaring them as fixture instead of making the hack even more magical

however if the hack could actually be simplified as a outcome of this, then im happy to +1 it

personally i treat xunit style setup methods as a legacy anti-patter that prevents both good naming of resources one needs and good integration

[gryznar]

@RonnyPfannschmidt if we could treat

setup_class like:

@classmethod
@pytest.fixture(scope='class', autouse=True)
def _setup_class(cls, *args):
    ....

teardown_class like:

@classmethod
@pytest.fixture(scope='class', autouse=True)
def _teardown_class(cls, *args):
    yield
    ....

setup_method like:

@pytest.fixture(scope='function', autouse=True)
def _setup_method(self, method=None, *args):
    ....

and teardown_method like:

@pytest.fixture(scope='function', autouse=True)
def _teardown_method(self, method=None, *args):
    yield
    ...

(and so on for module and function scope), where are args are possible fixtures and force them to run in order in which current xunit style setup currently runs, so all the compatibility issues could be solved and you could get rid off custom implementation. Am I right?

Finally xunit style fixtures could be just a reference to normal fixtures, but written much simpler.

[RonnyPfannschmidt]

@gryznar in the current implementation we only add those fixtures when needed for xunit setup (if we had them for everything, it would be too expensive and slow down non-xunit tests, alternative approaches to remove the cost have not yet been investigated )

additionally we have a indirection in there to deal with teardown and with the fact that xunit setup/teardown methods have optional parameters that are NOT like the request for fixtures

[gryznar]

@RonnyPfannschmidt I see, but you can agree with me for sure, that they are much more readable and obvious for end user, so maybe we can really consider upgrading / refactioring / rewriting them?

[RonnyPfannschmidt]

@gryznar i consider the setup/teardown methods to be "deceptively easy" i strongly prefer a well placed and named fixture

i observed often that setup methods end up as bag of what ought to be more than one self contained fixture, in turn making teardown a minefield

[gryznar]

You may be right but for less advanced user, properly used xunit-style setup seems to be the best solution. Please consider implementing fixtues in them if it is possible.

[gryznar]

As you can see, that's not only my personal request.

[RonnyPfannschmidt]

As stated earlier, I personally don't believe in the proposed approach

Im happy to provide inputs on a implementation, but I will not try to create one myself as of now

[nicoddemus]

@gryznar just to understand, when you say "xunit style setup", do you mean, pytest-style:

class Test:

    def setup_method(self):
        ...

Or do you mean the support for unittest?

class Test(unittest.TestCase):

    def setUp(self):
        ...

[gryznar]

Of course I mean the pytest implementation. Inheriting over TestCase does not allow to use fixtures at all. My use case: I am writing tests in selenium. Driver instance is created and passed to test via fixture. But a lot of tests have common beginning. Page instantiation and stuff like that. So the easiest way to solve that is to use setup_method and perform this code in that. But this is not possible due to dependency to fixture which cannot be passed to setup_method. Of course I can use autoused fixtures but that is not as readable as teardown_method, so that's why I hope, that fixtures will be supported. sob., 19 mar 2022, 13:21 użytkownik Bruno Oliveira ***@***.***> napisał:

…

@gryznar <https://github.com/gryznar> just to understand, when you say "xunit style setup", do you mean, pytest-style: class Test: def setup_method(self): ... Or do you mean the support for unittest? class Test(unittest.TestCase): def setUp(self): ... — Reply to this email directly, view it on GitHub <#5289 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AVR4OCELOJXHYDSNXRQYBDDVAXBEPANCNFSM4HN5SICA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[nicoddemus]

Of course I mean the pytest implementation.

Sorry if it was not so obvious to me. When I replied to @rsyring I thought we were discussing unittest style.

So the requirement is to write:

class Test:

    def setup_method(self, tmp_path):
        ...

Instead of:

class Test:

    @pytest.fixture(autouse=True)
    def my_setup(self, tmp_path):
        ...

This, if implemented, would remove the need for that 1 line (the decorator). Hmm.

Note that is possible to write an alias to reduce the verbosity:

setup_with_fixtures = pytest.fixture(autouse=True)

class Test:

    @setup_with_fixtures 
    def my_setup(self, tmp_path):
        ...

Also note that fixtures support yield for simpler tear down. Using xunit-style one has to write this:

class Test:

    def setup_method(self):
        self.conn = Connection()
        self.session = Session(self.conn)

    def teardown_method(self):
        self.session.close()
        self.conn.shutdown()

With fixtures you can keep everything in the same functions, which usually results in simpler code (you can use locals if you don't need access in tests):

class Test:

    def setup_method(self):
        conn = Connection()
        self.session = Session(conn)
        yield
        self.session.close()
        conn.shutdown()

and integrates well with context-managers (which is common in resources created in setup/teardown):

class Test:

    @setup_with_fixtures
    def my_setup(self):
        with Connection() as conn:
            with Session(conn) as self.session:
                yield

So by making setup_* functions support fixtures, we would make them "implicit" fixtures, having now two ways to declare a fixture (and document, test, and support for years to come).

Because of all the points above, I don't think it is worth to complicate both the API and the internals given that you can get to use fixtures in setup functions (and more features as pointed above) by adding a 1 line decorator.

Having said all that, I'm sure this is possible to implement as a plugin. I just don't think this has a place in the core.

[rsyring]

Sorry if it was not so obvious to me. When I replied to @rsyring I thought we were discussing unittest style.

Sorry for the confusion. I meant xunit style setup methods not classes actually inherited from unitttest.

Regarding the differences in opinion on what style (methods vs functions, yield vs two methods), I just want to point out that these are style questions. I personally like less lines of code and less decorators when I can do without them. But, I understand why others like fixtures better and I have devs on my team who use them a lot. Maybe even too much, but that's a rabbit trail we won't go down.

Having said all that, I'm sure this is possible to implement as a plugin. I just don't think this has a place in the core.

I appreciate the time that's been given to this discussion (FWIW, I also appreciate the quick fix on #9767) and I don't want to take more time away from more productive activities. If the adjustments to the core would be a net negative instead of net positive, i.e. references to cleaning up the split implementation, then I can head the plugin route.

Anyone want to give any pointers on implementation? I'm sure I can go figure it out, but if you have any suggestions or examples that might shave some time off, I'd appreciate it.

Thanks again.

[nicoddemus]

Sorry for the confusion. I meant xunit style setup methods not classes actually inherited from unitttest.

I see, sorry me for the confusion. Everything I said about the collector/results objects from unittest don't apply to xunit-style in the pytest implementation, only for unittest.TestCase subclasses.

Anyone want to give any pointers on implementation? I'm sure I can go figure it out, but if you have any suggestions or examples that might shave some time off, I'd appreciate it.

I can't look into the details right now, but I would try to hook into the collection, and when a class is imported, to look for setup_* methods and apply the @fixture decorator to them, and from that point on let pytest handle the method as a fixture. You might need to rename the method too, to avoid pytest picking it up later and handle it as a setup_* method itself.

[rsyring]

FWIW, I did some experimentation this morning:

https://github.com/level12/pytest-xunit-fixtures/tree/cf0cffd184548bddf0c7ab99cf9ad9d61e2a1cea (not everything here is expected to work currently, in fact most won't work).

and here are the issues I've ran into so far:

1. Fixture doesn't like to be used on a classmethod: 'classmethod' object has no attribute '__name__'
2. setup_method doesn't work b/c of the method argument: fixture 'method' not found
3. Wrapping setup w/ fixture stops it from being called automatically. Have to use autouse. Not a bug really, just another inconsistency for someone used to how xunit style methods use.
4. The setup method has to explicitly call for the app_ctx fixture, unlike the test methods, b/c it's not a test method. Another inconsistency with this workflow.
5. I anticipate I'm going to have problems with setup_class and scopes because app_ctx is function scoped but I want setup_class to be able to use it. But, if I use fixtures, that fixture will have to be set to class scope so as to only run once for each class.
6. Teardown methods are going to need an extra yield in them immediately, which is arguably unsightly and again not a 1-1 with what a xunit style developer would expect to have to do.

It's possible I don't know what I'm doing here, so please feel free to point out anything I'm doing wrong. At the same time, I've been developing python a long time and I've been using pytest for many years and I'm still having problems. And this is the second time I've really tried solving this problem, the first time was prior to creating this issue.

I still don't mind trying to go the plugin route, but I hope the commenters on this thread can at least admit we can agree that a robust xunit style workflow with access to fixtures isn't trivially solved with fixture wrapping.

Edit: that phrasing might have seemed more snide than I intended.

[kisscelia]

I just want to add args(e.g. global config parameter) to setup_class, and use it as self.config in the test_xxx method, but not add this config parameter as args to all test_xx method. :(

In conftest.py

@pytest.fixture(scope="session", autouse=True)
def config():
    ...
    return config

In testxxx.py, like this: # I have thousands of test cases....

class TestXxx(object):
    @classmethod
    def setup_class(cls, config):  # <- args, add once, used everywhere in this class
        cls.config = config

    def test_x1(self):
        print(self.config) 

    def test_x2(self):
        print(self.config)

not like this:

class TestXxx(object):

    def test_x1(self, config):  # <- args
        print(config)

    def test_x2(self, config):  # <- args
        print(config)

[nicoddemus]

@rsyring would using a class decorator be acceptable?

@setup_with_fixtures
class Test:

    def setup_method(self, tmp_path):
        ...

I think this would be simple to implement.

[rsyring]

@rsyring would using a class decorator be acceptable?

@nicoddemus thanks for asking. For our use case which involves thousands of test classes across multiple projects it would be better than nothing but not ideal. However, I'd be glad to see an implementation or prototype of that as I could likely learn from how it's implemented and then, when I have the time, extend the pattern to work automatically and/or using a plugin.

[RonnyPfannschmidt]

As far as I understood, the use case is already implementable by just adding fixtures to those classes, I'm strictly opposed just magically adding calling those methods as the composition with inheritance is absolutely unclear

If someone demonstrate a good way im happy to change my stance

[rsyring]

@RonnyPfannschmidt As I wrote above, unless I've missed something, it's demonstrably false that "just add a fixture" is a reasonable solution for the use cases this issue is concerned with.

I'm strictly opposed just magically adding calling those methods as the composition with inheritance is absolutely unclear

These methods are magically called already. The requested addition is to make fixtures available to the methods when called. I don't understand how the same functionality for these methods creates additional concerns around composition/inheritance (but that's very possibly just my ignorance of pytest internals showing through). Regardless, whatever the challenges are, wouldn't they be resolved with the exact same logic normal test methods (which enable fixtures) use? That's what I'd expect as a user of this implementation.

Can you be more specific or give an example that demonstrates your concern so I can make sure to consider that in the implementation?

In my other comment I've already acknowledged that there is concern about putting this in pytest and offered to go the plugin route. I assume if that gets off the ground and I can solve the problems you are concerned about in a reasonable way, that you can put that in pytest core. If not, it can stay a plugin. I don't know that I care either way other than it seems like worse dev UX for these methods to behave in a way that is arguably unexpected and that the workarounds that seem intuitive, like just using fixtures, don't actually solve the problem.

I just haven't had the time to dig into pytest implementation/plugins enough to figure out how to build the plugin yet. A class decorator implementation, even if it's not in this project/repo would be helpful to me as I'm sure it would save me some learning/digging time.

[Zac-HD]

Closing this, as per #12465 (comment) it looks like we can at least start by implementing an external plugin.