Pytest own tests and hypothesis requirement

[FRidh]

On Nix we automatically run the test suite of a package when building the package. The pytest 3.0 testsuite requires hypothesis, and vice versa. Such a circular dependency is problematic. An easy way to "solve" the issue at our side is to simply not run the pytest testsuite, but I think it's obvious we rather not do that :)

Is it possible to test a substantial set of pytest without hypothesis, and if so, how should this be done?

[FRidh]

cc @RonnyPfannschmidt

[RonnyPfannschmidt]

@FRidh all important behaviors are tested by non-hypothesis tests

currently hypothesis is only used to do exploitative expanding of some parametric unit-tests

for pedantic testing you might want to introduce an untested version of hypothesis as a non-propagating build input or vice versa (makes me wonder if checkInputs for nix would make sense)

[FRidh]

@RonnyPfannschmidt in Nix the problem is that hypothesis depends indirectly on pytest and propagating an untested version is not really possible (anymore).

It would be nice if there's a method to test only the non-hypothesis tests.

[RonnyPfannschmidt]

@FRidh i see, i believed the testsuite was still executable without hypothesis installed,

if that is no longer the case then i will try to investigate a solution over the holidays

[nicoddemus]

if that is no longer the case then i will try to investigate a solution over the holidays

How about if we mark the hypothesis tests? This way they can be excluded using the -m option.

[RonnyPfannschmidt]

@nicoddemus hypothesis has import time test decorators, a simple mark is not going to help

[nicoddemus]

Oh you are right.

Well I don't really see a solution then. We have the same issue with other libraries now, like six (which also uses pytest to test itself).

[blueyed]

importorskip could be extended to return something that keeps on raising "Skipped" via __getattr__ maybe?
Then it could also be used there.

Related issue: #4977

[Zac-HD]

FWIW Hypothesis doesn't actually depend on pytest - it's perfectly usable with the stdlib Unittest or Django test runners, etc. As of HypothesisWorks/hypothesis#2204 we don't even import pytest to provide the optional integration.

[Zac-HD]

Closing this because there's no practical resolution, and pytest and hypothesis only have test-time dependencies on each other: you can build and install them independently. A "build packages then run tests" approach would solve the Nix problem, and is pretty much standard for Python.