Stateful async POC

[touilleMan]

This is a simple proof of concept implementing async support to stateful testing.
The main idea is to allow GenericStateMachine sub-classes to define a _custom_runner method that StateMachineRunner will call when a run need to be done.

This allow different framework to implement they own custom runner (just like what has been done with #1343). For instance trio's _custom_runner create a nursery and allow step rules to access it through a special get_root_nursery method.

This is currently super rough stuff, so feedback greatly wanted ;-)

[touilleMan]

btw @njsmith you may be interested by this ;-)

[DRMacIver]

This looks neat, @touilleMan! I haven't had a chance to go through it in detail, but I'm in principle keen. Could you maybe flesh out some documentation/use cases on it? I'm almost completely unfamiliar with Trio and the Python async world, which makes it hard for me to evaluate beyond the nitpicky details of the code.

Also, would you like to join the maintainers team? It comes with very little in terms of powers (you get to approve code reviews if you want) or responsibilities (honestly you can keep doing exactly what you're doing), but you get a badge next to your name saying "maintainer" for it. :-)

[Zac-HD]

Welcome - if you would like to join - to the team 😄

---

I am really, really keen to get something like this working, but I think it's an open question where it should live - and I'm inclined to say that the appropriate place is in pytest-trio, with Hypothesis adding whatever general hooks would make that practical (for Trio, and asyncio, Twisted, curio, etc.).

This would probably mean looking at what parts Hypothesis could stabilise to allow easy implementation of a custom run_state_machine_as_test - @DRMacIver and I have talked about moving this from find(...) to an @given(data()) function so exposing the tools to make and hook in a custom version downstream seems reasonable.

Async stateful tests pose some harder problems in general though, because the order of the steps is no longer the only factor in the ordering - we might need a way for Hypothesis to take charge of the scheduler. In Trio's case this is actually OK as it's mostly-fair and we already seize the global PRNG so the random portion is reproducible. However, adversarial scheduling of coroutines and fault injection (timeouts, cancellations, mock network issues, etc.) would be very powerful for testing - as well as demanding library- and domain-specific implementation and user-facing controls.

TLDR; I don't think Hypothesis should provide generic async stateful support, instead we should make it as easy as possible for pytest-trio and similar tools to provide stateful testing customised for their users.

Also keen to keep talking this over, but maybe we should move to an issue?

[touilleMan]

Also, would you like to join the maintainers team?

Sure ! It would be my pleasure ;-)

Could you maybe flesh out some documentation/use cases on it?

I will add some documentation. For the usecase you can have a look at this test which implement a simple consumer/producer pattern.

but I think it's an open question where it should live - and I'm inclined to say that the appropriate place is in pytest-trio, with Hypothesis adding whatever general hooks would make that practical (for Trio, and asyncio, Twisted, curio, etc.).

Yup totally agree, all trio-related code has obviously nothing to do with hypothesis (btw @njsmith where do you think the hypothesis-trio compatibility stuff should be put in ? I think the simpler way would be to put it into pytest-trio, but on the other hand this is not pytest-related stuff strictly speaking...)

Async stateful tests pose some harder problems in general though, because the order of the steps is no longer the only factor in the ordering - we might need a way for Hypothesis to take charge of the scheduler

This seems like the right thing to do, but I guess it's something totally dependent of the async framework so I can't see a way to do this in a generic way from hypothesis. I would be more willing of providing a big empty hole for the framework to run and let it responsible for everything.

TLDR; I don't think Hypothesis should provide generic async stateful support, instead we should make it as easy as possible for pytest-trio and similar tools to provide stateful testing customised for their users.

Agree

[Zac-HD]

OK - on the Hypothesis end, I've added a comment to #1300 outlining our aim to support other stateful-like tests downstream.

This isn't really the right place to discuss the Trio end, but Hypothesis also recommends that extensions have names like hypothesis-trio... and an experimental-quality package could be just the thing 😉

I'll close this pull request because it's unlikely to be merged, but I'm excited to keep working on both ends of 'fuzz Trio programs with Hypothesis' problem and look forward to seeing where we go next.