pytest function scoped fixtures should run once per example, not once per test

[DRMacIver]

Because of the way Hypothesis just exposes a single test function, pytest just runs its fixtures once for that function.

This is a long running problem with Hypothesis's interaction with pytest and there is an open bug about it on the pytest side too.

Fixing this is tricky. The fixtures-integration branch has a partial prototype which makes some progress towards making it work, but it's very far from complete and has a number of issues which are likely to need substantial work to fix.

I am currently seeking funding to work on this ticket and it is unlikely to make much progress without said funding.

[Zac-HD]

Related to #59

[Zac-HD]

Note that the pytest-subtesthack package by @untitaker provides a decent workaround for this case.

[Zac-HD]

Actually... it might be possible to incorporate a version of pytest-subtesthack into our pytest plugin, that activates this logic only for Hypothesis tests. @untitaker, do you think this could work?

[untitaker]

Maybe but the version would look very differently, assuming you want this to be the new default behavior of "given"

[DRMacIver]

Actually... it might be possible to incorporate a version of pytest-subtesthack into our pytest plugin, that activates this logic only for Hypothesis tests

I think we need to be quite careful about this. There are a bunch of problems with pytest-subtesthack (or were last I looked) and it's a fairly major change in behaviour. If we're going to support pytest function scoped fixtures I think we need to do it properly.

[Zac-HD]

Fair enough! I didn't expect it to be that easy, but thought it was worth mentioning.

[untitaker]

I think @ossronny can judge better than me how bad the hack really is. I'm not really sure about adding this to hypothesis but I am not sure if it will be fixed in pytest somehow either