Only load the necesssary backends when testing with `skip_backend` or `only_backend` markers

[ghisvail]

Summary

I am currently packaging the scikit-hep stack for Guix, and I am facing troubles enabling the tests for the basic set of dependencies.

It looks like despite calling pytest with the only_numpy marker, it tries to load the other backends:

tests/conftest.py:63: in <module>
    (pyhf.tensor.pytorch_backend(), None),
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^
/gnu/store/jabv9192g9pi22w6cn6zwf57ajk8jm80-python-pyhf-0.7.6/lib/python3.11/site-packages/pyhf/tensor/__init__.py:54: in __getattr__
    raise exceptions.ImportBackendError(
E   pyhf.exceptions.ImportBackendError: ('There was a problem importing PyTorch. The pytorch backend cannot be used.', ModuleNotFoundError("No module named 'torch'"))

I did a quick review of the code in conftest.py, and indeed the backend fixture calls pyhf.set_backend(...) for the whole parameter set, irrespective of whether they are excluded via skip_$backend or only_$backend.

It would be nice if such behaviour could be changed to only call pyhf.set_backend("numpy") if -m only_numpy is specified.

Cheers,

Additional Information

Code of Conduct

• I agree to follow the Code of Conduct

[kratsg]

Hi @ghisvail so this wasn't really the intended design to not load the backends. If you look at the pyproject.toml - we require all backends installed. There could be some rewrites given the lazy-loading feature in future pythons that make it easier to handle, but this likely won't happen on the shorter timescale unless you find a way to rewrite the tests to allow it to not load in all backends.

[matthewfeickert]

@ghisvail yeah, this realistically isn't something that we're going to prioritize anytime soon, so for repacking efforts (can you link us to what you have so far for the Scikit-HEP stack out of curiosity?) our recommendation would be to do the most minimal checks possible. I think that even if you had all the backends that some tests would fail given changes in the dependencies that we need to address with the upcoming v0.8.0 release (which is severely late and my fault apologies).

All that being said, if you were very interested in contributing the ability to address this we would be happy to review a plan with you in this issue to help streamline a contributed PR.

[kratsg]

Can be done with https://docs.pytest.org/en/stable/reference/reference.html#pytest-importorskip -- which should make it largely easier.