Track exceptions when setting up and finalizing fixtures

[ShurikMen]

There is a task to track exceptions when setting up and finalizing fixtures.
When setting up, an exception can be obtained in the pytest_fixture_setup hook:

@pytest.hookimpl(hookwrapper=True)
def pytest_fixture_setup(fixturedef):
    result = yield
    if result.exception:
        print('SETUP EXCEPTION in {0}: {1}'.format(fixturedef.argname, result.exception))

But at finalizing, you can't just get a list of exceptions of all finalizers that have been executed. Because there is no analog of the result as in the pytest_fixture_setup hook.
I propose to finalize the pytest_fixture_post_finalizer specification and an additional argument that will contain the exclusion(s) of all fixture finalizers.

src/_pytest/hookspec.py:

def pytest_fixture_post_finalizer(
    fixturedef: "FixtureDef[Any]", request: "SubRequest", exception: "BaseException | None"
) -> None:

src/_pytest/fixtures.py::FixtureDef::finish

        node = request.node
        if len(exceptions) == 1:
            final_exception = exceptions[0]
        elif len(exceptions) > 1:
            msg = f'errors while tearing down fixture "{self.argname}" of {node}'
            final_exception = BaseExceptionGroup(msg, exceptions[::-1])
        else:
            final_exception = None
        node.ihook.pytest_fixture_post_finalizer(fixturedef=self, request=request, exception=final_exception)
        # Even if finalization fails, we invalidate the cached fixture
        # value and remove all finalizers because they may be bound methods
        # which will keep instances alive.
        self.cached_result = None
        self._finalizers.clear()
        if final_exception:
            raise final_exception

[ShurikMen]

Probably a more correct solution would be to transfer the completion of the fixtures to an independent hook pytest_fixture_teardown.

[bluetech]

I agree that the problem described in the issue is worth solving.

To me the original idea of passing the exception(s) to pytest_fixture_post_finalizer seems better than adding a new pytest_fixture_teardown hook:

• Adding a hook adds complexity and so should be done judiciously when there is an alternative

• This hook is in a hot path so adds some performance overhead

• The pytest_fixture_teardown implementation accesses a bunch of private attributes of FixtureDef, such that a plugin would never be able to sanely outright replace the implementation (at least in a stable, sanctioned way). So the hook would only be useful as a hookwrapper or as a lifecycle hook. But pytest_fixture_post_finalizer already fulfills that need.

• Finally, moving the code over to pytest_fixture_teardown breaks the implementation symmetry between execute and finish which feels wrong to me.

Therefore my preference would be to add the exception/ExceptionGroup to pytest_fixture_post_finalizer, unless there is another use case for pytest_fixture_teardown.

[ShurikMen]

I agree that the problem described in the issue is worth solving.

To me the original idea of passing the exception(s) to pytest_fixture_post_finalizer seems better than adding a new pytest_fixture_teardown hook:

• Adding a hook adds complexity and so should be done judiciously when there is an alternative
• This hook is in a hot path so adds some performance overhead
• The pytest_fixture_teardown implementation accesses a bunch of private attributes of FixtureDef, such that a plugin would never be able to sanely outright replace the implementation (at least in a stable, sanctioned way). So the hook would only be useful as a hookwrapper or as a lifecycle hook. But pytest_fixture_post_finalizer already fulfills that need.
• Finally, moving the code over to pytest_fixture_teardown breaks the implementation symmetry between execute and finish which feels wrong to me.

Therefore my preference would be to add the exception/ExceptionGroup to pytest_fixture_post_finalizer, unless there is another use case for pytest_fixture_teardown.

pytest_fixture_post_finalizer does not track full lifecycle of teardown fixtures. It only notifies of the fact that all finalizers in teardown functions have been completed.
For this reason, there is currently no symmetry between execute and finish, because the execution of the fixture setup code itself is not performed in the execute itself, the setup function code itself is executed in the pytest_fixture_setup hook. execute calls this hook if it has not found the result in the cache.

Therefore, I moved the execution of the finalizers function code to the pytest_fixture_teardown hook, similar to pytest_fixture_setup. It seemed like a better solution to me.

Now I use this hook to track the status of the test object before executing the teardown function and to verify that nothing happened to this object during code execution. Because in case of some errors, further execution of other fixtures and tests is impossible.

    @pytest.hookimpl(hookwrapper=True)
    def pytest_fixture_setup(self, fixturedef: FixtureDef, request: pytest.FixtureRequest) -> Generator[None]:
        # if fixture is not param
        if hasattr(request, 'module') and fixturedef.func.__name__ != 'get_direct_param_fixture_func':
            # check full stack state, before run setup function
            self._check_stack(request.module)
            # add markers to stdout output
            self._mark_state_in_dev_stdout(fixturedef.argname, 'start fixture setup', request.module)
        result = yield
        # if we have ProtocolException after fixture setup, then check and restore stand state
        if result.exception and isinstance(result.exception, ProtocolException):
            self._check_stand_state(request.module)

    @pytest.hookimpl(hookwrapper=True)
    def pytest_fixture_teardown(self, fixturedef: FixtureDef, request: pytest.FixtureRequest) -> Generator[None]:
        # if fixture is not param and have finilizers
        if fixturedef._finalizers and fixturedef.func.__name__ != 'get_direct_param_fixture_func' and hasattr(request, 'module'):
            # check full stack state, before run teardown functions
            self._check_stack(request.module)
            # add markers to stdout output
            self._mark_state_in_dev_stdout(fixturedef.argname, 'start fixture teardown', request.module)
        result = yield
        # if we have exeption after fixture teardown, then check and restore stand state
        if result.exception:
            try:
                raise result.exception
            except *ProtocolException:
                self._check_stand_state(request.module)
            except *BaseException:
                pass

[rplevka]

pytest_fixture_post_finalizer does not track full lifecycle of teardown fixtures. It only notifies of the fact that all finalizers in teardown functions have been completed.
For this reason, there is currently no symmetry between execute and finish, because the execution of the fixture setup code itself is not performed in the execute itself, the setup function code itself is executed in the pytest_fixture_setup hook. execute calls this hook if it has not found the result in the cache.

I agree with this. Our use case is quite simple - we'd like to act upon start of the tear-down part of the fixture (set appropriate logging context for the external logging solution). post-finalizer hook runs code after the tear down is finished (too late for us)

[rplevka]

@bluetech have you had any chance to revisit this, please?

[RonnyPfannschmidt]

i still agree with @bluetech bluetech - i need a better understanding of your use-case to select a appropriate hook

[ogajduse]

@rplevka explained our team's use case well in #12163 (comment). Let me know if you'd like more details about it.

@ShurikMen, Mind resuming on this issue?

[ShurikMen]

simple example

from collections.abc import Generator
import pytest


@pytest.fixture(scope='class')
def firts_fixture() -> Generator[None]:
    yield
    print('Start teardown for firts_fixture')
    raise ValueError('Error in first fixture')


@pytest.fixture
def second_fixture() -> Generator[None]:
    yield
    print('Start teardown for second_fixture')
    raise ValueError('Error in second fixture')


@pytest.mark.usefixtures(
    'second_fixture',
    'firts_fixture',
)
class TestOne:
    def test_some(self) -> None:
        pass

In my task, it is necessary to execute a certain code before executing the code after the yield and after executing this code.
If an exception occurs when executing the code after yield, it must be processed with certain checks.
To do this, I made a hook similar to pytest_fixture_setup, which wraps around the entire teardown, and not just after the code is completed in teardown.

My current solution:

plugins/plugin.py

from __future__ import annotations

import pytest

from _pytest.fixtures import FixtureDef, FixtureValue, SubRequest


def pytest_addhooks(pluginmanager: pytest.PytestPluginManager) -> None:
    from . import hookspec
    pluginmanager.add_hookspecs(hookspec)


def pytest_fixture_teardown(fixturedef: FixtureDef[FixtureValue], request: SubRequest) -> None:
    exceptions: list[BaseException] = []
    while fixturedef._finalizers:
        fin = fixturedef._finalizers.pop()
        try:
            fin()
        except BaseException as e:
            exceptions.append(e)
    node = request.node
    node.ihook.pytest_fixture_post_finalizer(fixturedef=fixturedef, request=request)
    # Even if finalization fails, we invalidate the cached fixture
    # value and remove all finalizers because they may be bound methods
    # which will keep instances alive.
    fixturedef.cached_result = None
    fixturedef._finalizers.clear()
    if len(exceptions) == 1:
        raise exceptions[0]
    elif len(exceptions) > 1:
        msg = f'errors while tearing down fixture "{fixturedef.argname}" of {node}'
        raise BaseExceptionGroup(msg, exceptions[::-1])


def finish(self: FixtureDef, request: SubRequest) -> None:
    node = request.node
    node.ihook.pytest_fixture_teardown(fixturedef=self, request=request)


FixtureDef.finish = finish

plugins/hookspec.py

from typing import Any

from _pytest.fixtures import FixtureDef, SubRequest


def pytest_fixture_teardown(fixturedef: 'FixtureDef[Any]', request: 'SubRequest') -> None:
    """Perform fixture teardown execution.

    :param fixturdef:
        The fixture definition object.
    :param request:
        The fixture request object.

    Use in conftest plugins
    =======================

    Any conftest file can implement this hook. For a given fixture, only
    conftest files in the fixture scope's directory and its parent directories
    are consulted.
    """

conftest.py

from collections.abc import Generator
import pytest


pytest_plugins = [
    'plugins.plugin'
]


@pytest.hookimpl(hookwrapper=True)
def pytest_fixture_teardown(fixturedef: pytest.FixtureDef) -> Generator[None]:
    print(f'Some code before teardown {fixturedef.func.__name__}')
    result = yield
    print(f'Exception in teardown {fixturedef.func.__name__}:', result.exception)

The result for the example above

10:59 $ pytest --tb no -sv --no-header
============================================================================= test session starts =============================================================================
collected 1 item                                                                                                                                                              

tests/test_asd.py::TestOne::test_some PASSEDSome code before teardown second_fixture
Start teardown for second_fixture
Exception in teardown second_fixture: Error in second fixture
Some code before teardown firts_fixture
Start teardown for firts_fixture
Exception in teardown firts_fixture: Error in first fixture

tests/test_asd.py::TestOne::test_some ERROR

=========================================================================== short test summary info ===========================================================================
ERROR tests/test_asd.py::TestOne::test_some - ExceptionGroup: errors during test teardown (2 sub-exceptions)
========================================================================= 1 passed, 1 error in 0.01s ==========================================================================

[RonnyPfannschmidt]

The given outline proposes a specific implementation without explaining the use case/ intent so -1 until the intent is clear

[ShurikMen]

The case is simple: execute the code before the teardown of each fixture and after the teardown of each fixture, checking the result of the teardown itself.