Scaffolding tests failing on main branch with thread leaks and RuntimeError

[ccs96307]

Hi team,

Before starting my work on issue #3332 (Support more sampling parameters with openai worker), I wanted to ensure the existing tests on the main branch were passing in my local environment.

I ran the scaffolding unit tests with the following command:

cd tests/unittests/

pytest scaffolding/

However, the tests failed with multiple thread leak errors and a RuntimeError. Here is the summary from the test output:

...

scaffolding/test_worker.py::test_trtllm_worker_generation
  /code/tensorrt_llm/.venv-3.12/lib/python3.12/site-packages/_pytest/runner.py:246: PluggyTeardownRaisedWarning: A plugin raised an exception during an old-style hookwrapper teardown.
  Plugin: threadleak, Hook: pytest_runtest_call
  Failed: Test leaked [<Thread(Thread-12 (_manager_spawn), started 129198052079296)>]
  For more information see https://pluggy.readthedocs.io/en/stable/api_reference.html#pluggy.PluggyTeardownRaisedWarning
    lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
============================================================================= slowest durations =============================================================================
10.02s call     scaffolding/test_bench.py::test_scaffolding_benchmark
10.01s call     scaffolding/test_parallel_process.py::test_parallel_process_helper_with_two_level
8.01s call     scaffolding/test_parallel_process.py::test_parallel_process_helper
6.51s setup    scaffolding/test_worker.py::test_trtoai_worker_generation[pytorch-enable_processpool]
5.75s call     scaffolding/test_scaffolding.py::test_unbatched_scaffolding_sync
5.52s call     scaffolding/test_scaffolding.py::test_batched_scaffolding_sync
5.48s call     scaffolding/test_scaffolding.py::test_majority_vote
5.48s call     scaffolding/test_scaffolding.py::test_async_scaffolding_generation
5.43s call     scaffolding/test_worker.py::test_trtllm_worker_generation

(20 durations < 0.005s hidden.  Use -vv to show these durations.)
========================================================================== short test summary info ==========================================================================
FAILED scaffolding/test_bench.py::test_scaffolding_benchmark - Failed: Test leaked [<Thread(Thread-3 (main_loop_thread), started 129206254659264)>]
FAILED scaffolding/test_scaffolding.py::test_unbatched_scaffolding_sync - Failed: Test leaked [<Thread(Thread-6 (_manager_spawn), started 129204058322624)>]
FAILED scaffolding/test_scaffolding.py::test_batched_scaffolding_sync - Failed: Test leaked [<Thread(Thread-7 (_manager_spawn), started 129202548373184)>]
FAILED scaffolding/test_scaffolding.py::test_async_scaffolding_generation - Failed: Test leaked [<Thread(Thread-8 (_manager_spawn), started 129201004869312)>]
FAILED scaffolding/test_scaffolding.py::test_majority_vote - Failed: Test leaked [<Thread(Thread-9 (_manager_spawn), started 129199595583168)>]
FAILED scaffolding/test_worker.py::test_trtllm_worker_generation - Failed: Test leaked [<Thread(Thread-12 (_manager_spawn), started 129198052079296)>]
ERROR scaffolding/test_worker.py::test_trtoai_worker_generation[pytorch-enable_processpool] - RuntimeError: Server exited unexpectedly.
======================================================== 6 failed, 3 passed, 6 warnings, 1 error in 67.46s (0:01:07) ========================================================

It seems like some threads are not cleaning up their threads properly?

Could you please help confirm if this is a known issue on the main branch? I want to make sure my development environment is set up correctly before proceeding.

Any help or guidance would be greatly appreciated~ Thanks!

[VALLIS-NERIA]

Hi, the leaking thread itself is not a issue. Some error caused the worker to crash so it cannot clean up the threads.
@WeiHaocheng can you please help?

[ccs96307]

I found the problem is my environment without enough VRAM (I just have 24GB)... totally my bad. I tried to used a smaller model (DeepSeek-R1-Distill-Qwen-7B => DeepSeek-R1-Distill-Qwen-1.5B) to avoid that.

but I still encountered the similar issue:

==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.12.3, pytest-8.4.0, pluggy-1.5.0                                                                                                                                   
rootdir: /code/tensorrt_llm/tests/unittest                                                                                                                                                    
configfile: pytest.ini                                                                                                                                                                        
plugins: asyncio-1.0.0, forked-1.6.0, threadleak-0.5.0, csv-3.0.0, mock-3.14.1, timeout-2.4.0, split-0.10.0, env-1.1.5, cov-6.1.1, anyio-4.9.0, xdoctest-1.0.2, shard-0.1.2, rerunfailures-15.
0, hypothesis-6.130.8, flakefinder-1.1.0, xdist-3.6.1, typeguard-4.3.0                                                                                                                        
asyncio: mode=Mode.STRICT, asyncio_default_fixture_loop_scope=module, asyncio_default_test_loop_scope=function                                                                                
collected 10 items                                                                                                                                                                            
Running 10 items in this shard                                                                                                                                                                
                                                                                                                                                                                              
scaffolding/test_bench.py                                                                                                                                                             [ 10%]  
scaffolding/test_parallel_process.py .Parallel process time: 2.004763126373291                                                                                                                
Sequence process time: 8.011444091796875                                                                                                                                                      
.                                                                                                                                                [ 30%]                                       
scaffolding/test_scaffolding.py ....                                                                                                                                                   [ 70%] 
scaffolding/test_task_collection.py .                                                                                                                                                  [ 80%] 
scaffolding/test_worker.py ..                                                                                                                                                          [100%] 

========================================================================================== FAILURES ======================================================================16:50:47 [1078/1940$
_________________________________________________________________________________ test_scaffolding_benchmark _________________________________________________________________________________
                                                                                                                                                                                              
item = <Function test_scaffolding_benchmark>                                                                                                                                                  
                                                                                                                                                                                              
    @pytest.hookimpl(hookwrapper=True)                                                                                                                                                        
    def pytest_runtest_call(item):                                                                                                                                                            
        start_threads = None                                                                                                                                                                  
        exclude_regex = item.config.getini("threadleak_exclude")                                                                                                                              
        if is_enabled(item):                                                                                                                                                                  
            start_threads = current_threads(exclude_regex)                                                                                                                                    
        yield                                                                                                                                                                                 
        if start_threads:                                                                                                                                                                     
            end_threads = current_threads(exclude_regex)                                                                                                                                      
            leaked_threads = end_threads - start_threads                                                                                                                                      
            if leaked_threads:                                                                                                                                                                
>               pytest.fail("Test leaked %s" % sorted_by_name(leaked_threads))                                                                                                                
E               Failed: Test leaked [<Thread(Thread-3 (main_loop_thread), started 124794629088960)>]                                                                                          
                                                                                                                                                                                              
../../.venv-3.12/lib/python3.12/site-packages/pytest_threadleak.py:48: Failed                                                                                                                 
---------------------------------------------------------------------------------- Captured stdout teardown ----------------------------------------------------------------------------------
F                                                                                                                                                                                             
====================================================================================== warnings summary ======================================================================================
scaffolding/test_bench.py::test_scaffolding_benchmark                                                                                                                                         
  /code/tensorrt_llm/.venv-3.12/lib/python3.12/site-packages/_pytest/runner.py:246: PluggyTeardownRaisedWarning: A plugin raised an exception during an old-style hookwrapper teardown.       
  Plugin: threadleak, Hook: pytest_runtest_call                                                                                                                                               
  Failed: Test leaked [<Thread(Thread-3 (main_loop_thread), started 124794629088960)>]                                                                                                        
  For more information see https://pluggy.readthedocs.io/en/stable/api_reference.html#pluggy.PluggyTeardownRaisedWarning                                                                      
    lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise                                                                                                                       
                                                                                                                                                                                              
scaffolding/test_worker.py::test_trtoai_worker_generation[pytorch-enable_processpool]                                                                                                         
  scaffolding/test_worker.py:66: PytestWarning: The test <Function test_trtoai_worker_generation[pytorch-enable_processpool]> is marked with '@pytest.mark.asyncio' but it is not an async fun
ction. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'.                                                                 
    @pytest.mark.asyncio(loop_scope="module")                                                                                                                                                 
                                                                                                                                                                                              
-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html                                                                                                                       
===================================================================================== slowest durations ======================================================================================
16.31s call     scaffolding/test_scaffolding.py::test_batched_scaffolding_sync                                                                                                                
15.04s setup    scaffolding/test_worker.py::test_trtoai_worker_generation[pytorch-enable_processpool]                                                                                         
14.82s call     scaffolding/test_scaffolding.py::test_unbatched_scaffolding_sync                                                                                                              
11.49s call     scaffolding/test_scaffolding.py::test_majority_vote                                                                                                                           
11.05s call     scaffolding/test_scaffolding.py::test_async_scaffolding_generation                                                                                                            
10.54s call     scaffolding/test_worker.py::test_trtllm_worker_generation                                                                                                                     10.02s call     scaffolding/test_parallel_process.py::test_parallel_process_helper_with_two_level                                                                                             
8.01s call     scaffolding/test_parallel_process.py::test_parallel_process_helper                                                                                                             
5.02s call     scaffolding/test_bench.py::test_scaffolding_benchmark                                                                                                                          
2.02s call     scaffolding/test_worker.py::test_trtoai_worker_generation[pytorch-enable_processpool]                                                                                          
                                                                                                                                                                                              
(20 durations < 0.005s hidden.  Use -vv to show these durations.)                                                                                                                             
================================================================================== short test summary info ===================================================================================
FAILED scaffolding/test_bench.py::test_scaffolding_benchmark - Failed: Test leaked [<Thread(Thread-3 (main_loop_thread), started 124794629088960)>]                                           
==================================================================== 1 failed, 9 passed, 2 warnings in 109.82s (0:01:49) =====================================================================

This time there is no any other information, so I think maybe it's the ScaffoldingLlm does not expectly shutdown to cause it.

I tried to add a new line at test_bench.py:

results, requests_execution_time, total_time = asyncio.run(
    async_scaffolding_benchmark(scaffolding_llm, task_collection_types,
                                requests, concurrency))

# I add an explicit shutdown
scaffolding_llm.shutdown()

And I can pass the unittest!

==================================================================================== test session starts =====================================================================================
platform linux -- Python 3.12.3, pytest-8.4.0, pluggy-1.5.0
rootdir: /code/tensorrt_llm/tests/unittest
configfile: pytest.ini
plugins: asyncio-1.0.0, forked-1.6.0, threadleak-0.5.0, csv-3.0.0, mock-3.14.1, timeout-2.4.0, split-0.10.0, env-1.1.5, cov-6.1.1, anyio-4.9.0, xdoctest-1.0.2, shard-0.1.2, rerunfailures-15.0, hypothesis-6.130.8, flakefinder-1.1.0, xdist-3.6.1, typeguard-4.3.0
asyncio: mode=Mode.STRICT, asyncio_default_fixture_loop_scope=module, asyncio_default_test_loop_scope=function
collected 10 items                                                                                                                                                                           
Running 10 items in this shard

scaffolding/test_bench.py .                                                                                                                                                            [ 10%]
scaffolding/test_parallel_process.py ..                                                                                                                                                [ 30%]
scaffolding/test_scaffolding.py ....                                                                                                                                                   [ 70%]
scaffolding/test_task_collection.py .                                                                                                                                                  [ 80%]
scaffolding/test_worker.py ..                                                                                                                                                          [100%]

====================================================================================== warnings summary ======================================================================================
scaffolding/test_worker.py::test_trtoai_worker_generation[pytorch-enable_processpool]
  scaffolding/test_worker.py:66: PytestWarning: The test <Function test_trtoai_worker_generation[pytorch-enable_processpool]> is marked with '@pytest.mark.asyncio' but it is not an async function. Please remove the asyncio mark. If the test is not marked explicitly, check for global marks applied via 'pytestmark'.
    @pytest.mark.asyncio(loop_scope="module")

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
===================================================================================== slowest durations ======================================================================================
20.23s call     scaffolding/test_scaffolding.py::test_batched_scaffolding_sync
14.53s setup    scaffolding/test_worker.py::test_trtoai_worker_generation[pytorch-enable_processpool]
12.87s call     scaffolding/test_scaffolding.py::test_unbatched_scaffolding_sync
11.62s call     scaffolding/test_scaffolding.py::test_async_scaffolding_generation
11.52s call     scaffolding/test_scaffolding.py::test_majority_vote
10.78s call     scaffolding/test_worker.py::test_trtllm_worker_generation
10.02s call     scaffolding/test_bench.py::test_scaffolding_benchmark
10.02s call     scaffolding/test_parallel_process.py::test_parallel_process_helper_with_two_level
8.01s call     scaffolding/test_parallel_process.py::test_parallel_process_helper
2.11s call     scaffolding/test_worker.py::test_trtoai_worker_generation[pytorch-enable_processpool]

(20 durations < 0.005s hidden.  Use -vv to show these durations.)
========================================================================= 10 passed, 1 warning in 116.90s (0:01:56) ==========================================================================

I'd be happy to create a Pull Request to apply this change to the test file.

Please let me know if you'd prefer to handle this internally or if you'd like me to proceed.

Thanks!

[WeiHaocheng]

@ccs96307 Great~ Thank you.