vllm v1 engine 启动模型无法卸载

[bao21987]

System Info / 系統信息

xinference docker 1.6.1
a100 cuda12.4

Running Xinference with Docker? / 是否使用 Docker 运行 Xinfernece？

• docker / docker
• pip install / 通过 pip install 安装
• installation from source / 从源码安装

Version info / 版本信息

xinference docker 1.6.1

The command used to start Xinference / 用以启动 xinference 的命令

docker run --name xinference -d -p 9997:9997 -p 9998:9998 -e XINFERENCE_HOME=/data -e VLLM_USE_V1=1
-v ****:/data
-v ****/saves:*****/saves
--gpus all xprobe/xinference:v1.6.1 sh -c "xinference-local -H 0.0.0.0

Reproduction / 复现过程

使用vllm部署qwen3-4B

Initializing a V1 LLM engine (v0.8.5) with config: model='xxxxx', speculative_config=None, tokenizer='xxxx', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='auto', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=None, served_model_name=xxxx, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"level":3,"custom_ops":["none"],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":512}

卸载模型失败：
2025-06-09 21:24:31,320 xinference.core.worker 147 INFO [request ce205be4-45b2-11f0-aa3e-0242ac110002] Enter terminate_model, args: <xinference.core.worker.WorkerActor object at 0x7fee2227b6a0>, kwargs: model_uid=qwen3_v1.5.19-0
2025-06-09 21:24:31,324 xinference.model.llm.vllm.core 158156 INFO Stopping vLLM engine
[2025-06-09 21:24:31] INFO pool.py:430: Sub pool can't be killed: psutil.Process(pid=158156, name='Model: qwen3_', status='running', started='20:40:07')
2025-06-09 21:24:31,429 xinference.core.worker 147 INFO [request ce205be4-45b2-11f0-aa3e-0242ac110002] Leave terminate_model, elapsed time: 0 s

Expected behavior / 期待表现

vllm v1 engine 模型可以正常卸载

[qinxuye]

在镜像里更新 xoscar 试下，pip install xoscar==0.7.8 ，然后重启服务。

[bao21987]

@qinxuye 试了更新xoscar版本，还是不行，再贴一下完整的日志
启动模型日志：

2025-06-10 14:05:36,099 xinference.core.worker 147 INFO [request ed043072-45c0-11f0-a0d2-0242ac110002] Enter launch_builtin_model, args: <xinference.core.worker.WorkerActor object at 0x7ff1fb5e60c0>, kwargs: model_uid=qwen3_v1.5.19-0,model_name=qwen3_v1.5.19,model_size_in_billions=4,model_format=pytorch,quantization=none,model_engine=vLLM,model_type=LLM,n_gpu=auto,request_limits=None,peft_model_config=None,gpu_idx=None,download_hub=None,model_path=None,xavier_config=None,enable_prefix_caching=True,guided_decoding_backend=auto
[2025-06-10 14:05:36] INFO pool.py:293: Creating sub pool via command: ['/usr/bin/python3.10', '-m', 'xoscar.backends.indigen', 'start_sub_pool', '-sn', 'psm_741bbc6e']
2025-06-10 14:05:37,042 xinference.model.llm.llm_family 147 INFO Caching from URI: /path/to/saves/Qwen3-4B/full/qwen3_v0.19
2025-06-10 14:05:37,075 xinference.model.llm.llm_family 147 INFO Cache /path/to/saves/Qwen3-4B/full/qwen3_v0.19 exists
INFO 06-10 14:05:42 [init.py:239] Automatically detected platform cuda.
2025-06-10 14:05:48,151 xinference.core.model 1111 INFO Start requests handler.
/opt/inference/xinference/model/llm/core.py:143: UserWarning: enable_thinking cannot be disabled for non hybrid model, will be ignored
warnings.warn(
2025-06-10 14:05:48,155 xinference.model.llm.vllm.core 1111 INFO Loading qwen3_v1.5.19 with following model config: {'enable_prefix_caching': True, 'guided_decoding_backend': 'auto', 'tokenizer_mode': 'auto', 'trust_remote_code': True, 'tensor_parallel_size': 1, 'pipeline_parallel_size': 1, 'block_size': 16, 'swap_space': 4, 'gpu_memory_utilization': 0.9, 'max_num_seqs': 256, 'quantization': None, 'max_model_len': None, 'scheduling_policy': 'fcfs'}Enable lora: False. Lora count: 0.
2025-06-10 14:05:48,178 transformers.configuration_utils 1111 INFO loading configuration file /path/to/saves/Qwen3-4B/full/qwen3_v0.19/config.json
loading configuration file /path/to/saves/Qwen3-4B/full/qwen3_v0.19/config.json
2025-06-10 14:05:48,179 transformers.configuration_utils 1111 INFO loading configuration file /path/to/saves/Qwen3-4B/full/qwen3_v0.19/config.json
loading configuration file /path/to/saves/Qwen3-4B/full/qwen3_v0.19/config.json
2025-06-10 14:05:48,181 transformers.configuration_utils 1111 INFO Model config Qwen3Config {
"architectures": [
"Qwen3ForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 151643,
"eos_token_id": 151645,
"head_dim": 128,
"hidden_act": "silu",
"hidden_size": 2560,
"initializer_range": 0.02,
"intermediate_size": 9728,
"max_position_embeddings": 40960,
"max_window_layers": 36,
"model_type": "qwen3",
"num_attention_heads": 32,
"num_hidden_layers": 36,
"num_key_value_heads": 8,
"rms_norm_eps": 1e-06,
"rope_scaling": null,
"rope_theta": 1000000,
"sliding_window": null,
"tie_word_embeddings": true,
"torch_dtype": "bfloat16",
"transformers_version": "4.52.4",
"use_cache": false,
"use_sliding_window": false,
"vocab_size": 151936
}

Model config Qwen3Config {
"architectures": [
"Qwen3ForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 151643,
"eos_token_id": 151645,
"head_dim": 128,
"hidden_act": "silu",
"hidden_size": 2560,
"initializer_range": 0.02,
"intermediate_size": 9728,
"max_position_embeddings": 40960,
"max_window_layers": 36,
"model_type": "qwen3",
"num_attention_heads": 32,
"num_hidden_layers": 36,
"num_key_value_heads": 8,
"rms_norm_eps": 1e-06,
"rope_scaling": null,
"rope_theta": 1000000,
"sliding_window": null,
"tie_word_embeddings": true,
"torch_dtype": "bfloat16",
"transformers_version": "4.52.4",
"use_cache": false,
"use_sliding_window": false,
"vocab_size": 151936
}

2025-06-10 14:05:48,213 transformers.models.auto.image_processing_auto 1111 INFO Could not locate the image processor configuration file, will try to use the model config instead.
Could not locate the image processor configuration file, will try to use the model config instead.
INFO 06-10 14:05:57 [config.py:717] This model supports multiple tasks: {'score', 'reward', 'generate', 'embed', 'classify'}. Defaulting to 'generate'.
WARNING 06-10 14:05:57 [arg_utils.py:1663] Detected VLLM_USE_V1=1 with Engine in background thread. Usage should be considered experimental. Please report any issues on Github.
INFO 06-10 14:05:57 [config.py:2003] Chunked prefill is enabled with max_num_batched_tokens=2048.
2025-06-10 14:05:57,800 transformers.tokenization_utils_base 1111 INFO loading file vocab.json
loading file vocab.json
2025-06-10 14:05:57,800 transformers.tokenization_utils_base 1111 INFO loading file merges.txt
loading file merges.txt
2025-06-10 14:05:57,800 transformers.tokenization_utils_base 1111 INFO loading file tokenizer.json
loading file tokenizer.json
2025-06-10 14:05:57,800 transformers.tokenization_utils_base 1111 INFO loading file added_tokens.json
loading file added_tokens.json
2025-06-10 14:05:57,800 transformers.tokenization_utils_base 1111 INFO loading file special_tokens_map.json
loading file special_tokens_map.json
2025-06-10 14:05:57,800 transformers.tokenization_utils_base 1111 INFO loading file tokenizer_config.json
loading file tokenizer_config.json
2025-06-10 14:05:57,800 transformers.tokenization_utils_base 1111 INFO loading file chat_template.jinja
loading file chat_template.jinja
2025-06-10 14:05:58,166 transformers.tokenization_utils_base 1111 INFO Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2025-06-10 14:05:58,251 transformers.generation.configuration_utils 1111 INFO loading configuration file /path/to/saves/Qwen3-4B/full/qwen3_v0.19/generation_config.json
loading configuration file /path/to/saves/Qwen3-4B/full/qwen3_v0.19/generation_config.json
2025-06-10 14:05:58,252 transformers.generation.configuration_utils 1111 INFO Generate config GenerationConfig {
"bos_token_id": 151643,
"do_sample": true,
"eos_token_id": [
151645,
151643
],
"pad_token_id": 151643,
"temperature": 0.6,
"top_k": 20,
"top_p": 0.95
}

Generate config GenerationConfig {
"bos_token_id": 151643,
"do_sample": true,
"eos_token_id": [
151645,
151643
],
"pad_token_id": 151643,
"temperature": 0.6,
"top_k": 20,
"top_p": 0.95
}

INFO 06-10 14:05:58 [core.py:58] Initializing a V1 LLM engine (v0.8.5) with config: model='/path/to/saves/Qwen3-4B/full/qwen3_v0.19', speculative_config=None, tokenizer='/path/to/saves/Qwen3-4B/full/qwen3_v0.19', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=40960, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='auto', reasoning_backend=None), observability_config=ObservabilityConfig(show_hidden_metrics=False, otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=None, served_model_name=/path/to/saves/Qwen3-4B/full/qwen3_v0.19, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"level":3,"custom_ops":["none"],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":512}
WARNING 06-10 14:05:59 [utils.py:2522] Methods determine_num_available_blocks,device_config,get_cache_block_size_bytes,initialize_cache not implemented in <vllm.v1.worker.gpu_worker.Worker object at 0x7fd3ad7f1b40>
INFO 06-10 14:05:59 [parallel_state.py:1004] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0
INFO 06-10 14:05:59 [cuda.py:221] Using Flash Attention backend on V1 engine.
INFO 06-10 14:05:59 [topk_topp_sampler.py:44] Currently, FlashInfer top-p & top-k sampling sampler is disabled because FlashInfer>=v0.2.3 is not backward compatible. Falling back to the PyTorch-native implementation of top-p & top-k sampling.
INFO 06-10 14:06:00 [gpu_model_runner.py:1329] Starting to load model /path/to/saves/Qwen3-4B/full/qwen3_v0.19...
Loading safetensors checkpoint shards: 0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 50% Completed | 1/2 [00:00<00:00, 1.13it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:01<00:00, 1.32it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:01<00:00, 1.28it/s]

INFO 06-10 14:06:01 [loader.py:458] Loading weights took 1.63 seconds
INFO 06-10 14:06:02 [gpu_model_runner.py:1347] Model loading took 7.5552 GiB and 1.903462 seconds
INFO 06-10 14:06:14 [backends.py:420] Using cache directory: /data/openmind_hub/vllm/torch_compile_cache/d91d8d3a04/rank_0_0 for vLLM's torch.compile
INFO 06-10 14:06:14 [backends.py:430] Dynamo bytecode transform time: 12.67 s
INFO 06-10 14:06:24 [backends.py:118] Directly load the compiled graph(s) for shape None from the cache, took 9.022 s
INFO 06-10 14:06:29 [monitor.py:33] torch.compile takes 12.67 s in total
INFO 06-10 14:06:30 [kv_cache_utils.py:634] GPU KV cache size: 449,520 tokens
INFO 06-10 14:06:30 [kv_cache_utils.py:637] Maximum concurrency for 40,960 tokens per request: 10.97x
INFO 06-10 14:06:56 [gpu_model_runner.py:1686] Graph capturing finished in 26 secs, took 0.59 GiB
INFO 06-10 14:06:56 [core.py:159] init engine (profile, create kv cache, warmup model) took 54.02 seconds
INFO 06-10 14:06:56 [core_client.py:439] Core engine process 0 ready.
2025-06-10 14:06:56,267 xinference.core.model 1111 INFO ModelActor(qwen3_v1.5.19-0) loaded
2025-06-10 14:06:56,269 xinference.core.worker 147 INFO [request ed043072-45c0-11f0-a0d2-0242ac110002] Leave launch_builtin_model, elapsed time: 80 s
2025-06-10 14:06:56,270 xinference.core.worker 147 INFO [request 1ccd36e6-45c1-11f0-a0d2-0242ac110002] Enter wait_for_load, args: <xinference.core.worker.WorkerActor object at 0x7ff1fb5e60c0>,qwen3_v1.5.19-0, kwargs:
2025-06-10 14:06:56,271 xinference.core.worker 147 INFO [request 1ccd36e6-45c1-11f0-a0d2-0242ac110002] Leave wait_for_load, elapsed time: 0 s

模型启动后， nvidia-smi 查看：

模型卸载日志：

2025-06-10 14:09:12,522 xinference.core.worker 147 INFO [request 6e03bfd0-45c1-11f0-a0d2-0242ac110002] Enter terminate_model, args: <xinference.core.worker.WorkerActor object at 0x7ff1fb5e60c0>, kwargs: model_uid=qwen3_v1.5.19-0
2025-06-10 14:09:12,525 xinference.model.llm.vllm.core 1111 INFO Stopping vLLM engine
[2025-06-10 14:09:12] INFO pool.py:432: Sub pool can't be killed: psutil.Process(pid=1111, name='Model: qwen3_', status='running', started='14:05:36')
2025-06-10 14:09:12,630 xinference.core.worker 147 INFO [request 6e03bfd0-45c1-11f0-a0d2-0242ac110002] Leave terminate_model, elapsed time: 0 s

[bao21987]

日志级别设为debug：

2025-06-10 14:47:32,420 xinference.core.supervisor 147 DEBUG [request c8dc4b2a-45c6-11f0-9359-0242ac110002] Enter terminate_model, args: <xinference.core.supervisor.SupervisorActor object at 0x7fb0bfade520>,qwen3_v1.5.19, kwargs:
2025-06-10 14:47:32,421 xinference.core.worker 147 INFO [request c8dc62fe-45c6-11f0-9359-0242ac110002] Enter terminate_model, args: <xinference.core.worker.WorkerActor object at 0x7fb27f1f61b0>, kwargs: model_uid=qwen3_v1.5.19-0
2025-06-10 14:47:32,423 xinference.core.worker 147 DEBUG Start to destroy model actor: ActorRef(uid=b'qwen3_v1.5.19-0', address='0.0.0.0:36431')
2025-06-10 14:47:32,425 xinference.model.llm.vllm.core 160 INFO Stopping vLLM engine
2025-06-10 14:47:32,428 xinference.core.worker 147 DEBUG Destroy model actor failed, model uid: qwen3_v1.5.19-0, error: [address=0.0.0.0:36431, pid=160] 'AsyncLLM' object has no attribute 'engine'
2025-06-10 14:47:32,428 xinference.core.worker 147 DEBUG Remove sub pools: ['0.0.0.0:36431']
[2025-06-10 14:47:32] INFO pool.py:432: Sub pool can't be killed: psutil.Process(pid=160, name='Model: qwen3_', status='zombie', started='14:43:13')
2025-06-10 14:47:32,530 xinference.core.worker 147 INFO [request c8dc62fe-45c6-11f0-9359-0242ac110002] Leave terminate_model, elapsed time: 0 s
2025-06-10 14:47:32,530 xinference.core.supervisor 147 DEBUG [request c8dc4b2a-45c6-11f0-9359-0242ac110002] Leave terminate_model, elapsed time: 0 s
2025-06-10 14:47:32,604 xinference.core.supervisor 147 DEBUG [request c8f8444c-45c6-11f0-9359-0242ac110002] Enter list_models, args: <xinference.core.supervisor.SupervisorActor object at 0x7fb0bfade520>, kwargs:
2025-06-10 14:47:32,604 xinference.core.worker 147 DEBUG [request c8f852ac-45c6-11f0-9359-0242ac110002] Enter list_models, args: <xinference.core.worker.WorkerActor object at 0x7fb27f1f61b0>, kwargs:
2025-06-10 14:47:32,604 xinference.core.worker 147 DEBUG [request c8f852ac-45c6-11f0-9359-0242ac110002] Leave list_models, elapsed time: 0 s
2025-06-10 14:47:32,604 xinference.core.supervisor 147 DEBUG [request c8f8444c-45c6-11f0-9359-0242ac110002] Leave list_models, elapsed time: 0 s

[qinxuye]

应该模型加载也没成功吧？

[bao21987]

@qinxuye 模型加载成功，推理正常，比vllm v0 engine推理性能好很多。只是卸载不掉

[qinxuye]

奇怪，pip show xoscar 是 0.7.8 了吗？然后更新了 xoscar 重启服务了么

[bao21987]

@qinxuye

重新制作了镜像，然后用新的镜像重启了实例

FROM xprobe/xinference:v1.6.1

RUN ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && echo 'Asia/Shanghai' >/etc/timezone

install python packages

RUN pip3 config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
&& pip3 install xoscar==0.7.8

[qinxuye]

目前你是指定 VLLM_USE_V1=1 来启动的吧？

[bao21987]

是的，指定了VLLM_USE_V1=1

[qinxuye]

我们复现下。

[bao21987]

@qinxuye 能复现吗，是否需要提供额外的信息

[qinxuye]

我试了下，最新的 main 分支可以正常停止。等本周版本发布了再试下。

[bao21987]

@qinxuye 使用xprobe/xinference:v1.7.0.post1 问题仍然存在