Qwen3-Embedding-8B和Qwen3-Reranker-8B报错

[noahachao]

System Info / 系統信息

mac os
Embedding错信息Server error: 400 - [address=127.0.0.1:47065, pid=95140] Model Qwen3-Embedding-8B cannot be run on engine sentence_transformers.
Reranker报错信息 error: 500 - [address=127.0.0.1:58592, pid=1019] Error while deserializing header: MetadataIncompleteBuffer

Running Xinference with Docker? / 是否使用 Docker 运行 Xinfernece？

• docker / docker
• pip install / 通过 pip install 安装
• installation from source / 从源码安装

Version info / 版本信息

1.7.0post1

The command used to start Xinference / 用以启动 xinference 的命令

XINFERENCE_ENABLE_VIRTUAL_ENV=1 xinference-local

Reproduction / 复现过程

1.选择Qwen3-Reranker-8B或者 Qwen3-Embedding-8B
2.默认为副本1、设备cpu
3.点击小火箭启动

Expected behavior / 期待表现

希望能够正常运行

[qinxuye]

报错贴全错误栈，只有报错本身一点用都没有。

[noahachao]

报错贴全错误栈，只有报错本身一点用都没有。

是指全部日志么？
Embedding的报错：
Traceback (most recent call last):
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xinference/api/restful_api.py", line 1077, in launch_model
model_uid = await (await self._get_supervisor_ref()).launch_builtin_model(
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xoscar/backends/context.py", line 262, in send
return self._process_result_message(result)
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xoscar/backends/context.py", line 111, in _process_result_message
raise message.as_instanceof_cause()
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xoscar/backends/pool.py", line 689, in send
result = await self._run_coro(message.message_id, coro)
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xoscar/backends/pool.py", line 389, in _run_coro
return await coro
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xoscar/api.py", line 418, in on_receive
return await super().on_receive(message) # type: ignore
File "xoscar/core.pyx", line 564, in on_receive
raise ex
File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.on_receive
async with self._lock:
File "xoscar/core.pyx", line 527, in xoscar.core._BaseActor.on_receive
with debug_async_timeout('actor_lock_timeout',
File "xoscar/core.pyx", line 532, in xoscar.core._BaseActor.on_receive
result = await result
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xinference/core/supervisor.py", line 1190, in launch_builtin_model
await _launch_model()
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xinference/core/supervisor.py", line 1125, in _launch_model
subpool_address = await _launch_one_model(
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xinference/core/supervisor.py", line 1079, in _launch_one_model
subpool_address = await worker_ref.launch_builtin_model(
File "xoscar/core.pyx", line 290, in __pyx_actor_method_wrapper
async with lock:
File "xoscar/core.pyx", line 293, in xoscar.core.__pyx_actor_method_wrapper
result = await result
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xinference/core/utils.py", line 93, in wrapped
ret = await func(*args, **kwargs)
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xinference/core/worker.py", line 1030, in launch_builtin_model
model, model_description = await asyncio.to_thread(
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/asyncio/threads.py", line 25, in to_thread
return await loop.run_in_executor(None, func_call)
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xinference/model/core.py", line 95, in create_model_instance
return create_embedding_model_instance(
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xinference/model/embedding/core.py", line 315, in create_embedding_model_instance
embedding_cls = check_engine_by_model_name_and_engine(
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xinference/model/embedding/embed_family.py", line 128, in check_engine_by_model_name_and_engine
raise ValueError(f"Model {model_name} cannot be run on engine {model_engine}.")
ValueError: [address=127.0.0.1:47065, pid=95140] Model Qwen3-Embedding-8B cannot be run on engine sentence_transformers.

rerank的报错： | 0/5 [00:00<?, ?it/s]
2025-06-16 14:01:13,142 xinference.core.worker 95140 ERROR Failed to load model Qwen3-Reranker-8B-0
Traceback (most recent call last):
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xinference/core/worker.py", line 1113, in launch_builtin_model
await model_ref.load()
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xoscar/backends/context.py", line 262, in send
return self._process_result_message(result)
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xoscar/backends/context.py", line 111, in _process_result_message
raise message.as_instanceof_cause()
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xoscar/backends/pool.py", line 689, in send
result = await self._run_coro(message.message_id, coro)
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xoscar/backends/pool.py", line 389, in _run_coro
return await coro
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xoscar/api.py", line 418, in on_receive
return await super().on_receive(message) # type: ignore
File "xoscar/core.pyx", line 564, in on_receive
raise ex
File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.on_receive
async with self._lock:
File "xoscar/core.pyx", line 527, in xoscar.core._BaseActor.on_receive
with debug_async_timeout('actor_lock_timeout',
File "xoscar/core.pyx", line 532, in xoscar.core._BaseActor.on_receive
result = await result
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xinference/core/model.py", line 477, in load
await asyncio.to_thread(self._model.load)
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/asyncio/threads.py", line 25, in to_thread
return await loop.run_in_executor(None, func_call)
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xinference/model/rerank/core.py", line 260, in load
model = self._model = AutoModelForCausalLM.from_pretrained(
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/transformers/models/auto/auto_factory.py", line 571, in from_pretrained
return model_class.from_pretrained(
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/transformers/modeling_utils.py", line 279, in _wrapper
return func(*args, **kwargs)
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/transformers/modeling_utils.py", line 4400, in from_pretrained
) = cls._load_pretrained_model(
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/transformers/modeling_utils.py", line 4818, in _load_pretrained_model
state_dict = load_state_dict(
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/transformers/modeling_utils.py", line 513, in load_state_dict
with safe_open(checkpoint_file, framework="pt") as f:
safetensors_rust.SafetensorError: [address=127.0.0.1:64201, pid=2544] Error while deserializing header: MetadataIncompleteBuffer
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
2025-06-16 14:01:13,177 xinference.core.worker 95140 ERROR [request 4c228b70-4a77-11f0-959f-3a69ab632b02] Leave launch_builtin_model, error: [address=127.0.0.1:64201, pid=2544] Error while deserializing header: MetadataIncompleteBuffer, elapsed time: 4 s
Traceback (most recent call last):
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xinference/core/utils.py", line 93, in wrapped
ret = await func(*args, **kwargs)
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xinference/core/worker.py", line 1113, in launch_builtin_model
await model_ref.load()
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xoscar/backends/context.py", line 262, in send
return self._process_result_message(result)
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xoscar/backends/context.py", line 111, in _process_result_message
raise message.as_instanceof_cause()
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xoscar/backends/pool.py", line 689, in send
result = await self._run_coro(message.message_id, coro)
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xoscar/backends/pool.py", line 389, in _run_coro
return await coro
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xoscar/api.py", line 418, in on_receive
return await super().on_receive(message) # type: ignore
File "xoscar/core.pyx", line 564, in on_receive
raise ex
File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.on_receive
async with self._lock:
File "xoscar/core.pyx", line 527, in xoscar.core._BaseActor.on_receive
with debug_async_timeout('actor_lock_timeout',
File "xoscar/core.pyx", line 532, in xoscar.core._BaseActor.on_receive
result = await result
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xinference/core/model.py", line 477, in load
await asyncio.to_thread(self._model.load)
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/asyncio/threads.py", line 25, in to_thread
return await loop.run_in_executor(None, func_call)
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xinference/model/rerank/core.py", line 260, in load
model = self._model = AutoModelForCausalLM.from_pretrained(
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/transformers/models/auto/auto_factory.py", line 571, in from_pretrained
return model_class.from_pretrained(
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/transformers/modeling_utils.py", line 279, in _wrapper
return func(*args, **kwargs)
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/transformers/modeling_utils.py", line 4400, in from_pretrained
) = cls._load_pretrained_model(
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/transformers/modeling_utils.py", line 4818, in _load_pretrained_model
state_dict = load_state_dict(
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/transformers/modeling_utils.py", line 513, in load_state_dict
with safe_open(checkpoint_file, framework="pt") as f:
safetensors_rust.SafetensorError: [address=127.0.0.1:64201, pid=2544] Error while deserializing header: MetadataIncompleteBuffer
2025-06-16 14:01:13,178 xinference.api.restful_api 95132 ERROR [address=127.0.0.1:64201, pid=2544] Error while deserializing header: MetadataIncompleteBuffer
Traceback (most recent call last):
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xinference/api/restful_api.py", line 1077, in launch_model
model_uid = await (await self._get_supervisor_ref()).launch_builtin_model(
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xoscar/backends/context.py", line 262, in send
return self._process_result_message(result)
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xoscar/backends/context.py", line 111, in _process_result_message
raise message.as_instanceof_cause()
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xoscar/backends/pool.py", line 689, in send
result = await self._run_coro(message.message_id, coro)
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xoscar/backends/pool.py", line 389, in _run_coro
return await coro
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xoscar/api.py", line 418, in on_receive
return await super().on_receive(message) # type: ignore
File "xoscar/core.pyx", line 564, in on_receive
raise ex
File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.on_receive
async with self._lock:
File "xoscar/core.pyx", line 527, in xoscar.core._BaseActor.on_receive
with debug_async_timeout('actor_lock_timeout',
File "xoscar/core.pyx", line 532, in xoscar.core._BaseActor.on_receive
result = await result
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xinference/core/supervisor.py", line 1190, in launch_builtin_model
await _launch_model()
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xinference/core/supervisor.py", line 1125, in _launch_model
subpool_address = await _launch_one_model(
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xinference/core/supervisor.py", line 1079, in _launch_one_model
subpool_address = await worker_ref.launch_builtin_model(
File "xoscar/core.pyx", line 290, in __pyx_actor_method_wrapper
async with lock:
File "xoscar/core.pyx", line 293, in xoscar.core.__pyx_actor_method_wrapper
result = await result
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xinference/core/utils.py", line 93, in wrapped
ret = await func(*args, **kwargs)
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xinference/core/worker.py", line 1113, in launch_builtin_model
await model_ref.load()
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xoscar/backends/context.py", line 262, in send
return self._process_result_message(result)
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xoscar/backends/context.py", line 111, in _process_result_message
raise message.as_instanceof_cause()
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xoscar/backends/pool.py", line 689, in send
result = await self._run_coro(message.message_id, coro)
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xoscar/backends/pool.py", line 389, in _run_coro
return await coro
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xoscar/api.py", line 418, in on_receive
return await super().on_receive(message) # type: ignore
File "xoscar/core.pyx", line 564, in on_receive
raise ex
File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.on_receive
async with self._lock:
File "xoscar/core.pyx", line 527, in xoscar.core._BaseActor.on_receive
with debug_async_timeout('actor_lock_timeout',
File "xoscar/core.pyx", line 532, in xoscar.core._BaseActor.on_receive
result = await result
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xinference/core/model.py", line 477, in load
await asyncio.to_thread(self._model.load)
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/asyncio/threads.py", line 25, in to_thread
return await loop.run_in_executor(None, func_call)
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/xinference/model/rerank/core.py", line 260, in load
model = self._model = AutoModelForCausalLM.from_pretrained(
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/transformers/models/auto/auto_factory.py", line 571, in from_pretrained
return model_class.from_pretrained(
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/transformers/modeling_utils.py", line 279, in _wrapper
return func(*args, **kwargs)
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/transformers/modeling_utils.py", line 4400, in from_pretrained
) = cls._load_pretrained_model(
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/transformers/modeling_utils.py", line 4818, in _load_pretrained_model
state_dict = load_state_dict(
File "/Users/blah/Library/Python/3.9/lib/python/site-packages/transformers/modeling_utils.py", line 513, in load_state_dict
with safe_open(checkpoint_file, framework="pt") as f:
safetensors_rust.SafetensorError: [address=127.0.0.1:64201, pid=2544] Error while deserializing header: MetadataIncompleteBuffer

[qinxuye]

权重文件看起来有问题。

[noahachao]

再追加另外一台设备rerank的报错：
2025-06-16 14:03:16,355 xinference.core.worker 11148 INFO [request 983354ae-4a77-11f0-88a1-4a51c5a13fc1] Enter launch_builtin_model, args: <xinference.core.worker.WorkerActor object at 0x149de3c70>, kwargs: model_uid=Qwen3-Reranker-8B-0,model_name=Qwen3-Reranker-8B,model_size_in_billions=None,model_format=None,quantization=None,model_engine=None,model_type=rerank,n_gpu=None,request_limits=None,peft_model_config=None,gpu_idx=None,download_hub=None,model_path=None,xavier_config=None
Using CPython 3.9.6 interpreter at: /Applications/Xcode.app/Contents/Developer/usr/bin/python3
Creating virtual environment at: .xinference/virtualenv/v2/Qwen3-Reranker-8B
Activate with: source .xinference/virtualenv/v2/Qwen3-Reranker-8B/bin/activate
2025-06-16 14:03:17,257 xinference.model.utils 11148 INFO Use model cache from a different hub.
/Users/noahwang/Library/Python/3.9/lib/python/site-packages/urllib3/init.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: urllib3/urllib3#3020
warnings.warn(
2025-06-16 14:03:19,841 xinference.core.model 14004 INFO Start requests handler.
2025-06-16 14:03:19,872 xinference.model.rerank.core 14004 INFO Loading rerank model: /Users/noahwang/.xinference/cache/Qwen3-Reranker-8B
2025-06-16 14:03:20,340 transformers.tokenization_utils_base 14004 INFO loading file vocab.json
loading file vocab.json
2025-06-16 14:03:20,340 transformers.tokenization_utils_base 14004 INFO loading file merges.txt
loading file merges.txt
2025-06-16 14:03:20,340 transformers.tokenization_utils_base 14004 INFO loading file tokenizer.json
loading file tokenizer.json
2025-06-16 14:03:20,340 transformers.tokenization_utils_base 14004 INFO loading file added_tokens.json
loading file added_tokens.json
2025-06-16 14:03:20,340 transformers.tokenization_utils_base 14004 INFO loading file special_tokens_map.json
loading file special_tokens_map.json
2025-06-16 14:03:20,340 transformers.tokenization_utils_base 14004 INFO loading file tokenizer_config.json
loading file tokenizer_config.json
2025-06-16 14:03:20,340 transformers.tokenization_utils_base 14004 INFO loading file chat_template.jinja
loading file chat_template.jinja
2025-06-16 14:03:20,493 transformers.tokenization_utils_base 14004 INFO Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2025-06-16 14:03:20,493 xinference.model.rerank.core 14004 WARNING Can't determine the rerank type based on the tokenizer Qwen2TokenizerFast(name_or_path='/Users/noahwang/.xinference/cache/Qwen3-Reranker-8B', vocab_size=151643, model_max_length=131072, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'eos_token': '<|im_end|>', 'pad_token': '<|endoftext|>', 'additional_special_tokens': ['<|im_start|>', '<|im_end|>', '<|object_ref_start|>', '<|object_ref_end|>', '<|box_start|>', '<|box_end|>', '<|quad_start|>', '<|quad_end|>', '<|vision_start|>', '<|vision_end|>', '<|vision_pad|>', '<|image_pad|>', '<|video_pad|>']}, clean_up_tokenization_spaces=False, added_tokens_decoder={
151643: AddedToken("<|endoftext|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151644: AddedToken("<|im_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151645: AddedToken("<|im_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151646: AddedToken("<|object_ref_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151647: AddedToken("<|object_ref_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151648: AddedToken("<|box_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151649: AddedToken("<|box_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151650: AddedToken("<|quad_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151651: AddedToken("<|quad_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151652: AddedToken("<|vision_start|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151653: AddedToken("<|vision_end|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151654: AddedToken("<|vision_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151655: AddedToken("<|image_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151656: AddedToken("<|video_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=True),
151657: AddedToken("<tool_call>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151658: AddedToken("</tool_call>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151659: AddedToken("<|fim_prefix|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151660: AddedToken("<|fim_middle|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151661: AddedToken("<|fim_suffix|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151662: AddedToken("<|fim_pad|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151663: AddedToken("<|repo_name|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151664: AddedToken("<|file_sep|>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151665: AddedToken("<tool_response>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151666: AddedToken("</tool_response>", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151667: AddedToken("", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
151668: AddedToken("", rstrip=False, lstrip=False, single_word=False, normalized=False, special=False),
}
), use normal type by default.
2025-06-16 14:03:20,502 transformers.tokenization_utils_base 14004 INFO loading file vocab.json
loading file vocab.json
2025-06-16 14:03:20,502 transformers.tokenization_utils_base 14004 INFO loading file merges.txt
loading file merges.txt
2025-06-16 14:03:20,503 transformers.tokenization_utils_base 14004 INFO loading file tokenizer.json
loading file tokenizer.json
2025-06-16 14:03:20,503 transformers.tokenization_utils_base 14004 INFO loading file added_tokens.json
loading file added_tokens.json
2025-06-16 14:03:20,503 transformers.tokenization_utils_base 14004 INFO loading file special_tokens_map.json
loading file special_tokens_map.json
2025-06-16 14:03:20,503 transformers.tokenization_utils_base 14004 INFO loading file tokenizer_config.json
loading file tokenizer_config.json
2025-06-16 14:03:20,503 transformers.tokenization_utils_base 14004 INFO loading file chat_template.jinja
loading file chat_template.jinja
2025-06-16 14:03:20,632 transformers.tokenization_utils_base 14004 INFO Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
2025-06-16 14:03:20,633 transformers.configuration_utils 14004 INFO loading configuration file /Users/noahwang/.xinference/cache/Qwen3-Reranker-8B/config.json
loading configuration file /Users/noahwang/.xinference/cache/Qwen3-Reranker-8B/config.json
2025-06-16 14:03:20,634 transformers.configuration_utils 14004 INFO Model config Qwen3Config {
"architectures": [
"Qwen3ForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 151643,
"eos_token_id": 151645,
"head_dim": 128,
"hidden_act": "silu",
"hidden_size": 4096,
"initializer_range": 0.02,
"intermediate_size": 12288,
"max_position_embeddings": 40960,
"max_window_layers": 36,
"model_type": "qwen3",
"num_attention_heads": 32,
"num_hidden_layers": 36,
"num_key_value_heads": 8,
"rms_norm_eps": 1e-06,
"rope_scaling": null,
"rope_theta": 1000000,
"sliding_window": null,
"tie_word_embeddings": false,
"torch_dtype": "bfloat16",
"transformers_version": "4.51.3",
"use_cache": true,
"use_sliding_window": false,
"vocab_size": 151669
}

Model config Qwen3Config {
"architectures": [
"Qwen3ForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 151643,
"eos_token_id": 151645,
"head_dim": 128,
"hidden_act": "silu",
"hidden_size": 4096,
"initializer_range": 0.02,
"intermediate_size": 12288,
"max_position_embeddings": 40960,
"max_window_layers": 36,
"model_type": "qwen3",
"num_attention_heads": 32,
"num_hidden_layers": 36,
"num_key_value_heads": 8,
"rms_norm_eps": 1e-06,
"rope_scaling": null,
"rope_theta": 1000000,
"sliding_window": null,
"tie_word_embeddings": false,
"torch_dtype": "bfloat16",
"transformers_version": "4.51.3",
"use_cache": true,
"use_sliding_window": false,
"vocab_size": 151669
}

2025-06-16 14:03:21,714 transformers.modeling_utils 14004 INFO loading weights file /Users/noahwang/.xinference/cache/Qwen3-Reranker-8B/model.safetensors.index.json
loading weights file /Users/noahwang/.xinference/cache/Qwen3-Reranker-8B/model.safetensors.index.json
2025-06-16 14:03:21,715 transformers.generation.configuration_utils 14004 INFO Generate config GenerationConfig {
"bos_token_id": 151643,
"eos_token_id": 151645
}

Generate config GenerationConfig {
"bos_token_id": 151643,
"eos_token_id": 151645
}

Loading checkpoint shards: 100%|██████████████████████████████████████████████| 5/5 [00:00<00:00, 2712.65it/s]
2025-06-16 14:03:21,756 transformers.modeling_utils 14004 INFO All model checkpoint weights were used when initializing Qwen3ForCausalLM.

All model checkpoint weights were used when initializing Qwen3ForCausalLM.

2025-06-16 14:03:21,757 transformers.modeling_utils 14004 INFO All the weights of Qwen3ForCausalLM were initialized from the model checkpoint at /Users/noahwang/.xinference/cache/Qwen3-Reranker-8B.
If your task is similar to the task the model of the checkpoint was trained on, you can already use Qwen3ForCausalLM for predictions without further training.
All the weights of Qwen3ForCausalLM were initialized from the model checkpoint at /Users/noahwang/.xinference/cache/Qwen3-Reranker-8B.
If your task is similar to the task the model of the checkpoint was trained on, you can already use Qwen3ForCausalLM for predictions without further training.
2025-06-16 14:03:21,759 transformers.generation.configuration_utils 14004 INFO loading configuration file /Users/noahwang/.xinference/cache/Qwen3-Reranker-8B/generation_config.json
loading configuration file /Users/noahwang/.xinference/cache/Qwen3-Reranker-8B/generation_config.json
2025-06-16 14:03:21,759 transformers.generation.configuration_utils 14004 INFO Generate config GenerationConfig {
"bos_token_id": 151643,
"do_sample": true,
"eos_token_id": [
151645,
151643
],
"pad_token_id": 151643,
"temperature": 0.6,
"top_k": 20,
"top_p": 0.95
}

Generate config GenerationConfig {
"bos_token_id": 151643,
"do_sample": true,
"eos_token_id": [
151645,
151643
],
"pad_token_id": 151643,
"temperature": 0.6,
"top_k": 20,
"top_p": 0.95
}

2025-06-16 14:03:21,762 xinference.core.worker 11148 ERROR Failed to load model Qwen3-Reranker-8B-0
Traceback (most recent call last):
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xinference/core/worker.py", line 1113, in launch_builtin_model
await model_ref.load()
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xoscar/backends/context.py", line 262, in send
return self._process_result_message(result)
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xoscar/backends/context.py", line 111, in _process_result_message
raise message.as_instanceof_cause()
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xoscar/backends/pool.py", line 689, in send
result = await self._run_coro(message.message_id, coro)
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xoscar/backends/pool.py", line 389, in _run_coro
return await coro
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xoscar/api.py", line 418, in on_receive
return await super().on_receive(message) # type: ignore
File "xoscar/core.pyx", line 564, in on_receive
raise ex
File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.on_receive
async with self._lock:
File "xoscar/core.pyx", line 527, in xoscar.core._BaseActor.on_receive
with debug_async_timeout('actor_lock_timeout',
File "xoscar/core.pyx", line 532, in xoscar.core._BaseActor.on_receive
result = await result
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xinference/core/model.py", line 477, in load
await asyncio.to_thread(self._model.load)
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/asyncio/threads.py", line 25, in to_thread
return await loop.run_in_executor(None, func_call)
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xinference/model/rerank/core.py", line 260, in load
model = self._model = AutoModelForCausalLM.from_pretrained(
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/transformers/models/auto/auto_factory.py", line 571, in from_pretrained
return model_class.from_pretrained(
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/transformers/modeling_utils.py", line 279, in _wrapper
return func(*args, **kwargs)
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/transformers/modeling_utils.py", line 4475, in from_pretrained
dispatch_model(model, **device_map_kwargs)
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/accelerate/big_modeling.py", line 501, in dispatch_model
raise ValueError(
ValueError: [address=0.0.0.0:57026, pid=14004] You are trying to offload the whole model to the disk. Please use the disk_offload function instead.
2025-06-16 14:03:21,780 xinference.core.worker 11148 ERROR [request 983354ae-4a77-11f0-88a1-4a51c5a13fc1] Leave launch_builtin_model, error: [address=0.0.0.0:57026, pid=14004] You are trying to offload the whole model to the disk. Please use the disk_offload function instead., elapsed time: 5 s
Traceback (most recent call last):
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xinference/core/utils.py", line 93, in wrapped
ret = await func(*args, **kwargs)
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xinference/core/worker.py", line 1113, in launch_builtin_model
await model_ref.load()
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xoscar/backends/context.py", line 262, in send
return self._process_result_message(result)
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xoscar/backends/context.py", line 111, in _process_result_message
raise message.as_instanceof_cause()
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xoscar/backends/pool.py", line 689, in send
result = await self._run_coro(message.message_id, coro)
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xoscar/backends/pool.py", line 389, in _run_coro
return await coro
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xoscar/api.py", line 418, in on_receive
return await super().on_receive(message) # type: ignore
File "xoscar/core.pyx", line 564, in on_receive
raise ex
File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.on_receive
async with self._lock:
File "xoscar/core.pyx", line 527, in xoscar.core._BaseActor.on_receive
with debug_async_timeout('actor_lock_timeout',
File "xoscar/core.pyx", line 532, in xoscar.core._BaseActor.on_receive
result = await result
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xinference/core/model.py", line 477, in load
await asyncio.to_thread(self._model.load)
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/asyncio/threads.py", line 25, in to_thread
return await loop.run_in_executor(None, func_call)
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xinference/model/rerank/core.py", line 260, in load
model = self._model = AutoModelForCausalLM.from_pretrained(
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/transformers/models/auto/auto_factory.py", line 571, in from_pretrained
return model_class.from_pretrained(
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/transformers/modeling_utils.py", line 279, in _wrapper
return func(*args, **kwargs)
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/transformers/modeling_utils.py", line 4475, in from_pretrained
dispatch_model(model, **device_map_kwargs)
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/accelerate/big_modeling.py", line 501, in dispatch_model
raise ValueError(
ValueError: [address=0.0.0.0:57026, pid=14004] You are trying to offload the whole model to the disk. Please use the disk_offload function instead.
2025-06-16 14:03:21,782 xinference.api.restful_api 11104 ERROR [address=0.0.0.0:57026, pid=14004] You are trying to offload the whole model to the disk. Please use the disk_offload function instead.
Traceback (most recent call last):
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xinference/api/restful_api.py", line 1077, in launch_model
model_uid = await (await self._get_supervisor_ref()).launch_builtin_model(
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xoscar/backends/context.py", line 262, in send
return self._process_result_message(result)
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xoscar/backends/context.py", line 111, in _process_result_message
raise message.as_instanceof_cause()
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xoscar/backends/pool.py", line 689, in send
result = await self._run_coro(message.message_id, coro)
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xoscar/backends/pool.py", line 389, in _run_coro
return await coro
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xoscar/api.py", line 418, in on_receive
return await super().on_receive(message) # type: ignore
File "xoscar/core.pyx", line 564, in on_receive
raise ex
File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.on_receive
async with self._lock:
File "xoscar/core.pyx", line 527, in xoscar.core._BaseActor.on_receive
with debug_async_timeout('actor_lock_timeout',
File "xoscar/core.pyx", line 532, in xoscar.core._BaseActor.on_receive
result = await result
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xinference/core/supervisor.py", line 1190, in launch_builtin_model
await _launch_model()
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xinference/core/supervisor.py", line 1125, in _launch_model
subpool_address = await _launch_one_model(
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xinference/core/supervisor.py", line 1079, in _launch_one_model
subpool_address = await worker_ref.launch_builtin_model(
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xoscar/backends/context.py", line 262, in send
return self._process_result_message(result)
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xoscar/backends/context.py", line 111, in _process_result_message
raise message.as_instanceof_cause()
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xoscar/backends/pool.py", line 689, in send
result = await self._run_coro(message.message_id, coro)
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xoscar/backends/pool.py", line 389, in _run_coro
return await coro
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xoscar/api.py", line 418, in on_receive
return await super().on_receive(message) # type: ignore
File "xoscar/core.pyx", line 564, in on_receive
raise ex
File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.on_receive
async with self._lock:
File "xoscar/core.pyx", line 527, in xoscar.core._BaseActor.on_receive
with debug_async_timeout('actor_lock_timeout',
File "xoscar/core.pyx", line 532, in xoscar.core._BaseActor.on_receive
result = await result
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xinference/core/utils.py", line 93, in wrapped
ret = await func(*args, **kwargs)
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xinference/core/worker.py", line 1113, in launch_builtin_model
await model_ref.load()
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xoscar/backends/context.py", line 262, in send
return self._process_result_message(result)
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xoscar/backends/context.py", line 111, in _process_result_message
raise message.as_instanceof_cause()
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xoscar/backends/pool.py", line 689, in send
result = await self._run_coro(message.message_id, coro)
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xoscar/backends/pool.py", line 389, in _run_coro
return await coro
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xoscar/api.py", line 418, in on_receive
return await super().on_receive(message) # type: ignore
File "xoscar/core.pyx", line 564, in on_receive
raise ex
File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.on_receive
async with self._lock:
File "xoscar/core.pyx", line 527, in xoscar.core._BaseActor.on_receive
with debug_async_timeout('actor_lock_timeout',
File "xoscar/core.pyx", line 532, in xoscar.core._BaseActor.on_receive
result = await result
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xinference/core/model.py", line 477, in load
await asyncio.to_thread(self._model.load)
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/asyncio/threads.py", line 25, in to_thread
return await loop.run_in_executor(None, func_call)
File "/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/xinference/model/rerank/core.py", line 260, in load
model = self._model = AutoModelForCausalLM.from_pretrained(
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/transformers/models/auto/auto_factory.py", line 571, in from_pretrained
return model_class.from_pretrained(
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/transformers/modeling_utils.py", line 279, in _wrapper
return func(*args, **kwargs)
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/transformers/modeling_utils.py", line 4475, in from_pretrained
dispatch_model(model, **device_map_kwargs)
File "/Users/noahwang/Library/Python/3.9/lib/python/site-packages/accelerate/big_modeling.py", line 501, in dispatch_model
raise ValueError(
ValueError: [address=0.0.0.0:57026, pid=14004] You are trying to offload the whole model to the disk. Please use the disk_offload function instead.
/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '

[RayDean]

我遇到了同样的问题：在一个容器中启动Qwen3-Embedding-8B和Qwen3-Reranker-8B，Reranker可以正常启动，embedding报错： File "/data/br/base/python/lib/python3.12/site-packages/xinference/model/embedding/embed_family.py", line 128, in check_engine_by_model_name_and_engine
raise ValueError(f"Model {model_name} cannot be run on engine {model_engine}.")
ValueError: Model Qwen3-Embedding-8B cannot be run on engine sentence_transformers.，我修改启动参数，添加了--model-engine vLLM，结果报错：ValueError: [address=0.0.0.0:37918, pid=206] Model Qwen3-Embedding-8B cannot be run on engine vLLM.
看来，无法用默认的sentence_transformers和vLLM来启动 Qwen3-Embedding-8B，怎么修改了？
附：xinference 版本：1.7.0post1, docker部署，python版本：3.12.9

[qinxuye]

sentence_transformers 和 vllm 这两个包都装了吗？

[RayDean]

安装sentence_transformers 后embedding和reranker模型都运行正常，谢谢老铁。

[github-actions]

This issue is stale because it has been open for 7 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 5 days since being marked as stale.