-
Notifications
You must be signed in to change notification settings - Fork 589
[Feature] support seed parameter #3161
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
Thanks for your contribution! |
@@ -69,6 +69,7 @@ def init_device(self): | |||
else: | |||
raise RuntimeError(f"Not support device type: {self.device_config.device}") | |||
|
|||
set_random_seed(self.fd_config.model_config.seed) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
XPU这里没改xpu_worker是合理的吗?建议做下XPU测试
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
已添加
fastdeploy/engine/args_utils.py
Outdated
@@ -316,6 +316,11 @@ class EngineArgs: | |||
Must be explicitly enabled via the `--enable-logprob` startup parameter to output logprob values. | |||
""" | |||
|
|||
seed: Optional[int] = None |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
这里看注释默认是随机生成,但上面config.py却默认给了0
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
已修改,vllm默认为0,每次固定输出结果,已修改
@@ -484,6 +489,12 @@ def add_cli_args(parser: FlexibleArgumentParser) -> FlexibleArgumentParser: | |||
default=EngineArgs.enable_logprob, | |||
help="Enable output of token-level log probabilities.", | |||
) | |||
model_group.add_argument( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
这里也给了默认为0的值
@@ -43,6 +43,7 @@ class SamplingMetadata: | |||
top_p: paddle.Tensor | |||
top_k: Optional[paddle.Tensor] = None | |||
min_p: Optional[paddle.Tensor] = None | |||
seed: Optional[paddle.Tensor] = None |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
seed值应该也需要做合法性检查
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
已修改,做合法性检查
seed_value = -1 | ||
else: | ||
seed_value = int(sampling_metadata.seed[0, 0]) | ||
_, next_tokens = top_k_top_p_sampling(probs, sampling_metadata.top_p, sampling_metadata.top_k, seed=seed_value) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
需要额外再判断sampling_metadata吗,是不是直接引用就行,性能更优?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
已修改,改为直接传入sampling_metadata.seed[0,0]
@@ -1112,8 +1108,6 @@ def _dummy_run( | |||
self.proposer.run(share_inputs=self.share_inputs) | |||
|
|||
# 7. Updata 'infer_seed' and step_cuda() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
这里做了这个删除是不是有问题,最早前内部做这个自增操作记得是为了解决某个采样问题的,可以跟lizhenyu确认下
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
已恢复
XPU的跑过测试了吗 |
* support seed * fix * add SamplingMetadata seed test * The next_tokens values are inconsistent! * add air and rejection seed test * fix * add SamplingParams seed test * fix seed=0 * Default to defualt * fix * fix args_utils * fix review * fix review * fix * fix * add xpu,gcu,iluvatar support seed * fix
支持用户传入seed参数
示例用法:
1. 服务用法:
1.1 输出每次随机
也可以使用
1.2 固定输出
也可以使用
```2. 离线方式
2.1 输出随机
2.2 输出固定