Skip to content

Conversation

yunfeng-scale
Copy link
Contributor

@yunfeng-scale yunfeng-scale commented Mar 20, 2024

Pull Request Summary

Support guided_json, guided_regex, guided_choice from vLLM

Test Plan and Usage Guide

tested within vllm entrypoint and from model engine

curl localhost:5005/predict -d '{"prompt":"hello", "guided_regex":"a beautiful world .*"}'
{"text": "a beautiful world is fuxix", "count_prompt_tokens": 2, "count_output_tokens": 16, "log_probs": null, "tokens": ["a", " ", "b", "ea", "u", "t", "iful", " ", "wo", "r", "l", "d", " is", " f", "ux", "ix"]}

@yunfeng-scale yunfeng-scale requested a review from a team March 20, 2024 19:02
@yunfeng-scale yunfeng-scale merged commit 5f6cd32 into main Mar 21, 2024
@yunfeng-scale yunfeng-scale deleted the yunfeng-guided-decoding branch March 21, 2024 04:47
This was referenced Mar 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants