Skip to content

Conversation

saiatmakuri
Copy link
Contributor

@saiatmakuri saiatmakuri commented Feb 13, 2024

Pull Request Summary

Bring over flag added in vllm-project/vllm#1976 to llm engine

Test Plan and Usage Guide

test locally
without the flag:

> curl -H "Content-Type: application/json" -H "Authorization: Basic <AUTH>" -d '{ "prompt": "Whats 2+2? ", "stop_sequences": ["\n"], "max_new_tokens": 10, "temperature": 0.1 }' http://localhost:5000/v1/llm/completions-sync?model_endpoint_name=llama-7b-vllm-test-temp
{"request_id":"907c759f-e704-4d32-98a5-4dbe388ec04a","output":{"text":"4.","num_prompt_tokens":9,"num_completion_tokens":3,"tokens":null}}

note: "4." above

with the flag:

> curl -H "Content-Type: application/json" -H "Authorization: Basic <AUTH>" -d '{ "prompt": "Whats 2+2? ", "stop_sequences": ["\n"], "max_new_tokens": 10, "temperature": 0.1, "include_stop_str_in_output": true }' http://localhost:5000/v1/llm/completions-sync?model_endpoint_name=llama-7b-vllm-test-temp
{"request_id":"7f465cd7-8d42-4470-96ab-6b8cdfb2b5b0","output":{"text":"4.\n","num_prompt_tokens":9,"num_completion_tokens":3,"tokens":null}}

note: "4.\n" above

@saiatmakuri saiatmakuri added the enhancement New feature or request label Feb 13, 2024
@saiatmakuri saiatmakuri self-assigned this Feb 13, 2024
@saiatmakuri saiatmakuri merged commit 5bff345 into main Feb 13, 2024
@saiatmakuri saiatmakuri deleted the saiatmakuri/include_stop_str_in_output branch February 13, 2024 22:15
@yunfeng-scale yunfeng-scale mentioned this pull request Mar 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants