Skip to content

Conversation

ian-scale
Copy link
Contributor

Pull Request Summary

What is this PR changing? Why is this change being made? Any caveats you'd like to highlight? Link any relevant documents, links, or screenshots here if applicable.
adds tensorrt-llm to enum for valid inference frameworks. currently Model.list() is broken due to the addition of endpoints that use tensorrt-llm as an inference framework when this framework isn't an option in the client.

Ex:

ValidationError: 1 validation error for ListLLMEndpointsResponse
model_endpoints -> 13 -> inference_framework
  value is not a valid enumeration member; permitted: 'deepspeed', 'text_generation_inference', 'vllm', 'lightllm' (type=type_error.enum; enum_values=[<LLMInferenceFramework.DEEPSPEED: 'deepspeed'>, <LLMInferenceFramework.TEXT_GENERATION_INFERENCE: 'text_generation_inference'>, <LLMInferenceFramework.VLLM: 'vllm'>, <LLMInferenceFramework.LIGHTLLM: 'lightllm'>])

Test Plan and Usage Guide

How did you validate that your PR works correctly? How do you run or demo the code? Provide enough detail so a reviewer can reasonably reproduce the testing procedure. Paste example command line invocations if applicable.

@ian-scale ian-scale requested review from yixu34 and a team November 27, 2023 18:55
Copy link
Member

@yixu34 yixu34 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any tests to update?

@ian-scale ian-scale enabled auto-merge (squash) November 27, 2023 20:45
@ian-scale
Copy link
Contributor Author

Any tests to update?
no tests to update, but we may want to create something that tests this issue since we've seen it before.

@ian-scale ian-scale merged commit 4483dff into main Nov 27, 2023
@ian-scale ian-scale deleted the ianmacleod/fix_model_list branch November 27, 2023 21:16
@ian-scale ian-scale added the bug Something isn't working label Nov 29, 2023
@yunfeng-scale yunfeng-scale mentioned this pull request Mar 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants