Skip to content

chore: partition LLM class into TorchLLM and TrtLLM #4900

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Jun 18, 2025
Merged

chore: partition LLM class into TorchLLM and TrtLLM #4900

merged 3 commits into from
Jun 18, 2025

Conversation

Superjomn
Copy link
Collaborator

@Superjomn Superjomn commented Jun 4, 2025 •
edited
Loading

This PR does

  1. Create TorchLLM and TrtLLM with most of the code shared, making it ready to switch TorchLLM as the default LLM for the upcoming version
  2. Keep alias of tensorrt_llm._torch.llm.LLM for TorchLLM and tensorrt_llm.LLM for TrtLLM
  3. Fix the api reference doc accordingly, especially keep the existing LLM unbroken
    • Add separate API references for both TrtLLM and TorchLLM, add support for TrtLlmArgs and TorchLlmArgs.

Document:

LLM's API reference works as before:
image

@Superjomn Superjomn requested a review from a team as a code owner June 4, 2025 06:55
@Superjomn Superjomn requested review from hlu1 and shaharmor98 June 4, 2025 06:55
@Superjomn Superjomn force-pushed the partition-llm branch 2 times, most recently from 1ea3bf9 to 98964ed Compare June 4, 2025 07:02
@Superjomn Superjomn marked this pull request as draft June 4, 2025 07:40
@Superjomn Superjomn requested review from lucaslie and suyoggupta June 5, 2025 00:55
lucaslie
lucaslie reviewed Jun 5, 2025
View reviewed changes
@lucaslie lucaslie mentioned this pull request Jun 5, 2025
Merged
lucaslie
lucaslie reviewed Jun 5, 2025
View reviewed changes
lucaslie
lucaslie reviewed Jun 5, 2025
View reviewed changes
@Superjomn Superjomn force-pushed the partition-llm branch 2 times, most recently from d3c572d to 4333698 Compare June 9, 2025 05:17
@Superjomn Superjomn marked this pull request as ready for review June 9, 2025 05:18
@Superjomn
Copy link
Collaborator Author

/bot run --add-multi-gpu-test --disable-fail-fast

@tensorrt-cicd
Copy link
Collaborator

PR_Github #8090 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #8090 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #5867 completed with status: 'FAILURE'

@Superjomn
Copy link
Collaborator Author

/bot run --add-multi-gpu-test --disable-fail-fast

@tensorrt-cicd
Copy link
Collaborator

PR_Github #8130 [ run ] triggered by Bot

lucaslie
lucaslie reviewed Jun 9, 2025
View reviewed changes
@tensorrt-cicd
Copy link
Collaborator

PR_Github #8130 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #5894 completed with status: 'FAILURE'

@Superjomn
Copy link
Collaborator Author

/bot run --add-multi-gpu-test --disable-fail-fast

@tensorrt-cicd
Copy link
Collaborator

PR_Github #8277 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #8278 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #8277 [ run ] completed with state ABORTED

@tensorrt-cicd
Copy link
Collaborator

PR_Github #8279 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #8278 [ run ] completed with state ABORTED

@Superjomn
Copy link
Collaborator Author

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #8965 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #8965 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #6543 completed with status: 'FAILURE'

@Superjomn
Copy link
Collaborator Author

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #8992 [ run ] triggered by Bot

shaharmor98
shaharmor98 approved these changes Jun 16, 2025
View reviewed changes
Copy link
Collaborator

@shaharmor98 shaharmor98 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM . Left two nits, feel free to resolve if you don't agree.

@tensorrt-cicd
Copy link
Collaborator

PR_Github #8992 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #6565 completed with status: 'FAILURE'

@Superjomn
Copy link
Collaborator Author

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #9041 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #9041 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #6608 completed with status: 'FAILURE'

@Superjomn
Copy link
Collaborator Author

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #9179 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #9179 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #6725 completed with status: 'FAILURE'

@Superjomn
Copy link
Collaborator Author

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #9207 [ run ] triggered by Bot

Superjomn added 3 commits June 18, 2025 07:58
@Superjomn
create TorchLLM and TrtLLM
75e501c 
Signed-off-by: Superjomn <[email protected]>
@Superjomn
fix doc
ee83e59 
Signed-off-by: Superjomn <[email protected]>
@Superjomn
fix comment
6abfc9b 
Signed-off-by: Superjomn <[email protected]>
@Superjomn
Copy link
Collaborator Author

/bot run

@Superjomn Superjomn requested review from QiJune and removed request for hlu1 June 17, 2025 23:59
@tensorrt-cicd
Copy link
Collaborator

PR_Github #9253 [ run ] triggered by Bot

@Superjomn Superjomn enabled auto-merge (squash) June 18, 2025 05:01
@tensorrt-cicd
Copy link
Collaborator

PR_Github #9253 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #6789 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

@Superjomn Superjomn merged commit 724e495 into NVIDIA:main Jun 18, 2025
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lucaslie lucaslie lucaslie approved these changes

@shaharmor98 shaharmor98 shaharmor98 approved these changes

@suyoggupta suyoggupta Awaiting requested review from suyoggupta

@QiJune QiJune Awaiting requested review from QiJune

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@Superjomn @tensorrt-cicd @lucaslie @shaharmor98