Skip to content

Re-implement LlmResponse in Python to reduce host overhead of pybind #5224

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 43 commits into from
Jun 17, 2025
Merged

Re-implement LlmResponse in Python to reduce host overhead of pybind #5224

merged 43 commits into from
Jun 17, 2025

Conversation

QiJune
Copy link
Collaborator

@QiJune QiJune commented Jun 16, 2025

A step of #3034

QiJune added 30 commits June 5, 2025 15:32
@QiJune
cache tokens in Python side to reduce pybind reading overhead
f038a82 
Signed-off-by: QI JUN <[email protected]>
@QiJune
refine
bba6719 
Signed-off-by: QI JUN <[email protected]>
@QiJune
Merge branch 'main' into clean_prepare
f826f5a
@QiJune
Merge branch 'main' into clean_prepare
8a3e92a
@QiJune
Merge branch 'main' into clean_prepare
44b12cc
@QiJune
Merge branch 'main' into clean_prepare
a8bd10d
@QiJune
pure Python LlmResponse
3c83644 
Signed-off-by: QI JUN <[email protected]>
@QiJune
pure Python LlmResponse
f0bb7c8 
Signed-off-by: QI JUN <[email protected]>
@QiJune
clean
60ca761 
Signed-off-by: QI JUN <[email protected]>
@QiJune
fix
5f7e9ea 
Signed-off-by: QI JUN <[email protected]>
@QiJune
fix
1b3a7b7 
Signed-off-by: QI JUN <[email protected]>
@QiJune
fix
df60073 
Signed-off-by: QI JUN <[email protected]>
@QiJune
fix
91c904e 
Signed-off-by: QI JUN <[email protected]>
@QiJune
fix
5d62cea 
Signed-off-by: QI JUN <[email protected]>
@QiJune
fix
50eb5b9 
Signed-off-by: QI JUN <[email protected]>
@QiJune
fix
ea2f8cc 
Signed-off-by: QI JUN <[email protected]>
@QiJune
fix
431926d 
Signed-off-by: QI JUN <[email protected]>
@QiJune
fix
3fb4d84 
Signed-off-by: QI JUN <[email protected]>
@QiJune
polish
04370ba 
Signed-off-by: QI JUN <[email protected]>
@QiJune
expose createSerializedResult api
acd09a4 
Signed-off-by: QI JUN <[email protected]>
@QiJune
fix
48a999c 
Signed-off-by: QI JUN <[email protected]>
@QiJune
fix
1e36d77 
Signed-off-by: QI JUN <[email protected]>
@QiJune
fix
4fd71bc 
Signed-off-by: QI JUN <[email protected]>
@QiJune
fix
1a4920b 
Signed-off-by: QI JUN <[email protected]>
@QiJune
fix
5e9888a 
Signed-off-by: QI JUN <[email protected]>
@QiJune
fix
9bfa834 
Signed-off-by: QI JUN <[email protected]>
@QiJune
fix
b37703c 
Signed-off-by: QI JUN <[email protected]>
@QiJune
serialize result
3e4844d 
Signed-off-by: QI JUN <[email protected]>
@QiJune
fix
e5e3873 
Signed-off-by: QI JUN <[email protected]>
@QiJune
fix
39a791a 
Signed-off-by: QI JUN <[email protected]>
@tensorrt-cicd
Copy link
Collaborator

PR_Github #8960 [ run ] triggered by Bot

kaiyux
kaiyux reviewed Jun 16, 2025
View reviewed changes
@tensorrt-cicd
Copy link
Collaborator

PR_Github #8960 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #6539 completed with status: 'FAILURE'

@QiJune
Copy link
Collaborator Author

QiJune commented Jun 16, 2025

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #8970 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #8970 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #6547 completed with status: 'FAILURE'

Superjomn
Superjomn reviewed Jun 16, 2025
View reviewed changes
@QiJune
Copy link
Collaborator Author

QiJune commented Jun 16, 2025

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #8997 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #8997 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #6570 completed with status: 'FAILURE'

@QiJune
Copy link
Collaborator Author

QiJune commented Jun 16, 2025

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #9068 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #9068 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #6630 completed with status: 'FAILURE'

@QiJune
Copy link
Collaborator Author

QiJune commented Jun 17, 2025

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #9119 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #9119 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #6672 completed with status: 'SUCCESS'

@QiJune QiJune requested a review from lfr-0531 June 17, 2025 13:21
kaiyux
kaiyux previously approved these changes Jun 17, 2025
View reviewed changes
@QiJune QiJune enabled auto-merge (squash) June 17, 2025 13:24
@kaiyux kaiyux dismissed their stale review June 17, 2025 13:27

retry

@QiJune QiJune merged commit f899c4d into NVIDIA:main Jun 17, 2025
3 checks passed
@kaiyux kaiyux mentioned this pull request Jun 17, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Superjomn Superjomn Superjomn left review comments

@kaiyux kaiyux kaiyux approved these changes

@Naveassaf Naveassaf Awaiting requested review from Naveassaf Naveassaf is a code owner automatically assigned from NVIDIA/trt-llm-torch-runtime-devs

@lfr-0531 lfr-0531 Awaiting requested review from lfr-0531

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@QiJune @tensorrt-cicd @Superjomn @kaiyux