Skip to content

feat: Return context response immediately when stream_interval > 1 #5836

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Jul 8, 2025
Merged

feat: Return context response immediately when stream_interval > 1 #5836

merged 2 commits into from
Jul 8, 2025

Conversation

kaiyux
Copy link
Member

@kaiyux kaiyux commented Jul 8, 2025 •
edited
Loading

This helps guarantee TTFT.

@kaiyux kaiyux requested a review from a team as a code owner July 8, 2025 09:05
@kaiyux kaiyux requested a review from achartier July 8, 2025 09:05
@kaiyux
Copy link
Member Author

kaiyux commented Jul 8, 2025

/bot run

@kaiyux kaiyux requested review from QiJune and lfr-0531 July 8, 2025 09:09
lfr-0531
lfr-0531 approved these changes Jul 8, 2025
View reviewed changes
Copy link
Collaborator

@lfr-0531 lfr-0531 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM~

@kaiyux kaiyux requested review from Superjomn and a team July 8, 2025 09:11
@tensorrt-cicd
Copy link
Collaborator

PR_Github #11272 [ run ] triggered by Bot

Superjomn
Superjomn reviewed Jul 8, 2025
View reviewed changes
Superjomn
Superjomn reviewed Jul 8, 2025
View reviewed changes
@kaiyux
Minor update
e28f32d 
Signed-off-by: Kaiyu Xie <[email protected]>
@kaiyux
Copy link
Member Author

kaiyux commented Jul 8, 2025

/bot run

@tensorrt-cicd
Copy link
Collaborator

PR_Github #11277 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #11272 [ run ] completed with state ABORTED

Superjomn
Superjomn approved these changes Jul 8, 2025
View reviewed changes
Copy link
Collaborator

@Superjomn Superjomn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kaiyux kaiyux enabled auto-merge (squash) July 8, 2025 12:39
@tensorrt-cicd
Copy link
Collaborator

PR_Github #11277 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #8342 completed with status: 'SUCCESS'
Pipeline passed with automatic retried tests. Check the rerun report for details.

@kaiyux kaiyux merged commit bb5b16f into NVIDIA:main Jul 8, 2025
3 checks passed
nvzhihanj pushed a commit that referenced this pull request Jul 11, 2025
@kaiyux @nvzhihanj
feat: Return context response immediately when stream_interval > 1 (#…
155b19e 
…5836)

Signed-off-by: Kaiyu Xie <[email protected]>
zhou-yuxin pushed a commit to zhou-yuxin/TensorRT-LLM that referenced this pull request Jul 15, 2025
@kaiyux @zhou-yuxin
feat: Return context response immediately when stream_interval > 1 (N…
9ca06e7 
…VIDIA#5836)

Signed-off-by: Kaiyu Xie <[email protected]>
Signed-off-by: Yuxin <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Superjomn Superjomn Superjomn approved these changes

@QiJune QiJune QiJune approved these changes

@lfr-0531 lfr-0531 lfr-0531 approved these changes

@achartier achartier Awaiting requested review from achartier achartier is a code owner automatically assigned from NVIDIA/trt-llm-torch-runtime-devs

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@kaiyux @tensorrt-cicd @Superjomn @QiJune @lfr-0531