Skip to content

[TRTLLM-5801][infra] Add more RTX Pro 6000 test stages #5126

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 16 commits into from
Aug 22, 2025
Merged

[TRTLLM-5801][infra] Add more RTX Pro 6000 test stages #5126

merged 16 commits into from
Aug 22, 2025

Conversation

EmmaQiaoCh
Copy link
Collaborator

@EmmaQiaoCh EmmaQiaoCh commented Jun 11, 2025 •
edited by coderabbitai bot
Loading

Summary by CodeRabbit

  • Tests
    • Updated GPU test matrix: replaced RTX 5090 PyTorch job with RTX Pro 6000 and added both 1‑GPU and multiple 4‑GPU PyTorch post-merge/DeepSeek suites; retired older redundant entries.
    • Moved one RTX Pro 6000 test from post-merge to pre-merge to run earlier in the pipeline.
  • Chores
    • CI now selects the appropriate NVIDIA driver automatically for RTX Pro 6000 vs other GPUs.

@EmmaQiaoCh
Copy link
Collaborator Author

/bot run --stage-list "RTXPro6000-4_GPUs-Pytorch-DeepSeek-[Post-Merge]-1"

@tensorrt-cicd
Copy link
Collaborator

PR_Github #8472 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #8472 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #6137 (Partly Tested) completed with status: 'ABORTED'

@EmmaQiaoCh
Copy link
Collaborator Author

/bot run --stage-list "RTXPro6000-4_GPUs-Pytorch-DeepSeek-[Post-Merge]-1"

@tensorrt-cicd
Copy link
Collaborator

PR_Github #8622 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #8622 [ run ] completed with state FAILURE
/LLM/main/L0_MergeRequest_PR pipeline #6258 (Partly Tested) completed with status: 'FAILURE'

@EmmaQiaoCh
Copy link
Collaborator Author

/bot run --stage-list "RTXPro6000-4_GPUs-Pytorch-DeepSeek-[Post-Merge]-1, RTXPro6000-4_GPUs-Pytorch-DeepSeek-[Post-Merge]-2" --disable-fail-fast

@tensorrt-cicd
Copy link
Collaborator

PR_Github #8731 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #8731 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #6334 (Partly Tested) completed with status: 'FAILURE'

@EmmaQiaoCh
Copy link
Collaborator Author

/bot run --stage-list "RTXPro6000-4_GPUs-Pytorch-DeepSeek-[Post-Merge]-1, RTXPro6000-4_GPUs-Pytorch-DeepSeek-[Post-Merge]-2, RTXPro6000-8_GPUs-Pytorch-DeepSeek-[Post-Merge]-1" --disable-fail-fast

@EmmaQiaoCh EmmaQiaoCh force-pushed the emma/add_rtxpro6000_tests branch from 07bfa14 to 6fad8bc Compare June 16, 2025 08:39
@EmmaQiaoCh
Copy link
Collaborator Author

/bot run --stage-list "RTXPro6000-4_GPUs-Pytorch-DeepSeek-[Post-Merge]-1, RTXPro6000-4_GPUs-Pytorch-DeepSeek-[Post-Merge]-2, RTXPro6000-8_GPUs-Pytorch-DeepSeek-[Post-Merge]-1" --disable-fail-fast

@tensorrt-cicd
Copy link
Collaborator

PR_Github #9005 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #9005 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #6575 (Partly Tested) completed with status: 'FAILURE'

@EmmaQiaoCh EmmaQiaoCh force-pushed the emma/add_rtxpro6000_tests branch from 6fad8bc to 2b104a1 Compare June 17, 2025 07:50
@EmmaQiaoCh
Copy link
Collaborator Author

/bot run --stage-list "RTXPro6000-4_GPUs-Pytorch-DeepSeek-[Post-Merge]-1, RTXPro6000-4_GPUs-Pytorch-DeepSeek-[Post-Merge]-2, RTXPro6000-8_GPUs-Pytorch-DeepSeek-[Post-Merge]-1" --disable-fail-fast

@tensorrt-cicd
Copy link
Collaborator

PR_Github #9161 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #9161 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #6711 (Partly Tested) completed with status: 'FAILURE'

@EmmaQiaoCh EmmaQiaoCh force-pushed the emma/add_rtxpro6000_tests branch from 8482a34 to 2955d4a Compare June 18, 2025 08:40
@EmmaQiaoCh
Copy link
Collaborator Author

/bot run --stage-list "RTXPro6000-4_GPUs-Pytorch-DeepSeek-[Post-Merge]-1, RTXPro6000-4_GPUs-Pytorch-DeepSeek-[Post-Merge]-2, RTXPro6000-8_GPUs-Pytorch-DeepSeek-[Post-Merge]-1" --disable-fail-fast

@tensorrt-cicd
Copy link
Collaborator

PR_Github #9350 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #9350 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #6863 (Partly Tested) completed with status: 'FAILURE'

@EmmaQiaoCh EmmaQiaoCh force-pushed the emma/add_rtxpro6000_tests branch from 6d71fd0 to 2f9a3dc Compare June 19, 2025 03:38
@EmmaQiaoCh
Copy link
Collaborator Author

/bot run --stage-list "RTXPro6000-4_GPUs-Pytorch-DeepSeek-[Post-Merge]-1, RTXPro6000-4_GPUs-Pytorch-DeepSeek-[Post-Merge]-2, RTXPro6000-8_GPUs-Pytorch-DeepSeek-[Post-Merge]-1" --disable-fail-fast --detailed-log

@tensorrt-cicd
Copy link
Collaborator

PR_Github #9443 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #9443 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #6934 (Partly Tested) completed with status: 'FAILURE'

@EmmaQiaoCh EmmaQiaoCh force-pushed the emma/add_rtxpro6000_tests branch from 273d387 to f7c4076 Compare June 24, 2025 09:17
@EmmaQiaoCh
Copy link
Collaborator Author

/bot run --stage-list "RTXPro6000-4_GPUs-Pytorch-DeepSeek-[Post-Merge]-1, RTXPro6000-4_GPUs-Pytorch-DeepSeek-[Post-Merge]-2, RTXPro6000-8_GPUs-Pytorch-DeepSeek-[Post-Merge]-1" --disable-fail-fast --detailed-log

@tensorrt-cicd
Copy link
Collaborator

PR_Github #9675 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #9675 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #7112 (Partly Tested) completed with status: 'FAILURE'

@EmmaQiaoCh
Add more ds cases on rtx-pro-6000
137290d 
Signed-off-by: qqiao <[email protected]>
@EmmaQiaoCh
Waive the hang cases first
70dcc21 
Signed-off-by: qqiao <[email protected]>
@EmmaQiaoCh
Test for latest open driver
a9cdf7f 
Signed-off-by: qqiao <[email protected]>
@EmmaQiaoCh
Run all the tests on 0082 which seems stable
e1e2421 
Signed-off-by: qqiao <[email protected]>
@EmmaQiaoCh
Add single gpu tests for rtxpro6000
c221e78 
Signed-off-by: qqiao <[email protected]>
@EmmaQiaoCh
Update for comments
30bd4a2 
Signed-off-by: qqiao <[email protected]>
@EmmaQiaoCh
Fix multi-gpu for rtxpro6000
975381a 
Signed-off-by: qqiao <[email protected]>
@EmmaQiaoCh
Test for node 081
62cd310 
Signed-off-by: qqiao <[email protected]>
@EmmaQiaoCh
Split the tests stage to test
41f4656 
Signed-off-by: qqiao <[email protected]>
@EmmaQiaoCh
Update cpu and mem for multi-gpu pod
823ebe1 
Signed-off-by: qqiao <[email protected]>
@EmmaQiaoCh
Test latest open driver for rtxpro6000
589b713 
Signed-off-by: qqiao <[email protected]>
@EmmaQiaoCh
move single gpu test to post-merge
b89e944 
Signed-off-by: qqiao <[email protected]>
@EmmaQiaoCh
Remove debug info
4515b14 
Signed-off-by: qqiao <[email protected]>
@EmmaQiaoCh EmmaQiaoCh force-pushed the emma/add_rtxpro6000_tests branch from 57f2efd to 4515b14 Compare August 22, 2025 02:35
@EmmaQiaoCh
Copy link
Collaborator Author

/bot run --stage-list "RTXPro6000-4_GPUs-Pytorch-Post-Merge-1, RTXPro6000-4_GPUs-Pytorch-Post-Merge-2, RTXPro6000-Pytorch-Post-Merge-1" --disable-fail-fast

@tensorrt-cicd
Copy link
Collaborator

PR_Github #16102 [ run ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #16102 [ run ] completed with state SUCCESS
/LLM/main/L0_MergeRequest_PR pipeline #12110 (Partly Tested) completed with status: 'SUCCESS'

@EmmaQiaoCh
Copy link
Collaborator Author

/bot skip --comment "rtxpro-6000 tests passed in previous CI, will skip others"

@EmmaQiaoCh EmmaQiaoCh enabled auto-merge (squash) August 22, 2025 06:49
@tensorrt-cicd
Copy link
Collaborator

PR_Github #16142 [ skip ] triggered by Bot

@tensorrt-cicd
Copy link
Collaborator

PR_Github #16142 [ skip ] completed with state SUCCESS
Skipping testing for commit 4515b14

@EmmaQiaoCh EmmaQiaoCh merged commit a49cf68 into NVIDIA:main Aug 22, 2025
4 checks passed
This was referenced Aug 22, 2025
Open
Draft
Merged
Merged
Draft
Draft
Merged
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@chzblych chzblych chzblych approved these changes

@coderabbitai coderabbitai[bot] coderabbitai[bot] left review comments

@peaceh-nv peaceh-nv peaceh-nv approved these changes

@niukuo niukuo Awaiting requested review from niukuo

@yiqingy0 yiqingy0 Awaiting requested review from yiqingy0

@ZhanruiSunCh ZhanruiSunCh Awaiting requested review from ZhanruiSunCh

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@EmmaQiaoCh @tensorrt-cicd @chzblych @peaceh-nv