[TRTLLM-5863][feat] Support MoE INT8 Weight-Only-Quantization in PyTorch Workflow #6629
Conversation
📝 WalkthroughWalkthroughThis update extends weight-only quantization support for Mixture-of-Experts (MoE) layers, adding per-channel and group scaling variants. It introduces new quantization method classes, updates the MoE runner and custom operator interfaces, adjusts tensor shape and quantization parameter handling, and adds comprehensive tests for the new quantization modes. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant PyTorchModule
participant FusedMoERunner
participant CutlassMoERunner
participant QuantizationMethod
User->>PyTorchModule: Calls fused_moe(..., use_woq_per_channel, use_woq_group_scaling)
PyTorchModule->>FusedMoERunner: Instantiates with quantization flags
FusedMoERunner->>CutlassMoERunner: create_weight_quant_runner() based on flags
FusedMoERunner->>QuantizationMethod: Extracts quant scales and parameters
FusedMoERunner->>CutlassMoERunner: Runs kernel with quantization config
CutlassMoERunner-->>FusedMoERunner: Returns result
FusedMoERunner-->>PyTorchModule: Returns output
PyTorchModule-->>User: Output tensor(s)
Estimated code review effort🎯 4 (Complex) | ⏱️ ~40 minutes Suggested labels
Suggested reviewers
Note ⚡️ Unit Test Generation is now available in beta!Learn more here, or try it out under "Finishing Touches" below. 📜 Recent review detailsConfiguration used: .coderabbit.yaml 📒 Files selected for processing (1)
🧰 Additional context used🧠 Learnings (1)📓 Common learnings⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
🔇 Additional comments (9)
✨ Finishing Touches
🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
Yuening-wa
force-pushed
the
user/yueningl/moe_int8_weight_only_quant_support
branch
2 times, most recently
from
b9b6ffe to
107befa
Compare
|
/bot run |
|
PR_Github #14138 [ run ] triggered by Bot |
|
PR_Github #14138 [ run ] completed with state |
|
/bot run |
|
PR_Github #14211 [ run ] triggered by Bot |
|
PR_Github #14211 [ run ] completed with state |
Yuening-wa
force-pushed
the
user/yueningl/moe_int8_weight_only_quant_support
branch
from
107befa to
c26bf90
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
🧹 Nitpick comments (3)
tests/unittest/_torch/helpers.py (1)
78-87: Fix typo in function name.The function implementation is correct and follows proper quantization tolerance calculation principles. However, there's a spelling error in the function name.
Apply this diff to fix the typo:
-def calc_woq_tolerence(x: torch.Tensor, weight_dtype: torch.dtype): +def calc_woq_tolerance(x: torch.Tensor, weight_dtype: torch.dtype):Note: You'll also need to update the import statements in files that use this function to match the corrected name.
tensorrt_llm/_torch/modules/fused_moe/quantization.py (2)
942-951: Clean up commented INT4 code.The commented INT4 code should be removed to improve code clarity. If INT4 support is planned, track it in a TODO comment instead.
# preprocess the weights for mixed gemm if module.quant_config.layer_quant_mode.is_int8_weight_only(): weight_dtype = torch.int8 - # elif module.quant_config.layer_quant_mode.is_int4_weight_only(): - # weight_dtype = torch.quint4x2 - # packer = torch.ops.trtllm.pack_int8_tensor_to_packed_int4 - # unpacker = torch.ops.trtllm.unpack_int4_packed_tensor_to_int8 - # w31_weight_shard = packer( - # unpacker(w31_weight_shard.cpu()).T.contiguous()).to( - # w31_weight_shard.device) + # TODO: Add INT4 weight-only support in the future
972-981: Remove duplicate commented INT4 code.Similar to the previous method, remove the commented INT4 code to maintain code cleanliness.
# preprocess the weights for mixed gemm if module.quant_config.layer_quant_mode.is_int8_weight_only(): weight_dtype = torch.int8 - # elif module.quant_config.layer_quant_mode.is_int4_weight_only(): - # weight_dtype = torch.quint4x2 - # packer = torch.ops.trtllm.pack_int8_tensor_to_packed_int4 - # unpacker = torch.ops.trtllm.unpack_int4_packed_tensor_to_int8 - # w31_weight_shard = packer( - # unpacker(w31_weight_shard.cpu()).T.contiguous()).to( - # w31_weight_shard.device) + # TODO: Add INT4 weight-only support in the future
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (9)
cpp/tensorrt_llm/thop/moeOp.cpp(9 hunks)tensorrt_llm/_torch/custom_ops/torch_custom_ops.py(5 hunks)tensorrt_llm/_torch/modules/fused_moe/fused_moe_cutlass.py(8 hunks)tensorrt_llm/_torch/modules/fused_moe/quantization.py(5 hunks)tensorrt_llm/layers/moe.py(1 hunks)tests/unittest/_torch/helpers.py(1 hunks)tests/unittest/_torch/modules/test_fused_moe.py(2 hunks)tests/unittest/_torch/thop/test_weight_only_quant_gemm.py(2 hunks)tests/unittest/trt/quantization/test_moe_weight_only_quant_matmul.py(6 hunks)
🧰 Additional context used
📓 Path-based instructions (3)
**/*.py
📄 CodeRabbit Inference Engine (CODING_GUIDELINES.md)
**/*.py: The code developed for TensorRT-LLM should conform to Python 3.8+.
Indent Python code with 4 spaces. Do not use tabs.
Always maintain the namespace when importing in Python, even if only one class or function from a module is used.
Python filenames should use snake_case (e.g., some_file.py).
Python classes should use PascalCase (e.g., class SomeClass).
Python functions and methods should use snake_case (e.g., def my_awesome_function():).
Python local variables should use snake_case. Prefix k for variable names that start with a number (e.g., k_99th_percentile).
Python global variables should use upper snake_case and prefix G (e.g., G_MY_GLOBAL).
Python constants should use upper snake_case (e.g., MY_CONSTANT).
Avoid shadowing variables declared in an outer scope in Python.
Initialize all externally visible members of a Python class in the constructor.
For interfaces that may be used outside a file, prefer docstrings over comments in Python.
Comments in Python should be reserved for code within a function, or interfaces that are local to a file.
Use Google style docstrings for Python classes and functions, which can be parsed by Sphinx.
Attributes and variables in Python can be documented inline, and attribute docstrings will be rendered under the docstring for the class.
Avoid using reflection in Python when functionality can be easily achieved without reflection.
When using try-except blocks in Python, limit the except to the smallest set of errors possible.
When using try-except blocks to handle multiple possible variable types in Python, keep the body of the try as small as possible, using the else block to implement the logic.
Files:
tests/unittest/_torch/helpers.pytensorrt_llm/layers/moe.pytests/unittest/trt/quantization/test_moe_weight_only_quant_matmul.pytests/unittest/_torch/thop/test_weight_only_quant_gemm.pytests/unittest/_torch/modules/test_fused_moe.pytensorrt_llm/_torch/modules/fused_moe/fused_moe_cutlass.pytensorrt_llm/_torch/custom_ops/torch_custom_ops.pytensorrt_llm/_torch/modules/fused_moe/quantization.py
**/*.{cpp,h,hpp,cc,cxx,cu,py}
📄 CodeRabbit Inference Engine (CODING_GUIDELINES.md)
All TensorRT-LLM Open Source Software code should contain an NVIDIA copyright header that includes the current year. This includes .cpp, .h, .cu, .py, and any other source files which are compiled or interpreted.
Files:
tests/unittest/_torch/helpers.pytensorrt_llm/layers/moe.pytests/unittest/trt/quantization/test_moe_weight_only_quant_matmul.pytests/unittest/_torch/thop/test_weight_only_quant_gemm.pytests/unittest/_torch/modules/test_fused_moe.pytensorrt_llm/_torch/modules/fused_moe/fused_moe_cutlass.pytensorrt_llm/_torch/custom_ops/torch_custom_ops.pycpp/tensorrt_llm/thop/moeOp.cpptensorrt_llm/_torch/modules/fused_moe/quantization.py
**/*.{cpp,h,hpp,cc,cxx}
📄 CodeRabbit Inference Engine (CODING_GUIDELINES.md)
**/*.{cpp,h,hpp,cc,cxx}: Closing braces of namespaces should have a comment saying the namespace it closes (e.g., } // namespace foo).
Prefer const or constexpr variables over #defines whenever possible, as the latter are not visible to the compiler.
A variable that is not modified after its initialization should be declared as const.
Except 0 (only used in comparison for checking signness/existence/emptiness) and nullptr, true, false, all other literals should only be used for variable initialization.
Use the Allman indentation style for braces in C++ code.
Put the semicolon for an empty for or while loop in a new line.
The statement forming the body of a switch, while, do .. while or for statement shall be a compound statement (use brace-delimited statements).
If and else should always be followed by brace-delimited statements, even if empty or a single statement.
C++ filenames should use camel case with first letter lowercase (e.g., thisIsAFilename.cpp), and all files involved in the compilation of a target must have filenames that are case-insensitive unique.
All types (including class names) are camel case with uppercase first letter (e.g., FooBarClass).
Local variables, methods, and namespaces use camel case with first letter lowercase (e.g., localFooBar).
Non-magic-number global variables that are non-static and not defined in anonymous namespace use camel case prefixed by a lower case 'g' (e.g., gDontUseGlobalFoos).
Non-magic-number global variables that are static or defined in an anonymous namespace use camel case prefixed by a lower case 's' (e.g., sMutableStaticGlobal).
Locally visible static variable uses camel case with lowercase prefix 's' as the first letter of the name (e.g., static std::once_flag sFlag;).
Class member variables use camelcase prefixed with an 'm' (e.g., mNbFooValues). Public member variables do not require the 'm' prefix but it is encouraged for clarity.
Enumerations, global constants, static constants at class-scope and function-...
Files:
cpp/tensorrt_llm/thop/moeOp.cpp
🧠 Learnings (4)
📓 Common learnings
Learnt from: moraxu
PR: NVIDIA/TensorRT-LLM#6303
File: tests/integration/test_lists/qa/examples_test_list.txt:494-494
Timestamp: 2025-07-28T17:06:08.621Z
Learning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.
📚 Learning: in tensorrt-llm testing, it's common to have both cli flow tests (test_cli_flow.py) and pytorch api ...
Learnt from: moraxu
PR: NVIDIA/TensorRT-LLM#6303
File: tests/integration/test_lists/qa/examples_test_list.txt:494-494
Timestamp: 2025-07-28T17:06:08.621Z
Learning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.
Applied to files:
tests/unittest/trt/quantization/test_moe_weight_only_quant_matmul.pytests/unittest/_torch/modules/test_fused_moe.pytensorrt_llm/_torch/modules/fused_moe/quantization.py
📚 Learning: in tensorrt-llm's multimodal processing pipeline, shared tensor recovery using `from_shared_tensor()...
Learnt from: yechank-nvidia
PR: NVIDIA/TensorRT-LLM#6254
File: tensorrt_llm/_torch/pyexecutor/model_engine.py:1201-1204
Timestamp: 2025-07-22T09:22:14.726Z
Learning: In TensorRT-LLM's multimodal processing pipeline, shared tensor recovery using `from_shared_tensor()` is only needed during the context phase. Generation requests reuse the already-recovered tensor data and only need to call `strip_for_generation()` to remove unnecessary multimodal data while preserving the recovered tensors. This avoids redundant tensor recovery operations during generation.
Applied to files:
tensorrt_llm/_torch/modules/fused_moe/quantization.py
📚 Learning: in tensorrt-llm, examples directory can have different dependency versions than the root requirement...
Learnt from: yibinl-nvidia
PR: NVIDIA/TensorRT-LLM#6506
File: examples/models/core/mixtral/requirements.txt:3-3
Timestamp: 2025-08-01T15:14:45.673Z
Learning: In TensorRT-LLM, examples directory can have different dependency versions than the root requirements.txt file. Version conflicts between root and examples dependencies are acceptable because examples are designed to be standalone and self-contained.
Applied to files:
tensorrt_llm/_torch/modules/fused_moe/quantization.py
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Pre-commit Check
🔇 Additional comments (15)
tensorrt_llm/layers/moe.py (1)
771-772: LGTM! Logical refinement to enable INT8 weight-only quantization.This change correctly narrows the error condition to only prevent INT8 weight quantization when group scaling is used (
group_size > 0), while allowing INT8 weight-only quantization without group scaling to proceed. This aligns perfectly with the PR's objective of adding MoE INT8 weight-only quantization support.tests/unittest/_torch/thop/test_weight_only_quant_gemm.py (2)
18-18: Good refactoring to use centralized tolerance calculation.Importing the helper function promotes code reuse and ensures consistent tolerance calculations across weight-only quantization tests.
Note: When the typo in the helper function name is fixed, this import will need to be updated to
calc_woq_tolerance.
70-71: Excellent refactoring to eliminate code duplication.Replacing the local tolerance calculation with the centralized helper function is a good practice that ensures consistency across tests while reducing code duplication.
Note: When the helper function name is corrected to
calc_woq_tolerance, this line will need to be updated accordingly.tensorrt_llm/_torch/custom_ops/torch_custom_ops.py (2)
43-77: LGTM! Clean integration of weight-only quantization flags.The new
use_woq_per_channelanduse_woq_group_scalingparameters are properly integrated into the MoERunner class, following the existing pattern for quantization flags.
146-147: LGTM! Correct handling of weight tensor shapes.The fake implementation properly accounts for the different weight tensor layout when
use_woq_per_channelis enabled, where the hidden dimension moves to index 2.Also applies to: 184-185, 258-268
tensorrt_llm/_torch/modules/fused_moe/fused_moe_cutlass.py (3)
147-156: LGTM! Well-defined properties for weight-only quantization modes.The properties correctly distinguish between per-channel and per-group weight-only quantization modes.
169-170: LGTM! Correct quantization method selection.The code properly returns
WeightOnlyFusedMoEMethodfor per-channel weight-only quantization.
250-257: Verify the flag setting logic for group scaling.The
use_woq_group_scalingflag is set toTruefor W4A8 quantization (line 252) and potentially set again based onhas_woq_per_group_scaling(line 257). Please verify this is intentional and consider adding a comment to clarify the precedence.tests/unittest/trt/quantization/test_moe_weight_only_quant_matmul.py (2)
72-110: LGTM! Correct handling of per-channel vs group-wise quantization.The code properly determines
use_int8from the quantization mode and correctly assigns eitherper_channel_scaleorweights_scaling_factorbased on the group size.
274-363: LGTM! Comprehensive test coverage for per-channel weight-only quantization.The new
_woq_moe_matmul_per_channelmethod andtest_moe_w8a16test provide thorough coverage for per-channel weight-only quantization, following the established testing patterns.tensorrt_llm/_torch/modules/fused_moe/quantization.py (3)
9-10: Good refactoring: Moving import to module level.Moving
preprocess_weights_for_mixed_gemmto module-level import is appropriate since it's now used by multiple quantization method classes.
64-67: LGTM! Consistent structure for weight-only quantization scales.The new namedtuple follows the established pattern and clearly defines the weight scale tensors needed for weight-only quantization.
869-869: Clarify the SM version override logic.The code sets
sm_version = 80for architectures >= 90. This seems like a workaround that could lead to confusion. Please add a comment explaining why newer architectures are treated as SM 80 for weight-only quantization.cpp/tensorrt_llm/thop/moeOp.cpp (2)
184-196: LGTM! Weight-only quantization initialization is well-structured.The initialization logic for weight-only quantization correctly handles different activation types and sets the appropriate inner dimension multiplier for INT4.
281-300: Correct handling of different tensor layouts.The dimension checks and size calculations properly handle both per-channel and standard weight-only quantization layouts.
Yuening-wa
force-pushed
the
user/yueningl/moe_int8_weight_only_quant_support
branch
from
e768d61 to
90730fc
Compare
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (2)
cpp/tensorrt_llm/thop/moeOp.cpp (2)
525-533: LGTM! Group size consistency issue resolved.The dimension calculation logic correctly mirrors the main execution path and uses the new
WEIGHT_QUANT_GROUP_SIZEconstant, which addresses the previous review comment about inconsistent group size definitions across the codebase.
610-615: LGTM! New member variables and constant improve code organization.The addition of the weight-only quantization flags and the
WEIGHT_QUANT_GROUP_SIZEconstant enhances code clarity and directly addresses the previous review comment about maintaining consistent group size definitions.
🧹 Nitpick comments (1)
cpp/tensorrt_llm/thop/moeOp.cpp (1)
96-123: LGTM with minor optimization opportunity.The template method correctly handles different weight quantization types and addresses the previous duplicated runner instantiation issue. However, there's still a minor duplication that could be optimized.
Consider consolidating the FP8-enabled and FP8-disabled paths for the non-group-scaling case:
#ifdef ENABLE_FP8 if (mUseW4A8GroupScaling) { return std::make_unique< kernels::CutlassMoeFCRunner<__nv_fp8_e4m3, cutlass::uint4b_t, TypeAct, TypeAct>>(); } - else - { - return std::make_unique<kernels::CutlassMoeFCRunner<TypeAct, cutlass::uint4b_t>>(); - } -#else +#endif return std::make_unique<kernels::CutlassMoeFCRunner<TypeAct, cutlass::uint4b_t>>(); -#endif
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
cpp/tensorrt_llm/thop/moeOp.cpp(9 hunks)tensorrt_llm/_torch/modules/fused_moe/quantization.py(5 hunks)tests/unittest/_torch/modules/test_fused_moe.py(2 hunks)
🚧 Files skipped from review as they are similar to previous changes (2)
- tests/unittest/_torch/modules/test_fused_moe.py
- tensorrt_llm/_torch/modules/fused_moe/quantization.py
🧰 Additional context used
🧠 Learnings (1)
📓 Common learnings
Learnt from: moraxu
PR: NVIDIA/TensorRT-LLM#6303
File: tests/integration/test_lists/qa/examples_test_list.txt:494-494
Timestamp: 2025-07-28T17:06:08.621Z
Learning: In TensorRT-LLM testing, it's common to have both CLI flow tests (test_cli_flow.py) and PyTorch API tests (test_llm_api_pytorch.py) for the same model. These serve different purposes: CLI flow tests validate the traditional command-line workflow, while PyTorch API tests validate the newer LLM API backend. Both are legitimate and should coexist.
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Pre-commit Check
🔇 Additional comments (5)
cpp/tensorrt_llm/thop/moeOp.cpp (5)
125-197: LGTM! Constructor properly handles new quantization modes.The constructor correctly:
- Accepts the new weight-only quantization parameters
- Initializes the corresponding member variables
- Sets the appropriate inner dimension multiplier for Int4 quantization
- Uses the new template method for weight quantization runner creation
- Provides proper error handling for unsupported activation types
282-301: LGTM! Dimension calculations correctly handle per-channel quantization.The conditional logic properly adjusts dimension checks and calculations based on the per-channel weight-only quantization mode:
- Switches tensor dimension indices appropriately for different quantization layouts
- Maintains consistency between fc1 and fc2 expert weight dimensions
- Correctly recalculates hidden_size and inter_size for per-channel mode
902-939: LGTM! Comprehensive quantization parameter handling.The method correctly handles different weight-only quantization modes:
- Per-channel quantization with appropriate 2-scale validation
- Int4 group scaling with comprehensive 8-parameter setup
- Proper use of the consistent
WEIGHT_QUANT_GROUP_SIZEconstant- Good error handling for unsupported configurations
The parameter validation and construction logic is thorough and well-structured.
958-976: LGTM! Helper methods improve code clarity.The new helper methods provide clean abstractions:
isWInt8Quant()clearly identifies int8 weight quantizationisWeightOnlyQuant()provides a unified check for weight-only quantization modesThese methods enhance readability and maintainability by encapsulating quantization type logic.
996-996: LGTM! Torch binding correctly updated for new parameters.The constructor binding properly exposes the new weight-only quantization parameters (
use_woq_per_channelanduse_woq_group_scaling) to the Python interface, maintaining API consistency.
|
/bot run |
|
PR_Github #14268 [ run ] triggered by Bot |
|
PR_Github #14268 [ run ] completed with state |
Yuening-wa
force-pushed
the
user/yueningl/moe_int8_weight_only_quant_support
branch
from
269658b to
455cb26
Compare
|
/bot run |
|
PR_Github #14680 [ run ] triggered by Bot |
Yuening-wa
force-pushed
the
user/yueningl/moe_int8_weight_only_quant_support
branch
from
bd6f1ac to
f1aa234
Compare
|
/bot run |
|
PR_Github #15223 [ run ] triggered by Bot |
|
Do u have e2e test on some models? |
I am working on the e2e test and considering to open another PR for it. |
Yuening-wa
force-pushed
the
user/yueningl/moe_int8_weight_only_quant_support
branch
from
f1aa234 to
45bc810
Compare
|
/bot run |
|
PR_Github #15368 [ run ] triggered by Bot |
|
PR_Github #15368 [ run ] completed with state |
Signed-off-by: Yuening Li <[email protected]>
Signed-off-by: Yuening Li <[email protected]>
Head branch was pushed to by a user without write access
Yuening-wa
force-pushed
the
user/yueningl/moe_int8_weight_only_quant_support
branch
from
45bc810 to
e01f35b
Compare
|
/bot run |
|
PR_Github #15396 [ run ] triggered by Bot |
|
PR_Github #15396 [ run ] completed with state |
…rch Workflow (NVIDIA#6629) Signed-off-by: Yuening Li <[email protected]>
…rch Workflow (NVIDIA#6629) Signed-off-by: Yuening Li <[email protected]> Signed-off-by: Wangshanshan <[email protected]>
…rch Workflow (NVIDIA#6629) Signed-off-by: Yuening Li <[email protected]> Signed-off-by: Wangshanshan <[email protected]>
…rch Workflow (NVIDIA#6629) Signed-off-by: Yuening Li <[email protected]>
…rch Workflow (NVIDIA#6629) Signed-off-by: Yuening Li <[email protected]> Signed-off-by: Wangshanshan <[email protected]>
…rch Workflow (NVIDIA#6629) Signed-off-by: Yuening Li <[email protected]>
…rch Workflow (NVIDIA#6629) Signed-off-by: Yuening Li <[email protected]> Signed-off-by: Wangshanshan <[email protected]>
Summary by CodeRabbit
New Features
Bug Fixes
Tests
Refactor
Description
Test Coverage
GitHub Bot Help
/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...Provide a user friendly way for developers to interact with a Jenkins server.
Run
/bot [-h|--help]to print this help message.See details below for each supported subcommand.
run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental)]Launch build/test pipelines. All previously running jobs will be killed.
--reuse-test (optional)pipeline-id(OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.--disable-reuse-test(OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.--disable-fail-fast(OPTIONAL) : Disable fail fast on build/tests/infra failures.--skip-test(OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.--stage-list "A10-PyTorch-1, xxx"(OPTIONAL) : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: Does NOT update GitHub check status.--gpu-type "A30, H100_PCIe"(OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.--test-backend "pytorch, cpp"(OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.--only-multi-gpu-test(OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.--disable-multi-gpu-test(OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.--add-multi-gpu-test(OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.--post-merge(OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx"(OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".--detailed-log(OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.--debug(OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in thestage-listparameter to access the appropriate container environment. Note: Does NOT update GitHub check status.For guidance on mapping tests to stage names, see
docs/source/reference/ci-overview.mdand the
scripts/test_to_stage_mapping.pyhelper.kill
killKill all running builds associated with pull request.
skip
skip --comment COMMENTSkip testing for latest commit on pull request.
--comment "Reason for skipping build/test"is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.reuse-pipeline
reuse-pipelineReuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.