v1.0.0rc5

[QiJune]

Announcement Highlights:

• Model Support
• Feature
  • Deepseek R1 FP8 Support on Blackwell (Deepseek R1 FP8 Support on Blackwell #6486)
  • Auto-enable ngram with concurrency <= 32. ([feat] Auto-enable ngram with concurrency <= 32. #6232)
  • Support turning on/off spec decoding dynamically ([TRTLLM-6392][feat] Support turning on/off spec decoding dynamically #6363)
  • Improve LoRA cache memory control ([TRTLLM-5830][feat] Improve LoRA cache memory control #6220)
  • Support structural tag in C++ runtime and upgrade xgrammar to 0.1.21 (feat: Support structural tag in C++ runtime and upgrade xgrammar to 0.1.21 #6408)
  • Update long rope for phi3.5/phi4-mini/phi4-mm (feat: TRTLLM-6450 update long rope for phi3.5/phi4-mini/phi4-mm  #6353)
  • Add support for external multimodal embeddings ([TRTLLM-6654][feat] Add support for external multimodal embeddings #6263)
  • Add support for disaggregation with pp with pytorch backend (feat: Add support for disaggregation with pp with pytorch backend #6369)
  • Add _prepare_and_schedule_batch function in PyExecutor (chore: add _prepare_and_schedule_batch function in PyExecutor #6365)
  • Add status tags to LLM API reference ([TRTLLM-5061] chore: add status tags to LLM API reference #5707)
  • Remove cudaStreamSynchronize when using relaxed acceptance (fix: remove cudaStreamSynchronize when using relaxed acceptance #5262)
  • Support JSON Schema in OpenAI-Compatible API (feat: Support JSON Schema in OpenAI-Compatible API #6321)
  • Support chunked prefill on spec decode 2 model ([TRTLLM-6453][feat] Support chunked prefill on spec decode 2 model #6104)
  • Enhance beam search support with CUDA graph integration ([TRTLLM-6650][feat] Enhance beam search support with CUDA graph integration #6217)
  • Enable Overlap scheduler + Beam Search in TRTLLM Sampler ([TRTLLM-6651][feat] Enable Overlap scheduler + Beam Search in TRTLLM Sampler #6223)
  • Add KV cache reuse support for multimodal models ([TRTLLM-5059][feat] Add KV cache reuse support for multimodal models #5444)
  • Multistream initial support for torch compile flow ([TRTLLM-4279] feat: Multistream initial support for torch compile flow #5847)
  • Support nanobind bindings (feat: nanobind bindings #6185)
  • Support Weight-Only-Quantization in PyTorch Workflow ([TRTLLM-5863][feat] Support Weight-Only-Quantization in PyTorch Workflow #5850)
  • Support pytorch LoRA adapter eviction ([TRTLLM-5826][feat] Support pytorch LoRA adapter eviction #5616)
• API
  • [BREAKING CHANGE] Change default backend to PyTorch in trtllm-serve ([BREAKING CHANGE]: change default backend to PyTorch in trtllm-serve #5717)
• Bug Fixes
  • fix: remove duplicate layer multiplication in KV cache size calculation (fix: remove duplicate layer multiplication in KV cache size calculation #6481)
  • fix illeagel memory access in MLA (fix: fix illeagel memory access #6437)
  • Fix nemotronNAS loading for TP>1 ([nvbug 5380101][fix] Fix nemotronNAS loading for TP>1 #6447)
  • Switch placement of image placeholder for mistral 3.1 ([fix] Switch placement of image placeholder for mistral 3.1 #6435)
  • Fix wide EP when using DeepEP with online EPLB ([fix] Fix wide EP when using DeepEP with online EPLB #6429)
  • Move kv_cache_free_gpu_mem_fraction arg to benchmark command in tests ([fix] Move kv_cache_free_gpu_mem_fraction arg to benchmark command in tests #6463)
  • Fix bugs caused by None attention_bias during Qwen3 model convert engine ([FIX] fix bugs caused by None attention_bias during Qwen3 model convert engine #6344)
  • Add a runtime flag to enable fail fast when attn window is too large to fit at least one sequence in KV cache ([nvbug/5374773] chore: Add a runtime flag to enable fail fast when attn window is too large to fit at least one sequence in KV cache #5974)
  • Fix PD + MTP + overlap scheduler accuracy issue ([nvbug/5322354] fix PD + MTP + overlap scheduler accuracy issue #6136)
  • Fix bug of Qwen3 when using fp4 on sm120 ([Fix][Chore][Qwen3] fix bug of using fp4 on sm120 #6065)
• Benchmark
  • Fixes to parameter usage and low latency configuration. ([fix] Fixes to parameter usage and low latency configuration. #6343)
  • Add Acceptance Rate calculation to benchmark_serving (Add Acceptance Rate calculation to benchmark_serving #6240)
• Performance
  • Enable AllReduce-associated fusion patterns in Llama3/4. ([TRTLLM-6445] feat: Enable AllReduce-associated fusion patterns in Llama3/4. #6205)
  • Optimize Mtp performance (Mtp optimizations round1 #5689)
  • Customize cublastLt algo for Llamba 3.3 70B TP4 (perf: customize cublastLt algo for Llamba 3.3 70B TP4 #6315)
  • Add non UB AR + Residual + Norm + Quant fusion (feat: Add non UB AR + Residual + Norm + Quant fusion #6320)
• Infrastructure
  • Remove auto_assign_reviewers option from .coderabbit.yaml ([infra] Remove auto_assign_reviewers option from .coderabbit.yaml #6490)
  • Use build stage wheels to speed up docker release image build (infra: [TRTLLM-5873] Use build stage wheels to speed up docker release image build #4939)
• Documentation
  • Add README for wide EP (doc: Add README for wide EP  #6356)
  • Update Llama4 deployment guide: update config & note concurrency ([nvbug/5361223] doc: Update Llama4 deployment guide: update config & note concurrency #6222)
  • Add Deprecation Policy section (doc: add Deprecation Policy section #5784)
• Known Issues
  • If you encounter the OSError: CUDA_HOME environment variable is not set error, set the CUDA_HOME environment variable
  • The aarch64 Docker image and wheel package for 1.0.0rc5 are broken. This will be fixed in the upcoming weekly release

What's Changed

• DeepEP LL support variable hidden size and tokens num by @yilin-void in DeepEP LL support variable hidden size and tokens num #6141
• [Fix][Chore][Qwen3] fix bug of using fp4 on sm120 by @byshiue in [Fix][Chore][Qwen3] fix bug of using fp4 on sm120 #6065
• fix: Ensure mlx5 library is installed for deep_ep and remove deprecated python bindings by @MartinMarciniszyn in fix: Ensure mlx5 library is installed for deep_ep and remove deprecated python bindings #6189
• [TRTLLM-5826][feat] Support pytorch LoRA adapter eviction by @amitz-nv in [TRTLLM-5826][feat] Support pytorch LoRA adapter eviction #5616
• W4A8 GEMM by @danielafrimi in W4A8 GEMM  #6005
• enh: Lift expectation of single image per sample in Gemma3 VLM by @brb-nv in enh: Lift expectation of single image per sample in Gemma3 VLM #6195
• test: add phi-4 multimodel and bielik-11b-v2.2 models for perf test by @ruodil in test: add phi-4 multimodel and bielik-11b-v2.2 models for perf test #5826
• fix: Flush stale PlanParams with custom attention mask by @brb-nv in fix: Flush stale PlanParams with custom attention mask #6163
• doc: remove cuda_graph_config: {} from doc since cuda_graph enabled b… by @nv-guomingz in doc: remove cuda_graph_config: {} from doc since cuda_graph enabled b… #6150
• [fix] Fix can_use_alltoall in fused_moe_wide_ep.py by @jinyangyuan-nvidia in [fix] Fix can_use_alltoall in fused_moe_wide_ep.py #6173
• [TRTLLM-5863][feat] Support Weight-Only-Quantization in PyTorch Workflow by @Yuening-wa in [TRTLLM-5863][feat] Support Weight-Only-Quantization in PyTorch Workflow #5850
• test: [CI] remove closed bugs by @xinhe-nv in test: [CI] remove closed bugs #6201
• feat: nanobind bindings by @Linda-Stadter in feat: nanobind bindings #6185
• infra: [TRTLLM-5250] Add sanity check stage for ngc-release images (Build wheels for devel image) by @ZhanruiSunCh in infra: [TRTLLM-5250] Add sanity check stage for ngc-release images (Build wheels for devel image) #4656
• doc: add Deprecation Policy section by @QiJune in doc: add Deprecation Policy section #5784
• [TRTLLM-4279] feat: Multistream initial support for torch compile flow by @liji-nv in [TRTLLM-4279] feat: Multistream initial support for torch compile flow #5847
• [Infra] - Waive failed cases on recent post-merge by @EmmaQiaoCh in [Infra] - Waive failed cases on recent post-merge #6212
• [BREAKING CHANGE]: change default backend to PyTorch in trtllm-serve by @LinPoly in [BREAKING CHANGE]: change default backend to PyTorch in trtllm-serve #5717
• test: Enable GB200 torch compile multi gpu tests by @yizhang-nv in test: Enable GB200 torch compile multi gpu tests #6145
• [fix] Correct the returned value of has_spec_drafter by @ziyixiong-nv in [fix] Correct the returned value of has_spec_drafter #6178
• [chore] Clean up quickstart_advanced.py by @mikeiovine in [chore] Clean up quickstart_advanced.py #6021
• [Chore] Replace MODEL_CACHE_DIR with LLM_MODELS_ROOT and unwaive triton_server/test_triton.py::test_gpt_ib[gpt-ib] by @SimengLiu-nv in [Chore] Replace MODEL_CACHE_DIR with LLM_MODELS_ROOT and unwaive triton_server/test_triton.py::test_gpt_ib[gpt-ib] #5859
• [TRTLLM-5059][feat] Add KV cache reuse support for multimodal models by @chang-l in [TRTLLM-5059][feat] Add KV cache reuse support for multimodal models #5444
• feat: Refactor the fetching request logic by @Shunkangz in feat: Refactor the fetching request logic #5786
• tests: add timeout_manager to tensorrt flow test cases by @crazydemo in tests: add timeout_manager to tensorrt flow test cases #5942
• feat: moe prepare support topk % 4 != 0 by @WeiHaocheng in feat: moe prepare support topk % 4 != 0 #5742
• [fix] Fix flaky mistral E2E test by @2ez4bz in [fix] Fix flaky mistral E2E test #6230
• bug: [https://nvbugs/5368507] Fix test_generate_with_seed. by @bobboli in bug: [https://nvbugs/5368507] Fix test_generate_with_seed. #6206
• chore: Mass integration of release/0.21 (part 4) by @dc3671 in chore: Mass integration of release/0.21 (part 4) #6211
• doc: add supported data modality and types on multimodal serve by @yechank-nvidia in doc: add supported data modality and types on multimodal serve #5988
• chore: bump version to 1.0.0rc5 by @yiqingy0 in chore: bump version to 1.0.0rc5 #6252
• [TRTLLM-6537][infra] extend multi-gpu tests related file list by @reasonsolo in [TRTLLM-6537][infra] extend multi-gpu tests related file list #6139
• test: update test list for RTX6KD by @StanleySun639 in test: update test list for RTX6KD #6213
• fix: bindings unit tests for nanobind by @Linda-Stadter in fix: bindings unit tests for nanobind #6221
• Add register_fake for finegrained_mixed_dtype_gemm torch_op by @danielafrimi in Add register_fake for finegrained_mixed_dtype_gemm torch_op #6255
• [Issue 6193] Fix gemma3vl weight loader by @johncalesp in [Issue 6193] Fix gemma3vl weight loader #6233
• [feat] Enable TP and batching for PixtralVisionModel / Mistral3VLM by @2ez4bz in [feat] Enable TP and batching for PixtralVisionModel / Mistral3VLM #6152
• set NVIDIA_IMEX_CHANNELS for dlcluster slurm job only by @yuanjingx87 in set NVIDIA_IMEX_CHANNELS for dlcluster slurm job only #6234
• [nvbug/5361223] doc: Update Llama4 deployment guide: update config & note concurrency by @raayandhar in [nvbug/5361223] doc: Update Llama4 deployment guide: update config & note concurrency #6222
• [AutoDeploy] merge feat/ad-2025-07-07 by @lucaslie in [AutoDeploy] merge feat/ad-2025-07-07 #6196
• [nvbugs/5401261][fix] Fix Triton backend disaggregated serving support by @Tabrizian in [nvbugs/5401261][fix] Fix Triton backend disaggregated serving support #6224
• [refactor] Simplification of Speculative decoding configs - Part 2 by @wili-65535 in [refactor] Simplification of Speculative decoding configs - Part 2 #5936
• doc: Refactor documents and examples of disaggregated serving and wide ep by @kaiyux in doc: Refactor documents and examples of disaggregated serving and wide ep #6054
• Add basic Nemo Ckpt Lora Loading in pytorch flow by @venkywonka in Add basic Nemo Ckpt Lora Loading in pytorch flow  #6019
• [https://nvbugs/5387771] fix deadlocks due to insufficient numSemaphores by @PerkzZheng in [https://nvbugs/5387771] fix deadlocks due to insufficient numSemaphores #6262
• fix: nvbug_5398806 by @hchings in fix: nvbug_5398806 #6239
• chore: set default device to cpu on Multimodal models by @yechank-nvidia in chore: set default device to cpu on Multimodal models #5994
• chore: remove duplicate should_stop_processing check by @QiJune in chore: remove duplicate should_stop_processing check #6242
• hopper-style context MLA by @zhou-yuxin in hopper-style context MLA #5713
• [nvbug/5322354] fix PD + MTP + overlap scheduler accuracy issue by @yweng0828 in [nvbug/5322354] fix PD + MTP + overlap scheduler accuracy issue #6136
• test: [CI] Add failed cases into waives.txt by @xinhe-nv in test: [CI] Add failed cases into waives.txt #6289
• [TRTLLM-6651][feat] Enable Overlap scheduler + Beam Search in TRTLLM Sampler by @stnie in [TRTLLM-6651][feat] Enable Overlap scheduler + Beam Search in TRTLLM Sampler #6223
• [Infra] - Skip failed cases by @EmmaQiaoCh in [Infra] - Skip failed cases #6299
• [AutoDeploy] disable flaky MoE nvfp4 test by @lucaslie in [AutoDeploy] disable flaky MoE nvfp4 test #6302
• [feat] Update .coderabbit.yaml with review settings and code guidelines by @venkywonka in [feat] Update .coderabbit.yaml with review settings and code guidelines #6251
• Waive tests by @Tabrizian in Waive tests #6312
• [Infra] - Increase unittest execution time since some test exceeds 1600 by @EmmaQiaoCh in [Infra] - Increase unittest execution time since some test exceeds 1600 #6277
• Revert "tests: add timeout_manager to tensorrt flow test cases (tests: add timeout_manager to tensorrt flow test cases #5942)" by @Tabrizian in Revert "tests: add timeout_manager to tensorrt flow test cases (#5942)" #6309
• doc: fix invalid links related with llm api example by @nv-guomingz in doc: fix invalid links related with llm api example #6317
• chore: remove unused variables in pyexecutor by @QiJune in chore: remove unused variables in pyexecutor #6280
• [TRTLLM-6444] Add some UCX trouble shooting docs and print UCX related logs by @reasonsolo in [TRTLLM-6444] Add some UCX trouble shooting docs and print UCX related logs #6085
• feat: Add non UB AR + Residual + Norm + Quant fusion by @liji-nv in feat: Add non UB AR + Residual + Norm + Quant fusion #6320
• Update fmhaRunner.cpp to fix guardwords scan error by @zhou-yuxin in Update fmhaRunner.cpp to fix guardwords scan error #6327
• tests: only get timeout value from pytest marker by @crazydemo in tests: only get timeout value from pytest marker #6287
• [Infra] - Wiave failed tests in post-merge by @EmmaQiaoCh in [Infra] - Wiave failed tests in post-merge #6331
• [Fix][nvbug 5401163][nvbug 5404726][Qwen3] Fix bug of MoE on tp > 1 with trtllm moe backend by @byshiue in [Fix][nvbug 5401163][nvbug 5404726][Qwen3] Fix bug of MoE on tp > 1 with trtllm moe backend #6235
• perf: customize cublastLt algo for Llamba 3.3 70B TP4 by @zhenhuaw-me in perf: customize cublastLt algo for Llamba 3.3 70B TP4 #6315
• [Fix] the bug in the trtllm-gen heurisitcf for MLA kernels. by @PerkzZheng in [Fix] the bug in the trtllm-gen heurisitcf for MLA kernels. #6284
• Improve TransferAgentTest.SyncMessage by @bo-nv in Improve TransferAgentTest.SyncMessage #6250
• [TRTLLM-6650][feat] Enhance beam search support with CUDA graph integration by @stnie in [TRTLLM-6650][feat] Enhance beam search support with CUDA graph integration #6217
• [fix] Update to remove popping of KV cache and other args. by @FrankD412 in [fix] Update to remove popping of KV cache and other args. #6310
• [fix][nvbugs/5399355] Fix Lamport buffer clear issue for MNNVL TwoShot Allreduce and add FP16 support. by @timlee0212 in [fix][nvbugs/5399355] Fix Lamport buffer clear issue for MNNVL TwoShot Allreduce and add FP16 support. #6237
• fix: integration tests with nanobind by @Linda-Stadter in fix: integration tests with nanobind #6326
• [TRTLLM-6453][feat] Support chunked prefill on spec decode 2 model by @mikeiovine in [TRTLLM-6453][feat] Support chunked prefill on spec decode 2 model #6104
• test: skip llama3.3 70b test on cg4 by @xinhe-nv in test: skip llama3.3 70b test on cg4 #6293
• [TRTLLM-5312] - Add bot run rules for triton tests by @yiqingy0 in [TRTLLM-5312] - Add bot run rules for triton tests #4988
• tests: add test_chunked_prefill for llama4 by @xinhe-nv in tests: add test_chunked_prefill for llama4 #5549
• [https://nvbugs/5340941] - fix: Correct custom ops used by Qwen3 Moe … by @liji-nv in [https://nvbugs/5340941] - fix: Correct custom ops used by Qwen3 Moe … #6285
• test: [CI] Add failed cases into waives.txt by @xinhe-nv in test: [CI] Add failed cases into waives.txt #6333
• [feat]: support logit_bias by @xq25478 in [feat]: support logit_bias #5354
• fix: Fixing kv_cache_events unit tests [nvbug5362412] by @pcastonguay in fix: Fixing kv_cache_events unit tests [nvbug5362412] #6265
• feat: Support JSON Schema in OpenAI-Compatible API by @noiji in feat: Support JSON Schema in OpenAI-Compatible API #6321
• [doc] Add NGram tech blog by @SimengLiu-nv in [doc] Add NGram tech blog #6311
• Mtp optimizations round1 by @ameynaik-hub in Mtp optimizations round1 #5689
• [fix][nvbugs/5390810] Improve the check for disaggregated serving test by @Tabrizian in [fix][nvbugs/5390810] Improve the check for disaggregated serving test #6301
• [nvbug/5374773] chore: Add a runtime flag to enable fail fast when attn window is too large to fit at least one sequence in KV cache by @moraxu in [nvbug/5374773] chore: Add a runtime flag to enable fail fast when attn window is too large to fit at least one sequence in KV cache #5974
• fix precompiled multi_query_token kernel not having is_fp8_out hash key by @jhaotingc in fix precompiled multi_query_token kernel not having is_fp8_out hash key #6279
• [fix] README link directs to intended doc by @lianakoleva in [fix] README link directs to intended doc #6340
• [https://nvbugs/5402719][fix]: Add cuda graph dummy requests to the spec_resource_manager by @ziyixiong-nv in [https://nvbugs/5402719][fix]: Add cuda graph dummy requests to the spec_resource_manager #6258
• [nvbug/5320234] fix: test_trtllm_bench_llmapi_launch by @Superjomn in [nvbug/5320234] fix: test_trtllm_bench_llmapi_launch #6359
• fix: remove cudaStreamSynchronize when using relaxed acceptance by @yweng0828 in fix: remove cudaStreamSynchronize when using relaxed acceptance #5262
• [TRTLLM-6445] feat: Enable AllReduce-associated fusion patterns in Llama3/4. by @hyukn in [TRTLLM-6445] feat: Enable AllReduce-associated fusion patterns in Llama3/4. #6205
• DeepEP LL dispatch FP4 by @yilin-void in DeepEP LL dispatch FP4 #6296
• [nvbugs/5401156][fix] Avoid import all models when import trtllm._common by @chang-l in [nvbugs/5401156][fix] Avoid import all models when import trtllm._common #6266
• [fix] Fix perf regression caused by MoE autotuner when using DeepEPLowLatency by @jinyangyuan-nvidia in [fix] Fix perf regression caused by MoE autotuner when using DeepEPLowLatency #6288
• Add Acceptance Rate calculation to benchmark_serving by @zerollzeng in Add Acceptance Rate calculation to benchmark_serving #6240
• [Infa] - waive failed cases and fix a typo by @EmmaQiaoCh in [Infa] - waive failed cases and fix a typo #6384
• [nvbug/5409414, 5355707] tests: adjust batchsize and decoding name by @crazydemo in [nvbug/5409414, 5355707] tests: adjust batchsize and decoding name #6292
• [TRTLLM-5061] chore: add status tags to LLM API reference by @Superjomn in [TRTLLM-5061] chore: add status tags to LLM API reference #5707
• fix: compatibility with CUDA < 12.9 on __CUDA_ARCH_SPECIFIC__ macro by @tongyuantongyu in fix: compatibility with CUDA < 12.9 on __CUDA_ARCH_SPECIFIC__ macro #5917
• chore: add _prepare_and_schedule_batch function in PyExecutor by @QiJune in chore: add _prepare_and_schedule_batch function in PyExecutor #6365
• test: waive failed cases by @xinhe-nv in test: waive failed cases #6394
• test: organize perf cases and add missing perflab cases in qa test list by @ruodil in test: organize perf cases and add missing perflab cases in qa test list #6283
• chore: delete useless gitkeep files. by @nv-guomingz in chore: delete useless gitkeep files. #6400
• [test] Add accuracy regression test for Mistral3.1 by @2ez4bz in [test] Add accuracy regression test for Mistral3.1 #6322
• [test] Unwaive mistral3.1 small E2E test by @2ez4bz in [test] Unwaive mistral3.1 small E2E test #6352
• [None][infra]Update slurm config keys by @yuanjingx87 in [None][infra]Update slurm config keys #6370
• [infra] Add an auto-labeling github action to TRTLLM by @poweiw in [infra] Add an auto-labeling github action to TRTLLM #6373
• [nvbugs/5404000] fix: waive request_perf_metrics_draft test on pre-Hopper GPUs by @achartier in [nvbugs/5404000] fix: waive request_perf_metrics_draft test on pre-Hopper GPUs #6339
• feat: Add Phi-4-Mini-Instruct in Pytorch backend for LLM API accuracy tests by @moraxu in feat: Add Phi-4-Mini-Instruct in Pytorch backend for LLM API accuracy tests #6303
• [infra] Remove auto_apply_labels option from .coderabbit.yaml reviews section by @venkywonka in [infra] Remove auto_apply_labels option from .coderabbit.yaml reviews section #6416
• [fix] Add trust_remote_code option to prepare_dataset. by @FrankD412 in [fix] Add trust_remote_code option to prepare_dataset. #6338
• infra: [TRTLLM-6499] Split L0_Test into two pipeline by single GPU and multi GPU(For SBSA) by @ZhanruiSunCh in infra: [TRTLLM-6499] Split L0_Test into two pipeline by single GPU and multi GPU(For SBSA) #6132
• doc: Add README for wide EP by @kaiyux in doc: Add README for wide EP  #6356
• [fix] Fixes to parameter usage and low latency configuration. by @FrankD412 in [fix] Fixes to parameter usage and low latency configuration. #6343
• test:[nvbug 5415268] add kv_cache_free_gpu_mem_fraction param and llama4 rcca cases by @ruodil in test:[nvbug 5415268] add kv_cache_free_gpu_mem_fraction param and llama4 rcca cases #6430
• chore: remove unused code in PyExecutor by @QiJune in chore: remove unused code in PyExecutor #6351
• [5385981] fix: Update the usage of VisionAttention init API. by @hyukn in [5385981] fix: Update the usage of VisionAttention init API. #6413
• test: [CI] Add failed cases into waives.txt by @xinhe-nv in test: [CI] Add failed cases into waives.txt #6423
• test: [CI] remove closed bugs by @xinhe-nv in test: [CI] remove closed bugs #6381
• doc: remove backend parameter for trtllm-bench when backend is set to… by @nv-guomingz in doc: remove backend parameter for trtllm-bench when backend is set to… #6428
• infra: [TRTLLM-5873] Use build stage wheels to speed up docker release image build by @ZhanruiSunCh in infra: [TRTLLM-5873] Use build stage wheels to speed up docker release image build #4939
• [fix] Add detokenization-based stop word logic to LLM API by @moraxu in [fix] Add detokenization-based stop word logic to LLM API #5948
• chore: remove unused kv_cache_dtype in api reference by @Superjomn in chore: remove unused kv_cache_dtype in api reference #6444
• chore: disallow arbitrary arguments in llm_args.xxxConfigs by @Superjomn in chore: disallow arbitrary arguments in llm_args.xxxConfigs #6367
• [FIX] fix bugs caused by None attention_bias during Qwen3 model convert engine by @fyf2016 in [FIX] fix bugs caused by None attention_bias during Qwen3 model convert engine #6344
• fix: support mixture of text & multimodal prompts by @yechank-nvidia in fix: support mixture of text & multimodal prompts #6345
• [fix] Move kv_cache_free_gpu_mem_fraction arg to benchmark command in tests by @venkywonka in [fix] Move kv_cache_free_gpu_mem_fraction arg to benchmark command in tests #6463
• Rename layer to comply with deepseek by @peaceh-nv in Rename layer to comply with deepseek #6393
• test: [CI] Add failed cases into waives.txt by @xinhe-nv in test: [CI] Add failed cases into waives.txt #6457
• [TRTLLM-6549] chore: record delay introduced by disaggregated serving in kv cache measure by @zhengd-nv in [TRTLLM-6549] chore: record delay introduced by disaggregated serving in kv cache measure #6135
• [fix] Fix wide EP when using DeepEP with online EPLB by @jinyangyuan-nvidia in [fix] Fix wide EP when using DeepEP with online EPLB #6429
• [fix] Switch placement of image placeholder for mistral 3.1 by @2ez4bz in [fix] Switch placement of image placeholder for mistral 3.1 #6435
• chore: clean code of PyExecutor by @QiJune in chore: clean code of PyExecutor #6445
• chore: remove draft_model_engine from init parameter list of PyExecutor by @QiJune in chore: remove draft_model_engine from init parameter list of PyExecutor #6325
• chore: add trtllm-serve json schema example into doc. by @nv-guomingz in chore: add trtllm-serve json schema example into doc. #6418
• tests: add TestNemotronH cuda graph tests by @xinhe-nv in tests: add TestNemotronH cuda graph tests #6390
• [nvbugs/5414909] fix: Qwen2-VL keyword on L20 by @yechank-nvidia in [nvbugs/5414909] fix: Qwen2-VL keyword on L20 #6427
• [TRTLLM-5633] - Merge current waive list with the TOT waive list by @yiqingy0 in [TRTLLM-5633] - Merge current waive list with the TOT waive list #5198
• [doc] update the doc of feature combination matrix by @leslie-fang25 in [doc] update the doc of feature combination matrix #6441
• [nvbug 5380101][fix] Fix nemotronNAS loading for TP>1 by @tomeras91 in [nvbug 5380101][fix] Fix nemotronNAS loading for TP>1 #6447
• feat: Add support for disaggregation with pp with pytorch backend by @pcastonguay in feat: Add support for disaggregation with pp with pytorch backend #6369
• [TRTLLM-6654][feat] Add support for external multimodal embeddings by @chang-l in [TRTLLM-6654][feat] Add support for external multimodal embeddings #6263
• chore: update trtllm-serve usage doc by removing backend parameter when it use torch as backend. by @nv-guomingz in chore: update trtllm-serve usage doc by removing backend parameter when it use torch as backend. #6419
• fix: Unwaive triton cpp test [nvbug 5401088] by @pcastonguay in fix: Unwaive triton cpp test [nvbug 5401088] #6412
• [Perf]: Add residual, norm for nemotron_nas models by @NVShreyas in [Perf]: Add residual, norm for nemotron_nas models #6455
• feat: TRTLLM-6450 update long rope for phi3.5/phi4-mini/phi4-mm by @Wanli-Jiang in feat: TRTLLM-6450 update long rope for phi3.5/phi4-mini/phi4-mm  #6353
• [nvbug/5410296][fix] Fix OOM in Llama 4 disagg-serve tests by @bo-nv in [nvbug/5410296][fix] Fix OOM in Llama 4 disagg-serve tests #6439
• Unwaive Gemma2 LoRA test on H100 by @brb-nv in Unwaive Gemma2 LoRA test on H100 #6461
• [nvbug/5409417] Unwaive llava test case by @amukkara in [nvbug/5409417] Unwaive llava test case #6460
• add propagation of trust_remote_code to OpenAIServer by @shaharmor98 in add propagation of trust_remote_code to OpenAIServer #6446
• test: Add time logging for lora tests by @brb-nv in test: Add time logging for lora tests #6466
• [PERF] Move calculation Qwen2-VL's rotary_cos_sin to LLM worker process by @vadiklyutiy in [PERF] Move calculation Qwen2-VL's rotary_cos_sin to LLM worker process #6004
• doc: update multimodal models on support-matrix.md by @yechank-nvidia in doc: update multimodal models on support-matrix.md #6431
• [doc][ci][Qwen3][nvbugs 5374145] Add Qwen3 235B eagle3 CI by @byshiue in [doc][ci][Qwen3][nvbugs 5374145] Add Qwen3 235B eagle3 CI #6477
• feat: Support structural tag in C++ runtime and upgrade xgrammar to 0.1.21 by @syuoni in feat: Support structural tag in C++ runtime and upgrade xgrammar to 0.1.21 #6408
• fix: fix illeagel memory access by @dongjiyingdjy in fix: fix illeagel memory access #6437
• test: add accuracy reference by @xinhe-nv in test: add accuracy reference #6479
• Fix e2e test failure for RTX6000 Pro by @farazkh80 in Fix e2e test failure for RTX6000 Pro  #6420
• doc: add bielik on support-matrix.md by @Wanli-Jiang in doc: add bielik on support-matrix.md #6480
• [infra] Remove auto_assign_reviewers option from .coderabbit.yaml by @venkywonka in [infra] Remove auto_assign_reviewers option from .coderabbit.yaml #6490
• [TRTLLM-5830][feat] Improve LoRA cache memory control by @amitz-nv in [TRTLLM-5830][feat] Improve LoRA cache memory control #6220
• [Infra][TRTLLM-5633] - Fix merge waive list by @yiqingy0 in [Infra][TRTLLM-5633] - Fix merge waive list #6504
• [None][infra] Pin the version for triton to 3.3.1 by @EmmaQiaoCh in [None][infra] Pin the version for triton to 3.3.1 #6508
• Bugfix/fix nemotron nas lora support by @shaharmor98 in Bugfix/fix nemotron nas lora support #6380
• [https://nvbugs/5404046][fix] Fix Nemotron-H flaky CUDA graph / overlap scheduler test by @tomeras91 in [https://nvbugs/5404046][fix] Fix Nemotron-H flaky CUDA graph / overlap scheduler test #6485
• [nvbug/5374773] chore: Update nanobind with fail_fast_on_attention_window_too_large changes by @moraxu in [nvbug/5374773] chore: Update nanobind with fail_fast_on_attention_window_too_large changes #6491
• [TRTLLM-6392][feat] Support turning on/off spec decoding dynamically by @ziyixiong-nv in [TRTLLM-6392][feat] Support turning on/off spec decoding dynamically #6363
• [feat] Auto-enable ngram with concurrency <= 32. by @SimengLiu-nv in [feat] Auto-enable ngram with concurrency <= 32. #6232
• fix: Fix poor generation with FP8 Gemma3 1B checkpoint by @brb-nv in fix: Fix poor generation with FP8 Gemma3 1B checkpoint #6499
• chore: Improve the AutoTuner log information. by @hyukn in chore: Improve the AutoTuner log information. #6368
• [TRTLLM-6611][feat] Add warnings and stricter validation to LoraManager adapter loading by @venkywonka in [TRTLLM-6611][feat] Add warnings and stricter validation to LoraManager adapter loading #6453
• Deepseek R1 FP8 Support on Blackwell by @zongfeijing in Deepseek R1 FP8 Support on Blackwell #6486
• fix: remove duplicate layer multiplication in KV cache size calculation by @jaedeok-nvidia in fix: remove duplicate layer multiplication in KV cache size calculation #6481

New Contributors

• @Yuening-wa made their first contribution in [TRTLLM-5863][feat] Support Weight-Only-Quantization in PyTorch Workflow #5850
• @zhou-yuxin made their first contribution in hopper-style context MLA #5713
• @ameynaik-hub made their first contribution in Mtp optimizations round1 #5689
• @lianakoleva made their first contribution in [fix] README link directs to intended doc #6340
• @fyf2016 made their first contribution in [FIX] fix bugs caused by None attention_bias during Qwen3 model convert engine #6344
• @NVShreyas made their first contribution in [Perf]: Add residual, norm for nemotron_nas models #6455
• @vadiklyutiy made their first contribution in [PERF] Move calculation Qwen2-VL's rotary_cos_sin to LLM worker process #6004

Full Changelog: v1.0.0rc4...v1.0.0rc5

---

This discussion was created from the release v1.0.0rc5.