[None][doc] GPT OSS Eagle3 #7140
Conversation
📝 WalkthroughTip 🔌 Remote MCP (Model Context Protocol) integration is now available!Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats. ✨ Finishing Touches🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR/Issue comments)Type Other keywords and placeholders
Status, Documentation and Community
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (12)
docs/source/blogs/tech_blog/blog10_GPT_OSS_Eagle3.md (12)
1-1: Promote title to H1 and add blog metadata/front‑matter (if blog engine expects it).Most pages in docs/blogs use a top-level H1. Also consider front‑matter (title/date/tags/authors) to integrate with the blog index.
Apply this minimal header change:
-## Running gpt-oss-120b with Eagle3 Speculative Decoding on GB200/B200 (TensorRT-LLM) +# Running gpt-oss-120b with Eagle3 Speculative Decoding on GB200/B200 (TensorRT-LLM)If your site supports MyST/Docs front‑matter, prepend:
--- title: Running gpt-oss-120b with Eagle3 Speculative Decoding on GB200/B200 (TensorRT-LLM) date: 2025-08-21 tags: [TensorRT-LLM, Eagle3, GB200, B200, GPT-OSS] authors: [Your Name] ---
7-11: Tighten prerequisite phrasing (grammar) and be explicit about GPU parallelism assumptions.Minor grammar and clarity polish; also foreshadow how tp/pp/ep/dp should match the actual GPU count.
-- NVIDIA GB200 or B200 GPUs (example below assumes 8 GPUs; adjust flags for your setup) +- NVIDIA GB200 or B200 GPUs (the example below assumes 8 GPUs; adjust parallelism flags to match your setup) - Fast SSD storage for model weights +- Fast SSD storage for model weights - Base model weights available under a directory named `gpt-oss-120b` (example path) +- Base model weights available under a directory named `gpt-oss-120b` - Eagle3 speculative model assets available under a directory named `eagle` +- Eagle3 speculative model assets available under a directory named `eagle`
14-18: Add a language to the fenced code block to satisfy markdownlint (MD040).-``` +```text /path/to/models/ ├─ gpt-oss-120b/ # base model directory └─ eagle/ # Eagle3 speculative decoding assets--- `20-31`: **Pin to a GA container tag or add a note to keep this in sync with the project’s recommended version.** The doc pins to release:1.1.0rc0 (an RC). If the repo README or install docs recommend a newer GA tag, this will drift. Add a note to check the current recommended tag before pulling. Would you like me to scan the repo docs and update this tag automatically across all references? --- `38-46`: **Mount models read-only in production and bind the YAML explicitly.** Serving should not need write access to model weights. Keeping the mount read-only reduces risk. If you adopt this, don’t create the YAML in-container; bind it from the host instead. ```diff docker run --rm --ipc=host -it \ --ulimit stack=67108864 \ --ulimit memlock=-1 \ --gpus all \ -p 8000:8000 \ - -v /path/to/models:/config/models:rw \ + -v /path/to/models:/config/models:ro \ + -v /path/to/eagle.yaml:/config/models/eagle/eagle.yaml:ro \ nvcr.io/nvidia/tensorrt-llm/release:1.1.0rc0 \ /bin/bashIf you prefer to keep creating the YAML inside the container, keep the existing rw mount here but add a caution that production should switch to ro binds.
55-59: Avoid persisting HF tokens to git credentials; pass tokens per-command instead.Minimize credential footprint inside containers by using the token flag on download commands instead of
huggingface-cli login --add-to-git-credential.-# Optional: authenticate if the repository requires it -# export HF_TOKEN=hf_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX -# huggingface-cli login --token "$HF_TOKEN" --add-to-git-credential +# Optional: authenticate if the repository requires it +# export HF_TOKEN=hf_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX # do not store this in shell historyI can also add a short “Security notes” callout consolidating NGC/HF credential handling if helpful.
61-71: Pass HF token explicitly on downloads; consider limiting files to reduce transfer time.Passing
--tokenavoids storing credentials;--includecan reduce transfer if repos contain artifacts you don’t need.huggingface-cli download openai/gpt-oss-120b \ --local-dir /config/models/gpt-oss-120b \ - --repo-type model + --repo-type model \ + ${HF_TOKEN:+--token "$HF_TOKEN"} @@ huggingface-cli download nvidia/gpt-oss-120b-Eagle3 \ --local-dir /config/models/eagle \ - --repo-type model + --repo-type model \ + ${HF_TOKEN:+--token "$HF_TOKEN"}
73-74: Format references as clickable links.-References: `https://huggingface.co/openai/gpt-oss-120b`, `https://huggingface.co/nvidia/gpt-oss-120b-Eagle3` +References: [openai/gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b), [nvidia/gpt-oss-120b-Eagle3](https://huggingface.co/nvidia/gpt-oss-120b-Eagle3)
86-95: Document tuning knobs (max_draft_len,max_batch_size) and GPU memory tradeoffs.A short note helps operators size these safely; e.g., larger
max_draft_lenand batch size increase memory.Add after the YAML:
- max_draft_len: higher values may improve throughput but increase VRAM usage.
- cuda_graph_config.max_batch_size: should not exceed the value used when capturing graphs; adjust based on your workload.
106-108: Right-size token/sequence limits to avoid OOM.131072 tokens/seq length are extreme and likely to exceed memory for 120B on 8 GPUs, especially with Eagle overhead. Suggest starting lower and documenting that these are tunables.
-... --max_num_tokens 131072 --max_seq_len 131072 +... --max_num_tokens 32768 --max_seq_len 32768
124-125: Call out unsupported params in a table or list for quick scanning.You already note greedy-only and ignored params. Consider a compact list/table and whether streaming is supported/unsupported in this setup.
129-139: Consider adding auth header example and a streaming variant.Some deployments gate the server with a static bearer token or reverse proxy. Showing
Authorization: Bearerhelps users adapt quickly. If streaming is supported later, include astream: trueexample.Example:
curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ + -H "Authorization: Bearer $API_KEY" \ -d '{ "model": "gpt-oss-120b", "messages": [ {"role": "user", "content": "Give me a two-sentence summary of Eagle3 speculative decoding."} ], "max_tokens": 128, "stream": false }'
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (1)
docs/source/blogs/tech_blog/blog10_GPT_OSS_Eagle3.md(1 hunks)
🧰 Additional context used
🪛 LanguageTool
docs/source/blogs/tech_blog/blog10_GPT_OSS_Eagle3.md
[grammar] ~7-~7: There might be a mistake here.
Context: ...mes 8 GPUs; adjust flags for your setup) - Fast SSD storage for model weights - Bas...
(QB_NEW_EN)
[grammar] ~8-~8: There might be a mistake here.
Context: ...up) - Fast SSD storage for model weights - Base model weights available under a dir...
(QB_NEW_EN)
[grammar] ~9-~9: There might be a mistake here.
Context: ...tory named gpt-oss-120b (example path) - Eagle3 speculative model assets availabl...
(QB_NEW_EN)
🪛 markdownlint-cli2 (0.17.2)
docs/source/blogs/tech_blog/blog10_GPT_OSS_Eagle3.md
14-14: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🔇 Additional comments (2)
docs/source/blogs/tech_blog/blog10_GPT_OSS_Eagle3.md (2)
117-118: Please verify health endpoint availability in a running instanceThe loop you ran returned
000for all paths, which typically means the service wasn’t reachable on localhost:8000. To confirm which endpoint your build actually exposes, please:
- Start the application locally (or point to a live deployment).
- Re-run the health‐check script against the running service.
Until you know which path returns a
200(or other healthy status), we can’t standardize on one URL in the docs. Once you’ve identified the correct endpoint(s), please updateblog10_GPT_OSS_Eagle3.mdaccordingly by:
Listing all supported health‐check paths (e.g.
/health,/v1/health,/health/live).Demonstrating a fallback check, for example:
status=$( curl -s -o /dev/null -w "%{http_code}" "http://localhost:8000/health" \ || curl -s -o /dev/null -w "%{http_code}" "http://localhost:8000/v1/health" \ || curl -s -o /dev/null -w "%{http_code}" "http://localhost:8000/health/live" ) echo "Health check status: $status"Standardizing the documented command on whichever path(s) your builds support.
Let me know which endpoint(s) work so we can finalize this section.
106-108: Fix parallelism flags: ensuretp_size * pp_size * ep_size * dp_sizeequals your total GPU countThe current example (
tp_size=8,ep_size=4) implies 32 GPUs. For an 8-GPU node, update to a matching configuration (assuming a dense model,pp_size=1,dp_size=1):-TRTLLM_ENABLE_PDL=1 trtllm-serve /config/models/gpt-oss-120b --host 0.0.0.0 --port 8000 --backend pytorch --max_batch_size 10 --tp_size 8 --ep_size 4 --trust_remote_code --extra_llm_api_options /config/models/eagle/eagle.yaml --max_num_tokens 131072 --max_seq_len 131072 +TRTLLM_ENABLE_PDL=1 trtllm-serve /config/models/gpt-oss-120b --host 0.0.0.0 --port 8000 --max_batch_size 10 --tp_size 8 --trust_remote_code --extra_llm_api_options /config/models/eagle/eagle.yaml --max_num_tokens 32768 --max_seq_len 32768Follow-up tasks:
- If you’re using a Mixture-of-Experts (MoE) model that requires expert parallelism, provide a consistent example (e.g.,
tp_size=4,ep_size=2on 8 GPUs).- Confirm whether the
--backend pytorchflag is still supported in your pinned TRT-LLM release (it may be deprecated).- The automated help check failed (
trtllm-serve: command not found). Please run the following inside the container or environment where TRT-LLM is installed to verify accepted flags, and update the example accordingly:trtllm-serve --help | sed -n '1,120p'
| trust_remote_code: true | ||
| kv_cache_config: | ||
| enable_block_reuse: false | ||
| free_gpu_memory_fraction: 0.8 | ||
| speculative_config: | ||
| decoding_type: Eagle | ||
| max_draft_len: 3 | ||
| speculative_model_dir: /config/models/eagle/ | ||
| cuda_graph_config: | ||
| max_batch_size: 10 | ||
| use_torch_sampler: true | ||
| moe_config: | ||
| backend: TRTLLM | ||
| EOF |
There was a problem hiding this comment.
Default trust_remote_code to false; explain when to enable.
Enabling remote code execution by default is risky. Recommend false with guidance to flip it only for trusted repos that require it.
-trust_remote_code: true
+trust_remote_code: false # set to true only if the model repo requires custom code and you trust the source📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| trust_remote_code: true | |
| kv_cache_config: | |
| enable_block_reuse: false | |
| free_gpu_memory_fraction: 0.8 | |
| speculative_config: | |
| decoding_type: Eagle | |
| max_draft_len: 3 | |
| speculative_model_dir: /config/models/eagle/ | |
| cuda_graph_config: | |
| max_batch_size: 10 | |
| use_torch_sampler: true | |
| moe_config: | |
| backend: TRTLLM | |
| EOF | |
| trust_remote_code: false # set to true only if the model repo requires custom code and you trust the source | |
| kv_cache_config: | |
| enable_block_reuse: false | |
| free_gpu_memory_fraction: 0.8 | |
| speculative_config: | |
| decoding_type: Eagle | |
| max_draft_len: 3 | |
| speculative_model_dir: /config/models/eagle/ | |
| cuda_graph_config: | |
| max_batch_size: 10 | |
| use_torch_sampler: true | |
| moe_config: | |
| backend: TRTLLM | |
| EOF |
🤖 Prompt for AI Agents
In docs/source/blogs/tech_blog/blog10_GPT_OSS_Eagle3.md around lines 82 to 95,
the configuration block defaults trust_remote_code to true which is unsafe;
change the default to false and add a short inline note explaining that users
should only set trust_remote_code: true for trusted repositories or models that
explicitly require remote code, and include a brief example or callout on
verifying source trust before enabling.
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
docs/source/blogs/tech_blog/blog10_GPT_OSS_Eagle3.md (1)
82-95: Do not enable remote code execution by default; set trust_remote_code: false and explain when to flip it.Defaulting to true is a security risk for readers copy-pasting this guide. Make it opt-in with a short inline note.
Apply this diff:
-trust_remote_code: true +trust_remote_code: false # set to true only if the repo explicitly requires custom code AND you trust the source
🧹 Nitpick comments (6)
docs/source/blogs/tech_blog/blog10_GPT_OSS_Eagle3.md (6)
14-18: Add a language hint to the directory tree code fence (markdownlint MD040).Apply this diff:
-``` +```text /path/to/models/ ├─ gpt-oss-120b/ # base model directory └─ eagle/ # Eagle3 speculative decoding assets--- `55-61`: **Avoid persisting HF credentials; prefer the env-var token flow.** Prevent accidental credential persistence in git-credential helpers or shell history. The CLI auto-detects HUGGINGFACE_HUB_TOKEN. Apply this diff: ```diff -# Optional: authenticate if the repository requires it -# export HF_TOKEN=hf_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX -# huggingface-cli login --token "$HF_TOKEN" --add-to-git-credential +# Optional: authenticate if the repository requires it (do not persist credentials) +# export HUGGINGFACE_HUB_TOKEN=hf_XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX +# Note: huggingface-cli picks up HUGGINGFACE_HUB_TOKEN automatically; no login neededAnd clear the token after downloads:
huggingface-cli download nvidia/gpt-oss-120b-Eagle3 \ --local-dir /config/models/eagle \ --repo-type model +unset HUGGINGFACE_HUB_TOKEN || true
73-74: Format references as links for better UX.Apply this diff:
-References: `https://huggingface.co/openai/gpt-oss-120b`, `https://huggingface.co/nvidia/gpt-oss-120b-Eagle3` +References: [openai/gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b), [nvidia/gpt-oss-120b-Eagle3](https://huggingface.co/nvidia/gpt-oss-120b-Eagle3)
48-49: Add a quick GPU visibility sanity check inside the container.Helps users catch Docker/NVIDIA runtime issues early.
Apply this diff:
Replace `/path/to/models` with the absolute path on your host. +Optional: verify GPUs are visible inside the container: + +```bash +nvidia-smi -L +```
110-121: Consider adding a sizing note to prevent OOM with large context lengths.max_num_tokens and max_seq_len at 131072 can be extremely memory-hungry. A one-line tip here to size these by VRAM and parallelism would reduce user footguns.
Apply this diff to add a tip:
The server will initialize, load, and optimize the models. Once ready, it will listen on port 8000. +Tip: Adjust --max_num_tokens and --max_seq_len based on available VRAM and tp/ep settings to avoid OOM on initialization.
7-11: Minor wording tweaks for the prerequisites list.Small clarity edits and consistent articles.
Apply this diff:
-- NVIDIA GB200 or B200 GPUs (example below assumes 8 GPUs; adjust flags for your setup) -- Fast SSD storage for model weights -- Base model weights available under a directory named `gpt-oss-120b` (example path) -- Eagle3 speculative model assets available under a directory named `eagle` +- NVIDIA GB200 or B200 GPUs (the example below assumes 8 GPUs; adjust flags for your setup). +- A fast SSD for model weights. +- Base model weights under a directory named `gpt-oss-120b`. +- Eagle3 speculative model assets under a directory named `eagle`.
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (1)
docs/source/blogs/tech_blog/blog10_GPT_OSS_Eagle3.md(1 hunks)
🧰 Additional context used
🪛 LanguageTool
docs/source/blogs/tech_blog/blog10_GPT_OSS_Eagle3.md
[grammar] ~7-~7: There might be a mistake here.
Context: ...mes 8 GPUs; adjust flags for your setup) - Fast SSD storage for model weights - Bas...
(QB_NEW_EN)
[grammar] ~8-~8: There might be a mistake here.
Context: ...up) - Fast SSD storage for model weights - Base model weights available under a dir...
(QB_NEW_EN)
[grammar] ~9-~9: There might be a mistake here.
Context: ...tory named gpt-oss-120b (example path) - Eagle3 speculative model assets availabl...
(QB_NEW_EN)
🪛 markdownlint-cli2 (0.17.2)
docs/source/blogs/tech_blog/blog10_GPT_OSS_Eagle3.md
14-14: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
🔇 Additional comments (2)
docs/source/blogs/tech_blog/blog10_GPT_OSS_Eagle3.md (2)
98-101: Confirm whether MoE settings are required for this model.The YAML includes moe_config.backend: TRTLLM. If gpt-oss-120b is dense, this stanza may be unnecessary and confusing.
If MoE is required, briefly explain how ep_size relates to the YAML and to serve-time flags; otherwise, remove moe_config from the example.
116-118: Double-check the health endpoint path.Some TensorRT-LLM builds expose /health, others use different paths or require a GET on /. Please confirm the default for trtllm-serve in this branch and update if needed.
| ```bash | ||
| TRTLLM_ENABLE_PDL=1 trtllm-serve /config/models/gpt-oss-120b --host 0.0.0.0 --port 8000 --backend pytorch --max_batch_size 10 --tp_size 8 --ep_size 4 --trust_remote_code --extra_llm_api_options /config/models/eagle/eagle.yaml --max_num_tokens 131072 --max_seq_len 131072 | ||
| ``` |
There was a problem hiding this comment.
💡 Verification agent
🧩 Analysis chain
Parallelism flags contradict the “8 GPUs” example and --trust_remote_code is unnecessary.
With tp_size 8 and ep_size 4 you imply at least 32 ranks; that doesn’t match the “example assumes 8 GPUs” note. Also, don’t pass --trust_remote_code in the serve command if we default it off in YAML.
Use a coherent 8‑GPU example by dropping ep_size (or set ep_size:1 explicitly) and remove the trust flag:
-TRTLLM_ENABLE_PDL=1 trtllm-serve /config/models/gpt-oss-120b --host 0.0.0.0 --port 8000 --backend pytorch --max_batch_size 10 --tp_size 8 --ep_size 4 --trust_remote_code --extra_llm_api_options /config/models/eagle/eagle.yaml --max_num_tokens 131072 --max_seq_len 131072
+TRTLLM_ENABLE_PDL=1 trtllm-serve /config/models/gpt-oss-120b --host 0.0.0.0 --port 8000 --backend pytorch --max_batch_size 10 --tp_size 8 --extra_llm_api_options /config/models/eagle/eagle.yaml --max_num_tokens 131072 --max_seq_len 131072If MoE is required and you truly need ep_size > 1, please add an explicit callout that readers must provision tp_size × ep_size (× pp_size if used) GPUs and update the “Prerequisites” bullet accordingly.
🏁 Script executed:
#!/bin/bash
# Locate the Prerequisites section in the blog post
grep -n "Prerequisites" docs/source/blogs/tech_blog/blog10_GPT_OSS_Eagle3.md
# Show a few lines of context around Prerequisites to confirm GPU count
grep -n -C3 "Prerequisites" docs/source/blogs/tech_blog/blog10_GPT_OSS_Eagle3.md
# Show the serve command example and surrounding commentary
sed -n '90,120p' docs/source/blogs/tech_blog/blog10_GPT_OSS_Eagle3.md
# Check if an example eagle.yaml is present in the repo to inspect default trust_remote_code
find . -type f -name "eagle.yaml"Length of output: 1703
Align GPU parallelism with prerequisites and remove the unnecessary trust flag
The example launch command in docs/source/blogs/tech_blog/blog10_GPT_OSS_Eagle3.md (around lines 106–108) currently specifies --tp_size 8 --ep_size 4, which implies 32 GPUs, contradicting the “example below assumes 8 GPUs” note in the Prerequisites. It also passes --trust_remote_code, which is typically disabled by default in your Eagle3 YAML.
• For an 8-GPU setup, drop the ep_size flag (or set --ep_size 1) so that tp_size×ep_size = 8.
• Remove --trust_remote_code if your YAML config already disables remote code loading.
• If you truly require MoE with ep_size > 1, please update the Prerequisites to reflect the total GPUs needed (i.e. tp_size × ep_size, plus any pp_size) and call this out alongside the command.
-TRTLLM_ENABLE_PDL=1 trtllm-serve /config/models/gpt-oss-120b \
- --host 0.0.0.0 --port 8000 --backend pytorch \
- --max_batch_size 10 --tp_size 8 --ep_size 4 --trust_remote_code \
- --extra_llm_api_options /config/models/eagle/eagle.yaml \
- --max_num_tokens 131072 --max_seq_len 131072
+TRTLLM_ENABLE_PDL=1 trtllm-serve /config/models/gpt-oss-120b \
+ --host 0.0.0.0 --port 8000 --backend pytorch \
+ --max_batch_size 10 --tp_size 8 \
+ --extra_llm_api_options /config/models/eagle/eagle.yaml \
+ --max_num_tokens 131072 --max_seq_len 131072Committable suggestion skipped: line range outside the PR's diff.
🤖 Prompt for AI Agents
In docs/source/blogs/tech_blog/blog10_GPT_OSS_Eagle3.md around lines 106 to 108,
the example launch command conflicts with the Prerequisites: it uses --tp_size 8
--ep_size 4 (implying 32 GPUs) and includes --trust_remote_code though the
Eagle3 YAML disables remote code. Update the command for an 8‑GPU example by
removing --ep_size (or setting --ep_size 1) so tp_size×ep_size = 8, and remove
--trust_remote_code if the YAML already disables remote code loading;
alternatively, if MoE is intended (ep_size > 1), update the Prerequisites text
to state the total GPU requirement (tp_size × ep_size plus any pp_size) and
mention this alongside the command.
Signed-off-by: Izzy Putterman <[email protected]>
Signed-off-by: Izzy Putterman <[email protected]>
Signed-off-by: Izzy Putterman <[email protected]>
IzzyPutterman
force-pushed
the
iputterman/gpt-oss-blog
branch
from
5c3bdbb to
fbb3804
Compare
There was a problem hiding this comment.
Actionable comments posted: 0
♻️ Duplicate comments (2)
docs/source/blogs/tech_blog/blog10_GPT_OSS_Eagle3.md (2)
82-95: Default trust_remote_code to false; document when to enable.Running with remote code enabled by default is risky. Default to false and add inline guidance.
-trust_remote_code: true +trust_remote_code: false # set to true only if the model repo requires custom code AND you trust the source
106-109: Parallelism flags imply 32 GPUs and contradict “8 GPUs” note; remove trust flag.tp_size 8 with ep_size 4 => 32 ranks. For the 8‑GPU example, drop ep_size (or set 1). Also remove --trust_remote_code if YAML defaults it off.
-TRTLLM_ENABLE_PDL=1 trtllm-serve /config/models/gpt-oss-120b --host 0.0.0.0 --port 8000 --backend pytorch --max_batch_size 10 --tp_size 8 --ep_size 4 --trust_remote_code --extra_llm_api_options /config/models/eagle/eagle.yaml --max_num_tokens 131072 --max_seq_len 131072 +TRTLLM_ENABLE_PDL=1 trtllm-serve /config/models/gpt-oss-120b \ + --host 0.0.0.0 --port 8000 --backend pytorch \ + --max_batch_size 10 --tp_size 8 \ + --extra_llm_api_options /config/models/eagle/eagle.yaml \ + --max_num_tokens 131072 --max_seq_len 131072If MoE with ep_size > 1 is intended, update Prerequisites to state the total GPU requirement: tp_size × ep_size (× pp_size if used).
🧹 Nitpick comments (7)
docs/source/blogs/tech_blog/blog10_GPT_OSS_Eagle3.md (7)
7-11: Tighten prerequisite bullets (grammar + clarity).Use short, punctuated sentences for readability.
-- NVIDIA GB200 or B200 GPUs (example below assumes 8 GPUs; adjust flags for your setup) -- Fast SSD storage for model weights -- Base model weights available under a directory named `gpt-oss-120b` (example path) -- Eagle3 speculative model assets available under a directory named `eagle` +- NVIDIA GB200 or B200 GPUs. The example below assumes 8 GPUs; adjust flags for your setup. +- Fast SSD storage for model weights. +- Base model weights under a directory named `gpt-oss-120b` (example path). +- Eagle3 speculative model assets under a directory named `eagle`.
14-18: Add a language hint to fenced block.Fixes markdownlint MD040.
-``` +```text /path/to/models/ ├─ gpt-oss-120b/ # base model directory └─ eagle/ # Eagle3 speculative decoding assets--- `59-71`: **Avoid symlink issues with huggingface-cli in containers.** Recommend disabling symlinks to prevent broken links on bind mounts/overlay FS. ```diff -huggingface-cli download openai/gpt-oss-120b \ +huggingface-cli download openai/gpt-oss-120b \ --local-dir /config/models/gpt-oss-120b \ + --local-dir-use-symlinks False \ --repo-type model @@ -huggingface-cli download nvidia/gpt-oss-120b-Eagle3 \ +huggingface-cli download nvidia/gpt-oss-120b-Eagle3 \ --local-dir /config/models/eagle \ + --local-dir-use-symlinks False \ --repo-type model
73-74: Use proper links instead of inline code ticks.Improves readability and keeps autolinking consistent.
-References: `https://huggingface.co/openai/gpt-oss-120b`, `https://huggingface.co/nvidia/gpt-oss-120b-Eagle3` +References: https://huggingface.co/openai/gpt-oss-120b, https://huggingface.co/nvidia/gpt-oss-120b-Eagle3
99-102: Version guard for sampler config.Call out the exact TRT-LLM version mapping to avoid confusion between use_torch_sampler vs sampler_type.
Add a short note:
- release:1.1.0rc0 => keep use_torch_sampler: true
- main/top-of-tree => use sampler_type: TorchSampler
Would you like me to add a tiny compatibility table here?
118-121: Optional: stricter health check.Use curl -f to fail on non-2xx and surface errors in CI/tutorials.
-curl -s -o /dev/null -w "Status: %{http_code}\n" "http://localhost:8000/health" +curl -fsS -o /dev/null -w "Status: %{http_code}\n" "http://localhost:8000/health"
125-139: Clarify ignored params are server-side no-ops.Add one sentence so users don’t expect client-side enforcement.
-Note: This Eagle3 + TensorRT-LLM endpoint currently supports only greedy sampling. The following Chat Completions parameters are ignored (no-ops): temperature, top_p, top_k, seed. +Note: This Eagle3 + TensorRT-LLM endpoint currently supports only greedy sampling. The following Chat Completions parameters are ignored server-side (no-ops): temperature, top_p, top_k, seed.
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (1)
docs/source/blogs/tech_blog/blog10_GPT_OSS_Eagle3.md(1 hunks)
🧰 Additional context used
🪛 LanguageTool
docs/source/blogs/tech_blog/blog10_GPT_OSS_Eagle3.md
[grammar] ~7-~7: There might be a mistake here.
Context: ...mes 8 GPUs; adjust flags for your setup) - Fast SSD storage for model weights - Bas...
(QB_NEW_EN)
[grammar] ~8-~8: There might be a mistake here.
Context: ...up) - Fast SSD storage for model weights - Base model weights available under a dir...
(QB_NEW_EN)
[grammar] ~9-~9: There might be a mistake here.
Context: ...tory named gpt-oss-120b (example path) - Eagle3 speculative model assets availabl...
(QB_NEW_EN)
🪛 markdownlint-cli2 (0.17.2)
docs/source/blogs/tech_blog/blog10_GPT_OSS_Eagle3.md
14-14: Fenced code blocks should have a language specified
(MD040, fenced-code-language)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)
- GitHub Check: Pre-commit Check
🔇 Additional comments (1)
docs/source/blogs/tech_blog/blog10_GPT_OSS_Eagle3.md (1)
1-4: Good high-level framing.Clear scope and constraints (GB200/B200 only) are helpful.
Summary by CodeRabbit
Description
Test Coverage
GitHub Bot Help
/bot [-h] ['run', 'kill', 'skip', 'reuse-pipeline'] ...Provide a user friendly way for developers to interact with a Jenkins server.
Run
/bot [-h|--help]to print this help message.See details below for each supported subcommand.
run [--reuse-test (optional)pipeline-id --disable-fail-fast --skip-test --stage-list "A10-PyTorch-1, xxx" --gpu-type "A30, H100_PCIe" --test-backend "pytorch, cpp" --add-multi-gpu-test --only-multi-gpu-test --disable-multi-gpu-test --post-merge --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx" --detailed-log --debug(experimental)]Launch build/test pipelines. All previously running jobs will be killed.
--reuse-test (optional)pipeline-id(OPTIONAL) : Allow the new pipeline to reuse build artifacts and skip successful test stages from a specified pipeline or the last pipeline if no pipeline-id is indicated. If the Git commit ID has changed, this option will be always ignored. The DEFAULT behavior of the bot is to reuse build artifacts and successful test results from the last pipeline.--disable-reuse-test(OPTIONAL) : Explicitly prevent the pipeline from reusing build artifacts and skipping successful test stages from a previous pipeline. Ensure that all builds and tests are run regardless of previous successes.--disable-fail-fast(OPTIONAL) : Disable fail fast on build/tests/infra failures.--skip-test(OPTIONAL) : Skip all test stages, but still run build stages, package stages and sanity check stages. Note: Does NOT update GitHub check status.--stage-list "A10-PyTorch-1, xxx"(OPTIONAL) : Only run the specified test stages. Examples: "A10-PyTorch-1, xxx". Note: Does NOT update GitHub check status.--gpu-type "A30, H100_PCIe"(OPTIONAL) : Only run the test stages on the specified GPU types. Examples: "A30, H100_PCIe". Note: Does NOT update GitHub check status.--test-backend "pytorch, cpp"(OPTIONAL) : Skip test stages which don't match the specified backends. Only support [pytorch, cpp, tensorrt, triton]. Examples: "pytorch, cpp" (does not run test stages with tensorrt or triton backend). Note: Does NOT update GitHub pipeline status.--only-multi-gpu-test(OPTIONAL) : Only run the multi-GPU tests. Note: Does NOT update GitHub check status.--disable-multi-gpu-test(OPTIONAL) : Disable the multi-GPU tests. Note: Does NOT update GitHub check status.--add-multi-gpu-test(OPTIONAL) : Force run the multi-GPU tests in addition to running L0 pre-merge pipeline.--post-merge(OPTIONAL) : Run the L0 post-merge pipeline instead of the ordinary L0 pre-merge pipeline.--extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx"(OPTIONAL) : Run the ordinary L0 pre-merge pipeline and specified test stages. Examples: --extra-stage "H100_PCIe-TensorRT-Post-Merge-1, xxx".--detailed-log(OPTIONAL) : Enable flushing out all logs to the Jenkins console. This will significantly increase the log volume and may slow down the job.--debug(OPTIONAL) : Experimental feature. Enable access to the CI container for debugging purpose. Note: Specify exactly one stage in thestage-listparameter to access the appropriate container environment. Note: Does NOT update GitHub check status.For guidance on mapping tests to stage names, see
docs/source/reference/ci-overview.mdand the
scripts/test_to_stage_mapping.pyhelper.kill
killKill all running builds associated with pull request.
skip
skip --comment COMMENTSkip testing for latest commit on pull request.
--comment "Reason for skipping build/test"is required. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.reuse-pipeline
reuse-pipelineReuse a previous pipeline to validate current commit. This action will also kill all currently running builds associated with the pull request. IMPORTANT NOTE: This is dangerous since lack of user care and validation can cause top of tree to break.