[AutoDeploy][Feature]: Spec Dec + Guided Decoding Support

[govind-ramnarayan]

🚀 The feature, motivation and pitch

Spec dec + guided decoding together cause some complications - roughly speaking, the sampling needs to do guided decoding, but then the draft tokens also need to be guided otherwise they will never be accepted.

Currently we just check that we don't have both enabled at the same time for AutoDeploy.

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

🔗 Similar Issues

Possible Duplicates

• ISSUE-8245

Related Issues

• [Feature]: Add support for guided decoding in AutoDeploy #8245
• [AutoDeploy][Feature]: Draft-Target Speculative Decoding #9147
• [Feature]: Support for guided decoding in speculative decoding mode #7006
• [AutoDeploy][Feature] Overlap scheduling + speculative decoding in AutoDeploy #9421

👤 Suggested Assignees

• govind-ramnarayan
• suyoggupta
• Pernekhan

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord or schedule a call!