Skip to content

Conversation

0x-fang
Copy link
Contributor

@0x-fang 0x-fang commented Aug 14, 2025

Description

This PR fixes the bug that cause streaming is broken for Amazon Nova model. The response was returned as a whole rather than streaming back chunks to customers, even though .stream() is used.

Root Cause and Fix

First of all, when using Amazon Nova model, langchain-aws will use ChatBedrockConverse even though we create ChatBedrock (code). Inside ChatBedrockConverse, the input parameter provider is default to empty string rather than None (code), therefore, the set_disable_streaming function will use provider="" to disable the streaming, even though the model can support streaming. To fix it, this PR updates set_disable_streaming function to extract provider from model Id correctly when provider is empty string as default value, so it can setup the streaming flag correctly.

Alternatively, user can set the provider parameter explicitly when initializing the ChatBedrock as below, which can make streaming work for Nova models without any code change.

llm = ChatBedrock(
    provider="amazon",
    model="us.amazon.nova-pro-v1:0",
)

for chunk in llm.stream(prompt):
    print(chunk)

Issue

#544

Copy link
Collaborator

@michaelnchin michaelnchin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @0x-fang !

@michaelnchin michaelnchin enabled auto-merge (squash) August 14, 2025 21:37
@michaelnchin michaelnchin disabled auto-merge August 14, 2025 21:37
@michaelnchin michaelnchin enabled auto-merge (squash) August 15, 2025 17:35
auto-merge was automatically disabled August 15, 2025 17:36

Pull request was closed

@michaelnchin michaelnchin reopened this Aug 15, 2025
@michaelnchin michaelnchin merged commit 9ab4dbe into langchain-ai:main Aug 15, 2025
15 of 17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Cannot use streaming with Amazon Nova models through ChatBedrock when tools are bound
2 participants