Fix: fix the streaming issue for nova models #580
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR fixes the bug that cause streaming is broken for Amazon Nova model. The response was returned as a whole rather than streaming back chunks to customers, even though
.stream()
is used.Root Cause and Fix
First of all, when using Amazon Nova model, langchain-aws will use ChatBedrockConverse even though we create ChatBedrock (code). Inside ChatBedrockConverse, the input parameter
provider
is default to empty string rather than None (code), therefore, the set_disable_streaming function will useprovider=""
to disable the streaming, even though the model can support streaming. To fix it, this PR updates set_disable_streaming function to extract provider from model Id correctly whenprovider
is empty string as default value, so it can setup the streaming flag correctly.Alternatively, user can set the
provider
parameter explicitly when initializing the ChatBedrock as below, which can make streaming work for Nova models without any code change.Issue
#544