Fix: Bedrock Application Inference Profile (AIP) is not streaming response with ChatBedrockConverse #568
+272
−25
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR fixed the streaming issue with ChatBedrockConverse when input is Application Inference Profile (AIP). The issue happened because langchain-aws fails to identify the foundation model used in AIP (e.g.,
arn:aws:bedrock:us-east-1:111111484058:application-inference-profile/c3myu2h6fllr
), therefore, it cannot set the streaming_support flag for the AIP correctly. So the entire response was returned to user as a whole rather than streaming the response back.Solution
We have to create a Bedrock client to call Bedrock get_inference_profile control plane API, and parse the foundation model id from the response. However, the existing
set_disable_streaming
function works on the raw user input before the ChatBedrockConverse object is instantiated, so we have to extract the logic of determining streaming_support for models into a common function_get_streaming_support
and invoke it in both places (i.e., the original place that works on raw user input, and the new place that works on the resolved model Id from get_inference_profile API call).Issue
#538
Test