fix(llmobs): bedrock converse tracing is resistant to modifying the stream #13659
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
Bootstrap import analysisComparison of import times between this PR and base. SummaryThe average import time from this PR is: 285 ± 8 ms. The average import time from base is: 278 ± 3 ms. The import time difference between this PR and base is: 7.1 ± 0.3 ms. Import time breakdownThe following import paths have grown:
|
BenchmarksBenchmark execution time: 2025-06-17 19:04:24 Comparing candidate commit c8bce8e in PR branch Found 3 performance improvements and 0 performance regressions! Performance is the same for 520 metrics, 3 unstable metrics. scenario:iastdjangostartup-appsec
scenario:iastdjangostartup-iast
scenario:iastdjangostartup-tracer
|
lievan
changed the title
emmettbutler
approved these changes
Jun 13, 2025
ncybul
reviewed
Jun 13, 2025
Kyle-Verhoog
approved these changes
Jun 17, 2025
lievan
changed the title
lievan
force-pushed
the
evan.li/fix-bedrock-streaming
branch
2 times, most recently
from
13986c6 to
c8bce8e
Compare
|
The backport to To backport manually, run these commands in your terminal: # Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add .worktrees/backport-2.21 2.21
# Navigate to the new working tree
cd .worktrees/backport-2.21
# Create a new branch
git switch --create backport-13659-to-2.21
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 5592908cfe4af4aa4dffe12f6c30191bbac361c2
# Push it to GitHub
git push --set-upstream origin backport-13659-to-2.21
# Go back to the original working tree
cd ../..
# Delete the working tree
git worktree remove .worktrees/backport-2.21Then, create a pull request where the |
sydney-tung
pushed a commit
that referenced
this pull request
Jun 24, 2025
…tream (#13659) Fixes an issue where modifying chunks returned from a bedrock stream impacted our tracing of those chunks. Instead, our tracing should reflect the original response returned regardless of whether or not it was modified. This is relevant when libraries like langchain delete data from the raw streamed response (e.g. [popping](https://github.com/langchain-ai/langchain-aws/blob/40abb584979a349019d89bbf1cba7d8c56d23664/libs/aws/langchain_aws/chat_models/bedrock_converse.py#L995) `usageMetadata`). The fix uses an approach where we immediately process stream chunks as they are iterated over, instead of waiting until the entire stream has finished. The data flow is like this: 1. a streamed chunk is read from `TracedBotocoreConverseStream` 2. we send that chunk to `_output_stream_processor`, which reads all the relevant data from that chunk and builds the final output messages, token usage, metadata. This should block until we reach the next yield, at which point we've read all the data we needed for this chunk. The parsing logic is **unchanged** from the previous helper we used to parse the stream stream, except this method is now a generator. 4. we yield the chunk back to the user In terms of manual testing, i've verified that it fixes the langchain x bedrock converse streaming issue <img width="908" alt="image" src="https://github.com/user-attachments/assets/e4d1e4e5-3d1c-4f2d-a822-f7c7cdb169a6" /> if this logic looks good, it may be a pattern we will want to implement for our other integrations as well ## Checklist - [ ] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [ ] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) --------- Co-authored-by: lievan <[email protected]>
happynancee
pushed a commit
that referenced
this pull request
Jul 7, 2025
…tream (#13659) Fixes an issue where modifying chunks returned from a bedrock stream impacted our tracing of those chunks. Instead, our tracing should reflect the original response returned regardless of whether or not it was modified. This is relevant when libraries like langchain delete data from the raw streamed response (e.g. [popping](https://github.com/langchain-ai/langchain-aws/blob/40abb584979a349019d89bbf1cba7d8c56d23664/libs/aws/langchain_aws/chat_models/bedrock_converse.py#L995) `usageMetadata`). The fix uses an approach where we immediately process stream chunks as they are iterated over, instead of waiting until the entire stream has finished. The data flow is like this: 1. a streamed chunk is read from `TracedBotocoreConverseStream` 2. we send that chunk to `_output_stream_processor`, which reads all the relevant data from that chunk and builds the final output messages, token usage, metadata. This should block until we reach the next yield, at which point we've read all the data we needed for this chunk. The parsing logic is **unchanged** from the previous helper we used to parse the stream stream, except this method is now a generator. 4. we yield the chunk back to the user In terms of manual testing, i've verified that it fixes the langchain x bedrock converse streaming issue <img width="908" alt="image" src="https://github.com/user-attachments/assets/e4d1e4e5-3d1c-4f2d-a822-f7c7cdb169a6" /> if this logic looks good, it may be a pattern we will want to implement for our other integrations as well ## Checklist - [ ] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [ ] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) --------- Co-authored-by: lievan <[email protected]>
alyshawang
pushed a commit
that referenced
this pull request
Jul 25, 2025
…tream (#13659) Fixes an issue where modifying chunks returned from a bedrock stream impacted our tracing of those chunks. Instead, our tracing should reflect the original response returned regardless of whether or not it was modified. This is relevant when libraries like langchain delete data from the raw streamed response (e.g. [popping](https://github.com/langchain-ai/langchain-aws/blob/40abb584979a349019d89bbf1cba7d8c56d23664/libs/aws/langchain_aws/chat_models/bedrock_converse.py#L995) `usageMetadata`). The fix uses an approach where we immediately process stream chunks as they are iterated over, instead of waiting until the entire stream has finished. The data flow is like this: 1. a streamed chunk is read from `TracedBotocoreConverseStream` 2. we send that chunk to `_output_stream_processor`, which reads all the relevant data from that chunk and builds the final output messages, token usage, metadata. This should block until we reach the next yield, at which point we've read all the data we needed for this chunk. The parsing logic is **unchanged** from the previous helper we used to parse the stream stream, except this method is now a generator. 4. we yield the chunk back to the user In terms of manual testing, i've verified that it fixes the langchain x bedrock converse streaming issue <img width="908" alt="image" src="https://github.com/user-attachments/assets/e4d1e4e5-3d1c-4f2d-a822-f7c7cdb169a6" /> if this logic looks good, it may be a pattern we will want to implement for our other integrations as well ## Checklist - [ ] PR author has checked that all the criteria below are met - The PR description includes an overview of the change - The PR description articulates the motivation for the change - The change includes tests OR the PR description describes a testing strategy - The PR description notes risks associated with the change, if any - Newly-added code is easy to change - The change follows the [library release note guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html) - The change includes or references documentation updates if necessary - Backport labels are set (if [applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)) ## Reviewer Checklist - [ ] Reviewer has checked that all the criteria below are met - Title is accurate - All changes are related to the pull request's stated goal - Avoids breaking [API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces) changes - Testing strategy adequately addresses listed risks - Newly-added code is easy to change - Release note makes sense to a user of the library - If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment - Backport labels are set in a manner that is consistent with the [release branch maintenance policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting) --------- Co-authored-by: lievan <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes an issue where modifying chunks returned from a bedrock stream impacted our tracing of those chunks. Instead, our tracing should reflect the original response returned regardless of whether or not it was modified.
This is relevant when libraries like langchain delete data from the raw streamed response (e.g. popping
usageMetadata).The fix uses an approach where we immediately process stream chunks as they are iterated over, instead of waiting until the entire stream has finished.
The data flow is like this:
TracedBotocoreConverseStream_output_stream_processor, which reads all the relevant data from that chunk and builds the final output messages, token usage, metadata. This should block until we reach the next yield, at which point we've read all the data we needed for this chunk. The parsing logic is unchanged from the previous helper we used to parse the stream stream, except this method is now a generator.In terms of manual testing, i've verified that it fixes the langchain x bedrock converse streaming issue
if this logic looks good, it may be a pattern we will want to implement for our other integrations as well
Checklist
Reviewer Checklist