Skip to content

feat(llmobs): support tracing bedrock converse stream#12873

Merged
lievan merged 19 commits intomainfrom
evan.li/bedrock-converse-streaming
Apr 3, 2025
Merged

feat(llmobs): support tracing bedrock converse stream#12873
lievan merged 19 commits intomainfrom
evan.li/bedrock-converse-streaming

Conversation

@lievan
Copy link
Copy Markdown
Contributor

@lievan lievan commented Mar 25, 2025

Supports tracing converse_stream method from the bedrock converse API

Since output messages, token information, and stop reason are all in the stream chunks returned by the API, we leave the responsibility to parse these out solely in _extract_output_message_for_converse_stream. This contains the bulk of the logic in the PR.

Checklist

  • PR author has checked that all the criteria below are met
  • The PR description includes an overview of the change
  • The PR description articulates the motivation for the change
  • The change includes tests OR the PR description describes a testing strategy
  • The PR description notes risks associated with the change, if any
  • Newly-added code is easy to change
  • The change follows the library release note guidelines
  • The change includes or references documentation updates if necessary
  • Backport labels are set (if applicable)

Reviewer Checklist

  • Reviewer has checked that all the criteria below are met
  • Title is accurate
  • All changes are related to the pull request's stated goal
  • Avoids breaking API changes
  • Testing strategy adequately addresses listed risks
  • Newly-added code is easy to change
  • Release note makes sense to a user of the library
  • If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
  • Backport labels are set in a manner that is consistent with the release branch maintenance policy

@lievan lievan marked this pull request as ready for review March 25, 2025 13:59
@lievan lievan requested review from a team as code owners March 25, 2025 13:59
@lievan lievan requested review from juanjux, sanchda and wantsui March 25, 2025 13:59
@lievan lievan changed the title Evan.li/bedrock converse streaming feat(llmobs): support tracing bedrock converse stream Mar 25, 2025
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Mar 25, 2025

CODEOWNERS have been resolved as:

releasenotes/notes/converse-stream-c0ac59a206aa16ad.yaml                @DataDog/apm-python
tests/snapshots/tests.contrib.botocore.test_bedrock.test_converse_stream.json  @DataDog/apm-python
ddtrace/contrib/internal/botocore/patch.py                              @DataDog/apm-core-python @DataDog/apm-idm-python
ddtrace/contrib/internal/botocore/services/bedrock.py                   @DataDog/ml-observability
ddtrace/llmobs/_integrations/bedrock.py                                 @DataDog/ml-observability
ddtrace/llmobs/_integrations/utils.py                                   @DataDog/ml-observability
tests/contrib/botocore/bedrock_cassettes/bedrock_converse_stream.yaml   @DataDog/ml-observability
tests/contrib/botocore/bedrock_utils.py                                 @DataDog/ml-observability
tests/contrib/botocore/test_bedrock.py                                  @DataDog/ml-observability
tests/contrib/botocore/test_bedrock_llmobs.py                           @DataDog/ml-observability

Comment thread ddtrace/llmobs/_integrations/bedrock.py Outdated
Comment thread ddtrace/llmobs/_integrations/bedrock.py
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Mar 25, 2025

Bootstrap import analysis

Comparison of import times between this PR and base.

Summary

The average import time from this PR is: 231 ± 3 ms.

The average import time from base is: 233 ± 4 ms.

The import time difference between this PR and base is: -2.0 ± 0.2 ms.

Import time breakdown

The following import paths have shrunk:

ddtrace.auto 1.998 ms (0.87%)
ddtrace.bootstrap.sitecustomize 1.333 ms (0.58%)
ddtrace.bootstrap.preload 1.333 ms (0.58%)
ddtrace.internal.products 1.333 ms (0.58%)
ddtrace.internal.remoteconfig.client 0.636 ms (0.28%)
ddtrace 0.665 ms (0.29%)

@pr-commenter
Copy link
Copy Markdown

pr-commenter Bot commented Mar 25, 2025

Benchmarks

Benchmark execution time: 2025-04-03 16:20:42

Comparing candidate commit 6df1cb2 in PR branch evan.li/bedrock-converse-streaming with baseline commit 9159b81 in branch main.

Found 2 performance improvements and 0 performance regressions! Performance is the same for 496 metrics, 2 unstable metrics.

scenario:iast_aspects-format_map_aspect

  • 🟩 execution_time [-1.185µs; -1.149µs] or [-23.370%; -22.644%]

scenario:iast_aspects-replace_aspect

  • 🟩 execution_time [-1.358µs; -1.316µs] or [-20.547%; -19.918%]

Comment thread ddtrace/contrib/internal/botocore/patch.py Outdated
Comment thread releasenotes/notes/converse-stream-c0ac59a206aa16ad.yaml Outdated
Comment thread ddtrace/llmobs/_integrations/bedrock.py Outdated
Comment thread ddtrace/llmobs/_integrations/bedrock.py Outdated
Comment thread ddtrace/contrib/internal/botocore/services/bedrock.py Outdated
lievan and others added 3 commits March 26, 2025 10:43
Co-authored-by: Emmett Butler <723615+emmettbutler@users.noreply.github.com>
Co-authored-by: Emmett Butler <723615+emmettbutler@users.noreply.github.com>
…og/dd-trace-py into evan.li/bedrock-converse-streaming
Copy link
Copy Markdown
Member

@Kyle-Verhoog Kyle-Verhoog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just one suggestion, otherwise lgtm

Comment thread ddtrace/llmobs/_integrations/bedrock.py Outdated
Copy link
Copy Markdown
Member

@Kyle-Verhoog Kyle-Verhoog left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just a nit but good from me

Comment thread ddtrace/llmobs/_integrations/bedrock.py Outdated
Comment thread ddtrace/llmobs/_integrations/bedrock.py Outdated
@lievan lievan enabled auto-merge (squash) April 3, 2025 15:42
@lievan lievan merged commit 0e3e27d into main Apr 3, 2025
324 checks passed
@lievan lievan deleted the evan.li/bedrock-converse-streaming branch April 3, 2025 16:21
chojomok pushed a commit that referenced this pull request Apr 7, 2025
Supports tracing `converse_stream` method from the bedrock converse API

Since output messages, token information, and stop reason are all in the
stream chunks returned by the API, we leave the responsibility to parse
these out solely in `_extract_output_message_for_converse_stream`. This
contains the bulk of the logic in the PR.

## Checklist
- [x] PR author has checked that all the criteria below are met
- The PR description includes an overview of the change
- The PR description articulates the motivation for the change
- The change includes tests OR the PR description describes a testing
strategy
- The PR description notes risks associated with the change, if any
- Newly-added code is easy to change
- The change follows the [library release note
guidelines](https://ddtrace.readthedocs.io/en/stable/releasenotes.html)
- The change includes or references documentation updates if necessary
- Backport labels are set (if
[applicable](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting))

## Reviewer Checklist
- [x] Reviewer has checked that all the criteria below are met 
- Title is accurate
- All changes are related to the pull request's stated goal
- Avoids breaking
[API](https://ddtrace.readthedocs.io/en/stable/versioning.html#interfaces)
changes
- Testing strategy adequately addresses listed risks
- Newly-added code is easy to change
- Release note makes sense to a user of the library
- If necessary, author has acknowledged and discussed the performance
implications of this PR as reported in the benchmarks PR comment
- Backport labels are set in a manner that is consistent with the
[release branch maintenance
policy](https://ddtrace.readthedocs.io/en/latest/contributing.html#backporting)

---------

Co-authored-by: lievan <evan.li@datadoqhq.com>
Co-authored-by: Emmett Butler <723615+emmettbutler@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants