feat(openai): support token metrics for completions and chat completions when not provided

[sabrenner]

What does this PR do?

Adds support for tagging token metrics and submitting them as metrics when they are not provided. This only happens for completions and chat.completions in the case of streamed responses.

This is a bit of a gray area, as although the Node.js OpenAI SDK does not provide these usage counts at all for streamed responses, the OpenAI API itself does, through the following option:

stream_options: {
  include_usage: true
}

Which, in the Node.js SDK, looks like:

const stream = await openai.chat.completions.create({
  model,
  messages:,
  stream: true,
  stream_options: {
    include_usage: true,
  },
});

While stream_options is not yet a supported option in the Node.js SDK, it is passed through as a part of the request body to the OpenAI API, and is reflected in the returned response for streamed cases, where the last chunk includes the usage.

However, in the case users do not want to specify this option all throughout their code, or are uneasy about using an unsupported option for the SDK, this change also introduces computing these token metrics client-side. This is done either by:

1. Leveraging tiktoken if it is installed, a popular library for token estimation for a variety if models. This is done by trying to require it in a try/catch, something I don't think is common for the Node.js tracer, but is done in the Python tracer, and is implemented here as well for consistency in experience.
2. In the case tiktoken isn't installed, or if there's an issue with using tiktoken, some basic estimations are used. This, again, follows the standard set in the Python tracer, and does not seem to have a noticeable performance impact (at least from tests, but will try to add some benchmarks).

Motivation

Expand support for OpenAI integration.

Plugin Checklist

• Unit tests.
• TypeScript definitions.
• TypeScript tests.
• API documentation.
• CircleCI jobs/workflows.
• Plugin is exported.

Additional Notes for Reviewers

Big chunk of LOC are because of test fixtures. Additionally, let me know if there are any concerns with optionally requiring tiktoken - I can try and use an approach that doesn't rely on a try/catch block, but instead checks for the dependency in the node_modules directly. However, I feel it is important to try and leverage tiktoken for the sake of consistency between tracers, and for more accurate token counts in the case that they aren't on the returned chunks.

[github-actions]

Overall package size

Self size: 6.6 MB
Deduped: 61.86 MB
No deduping: 62.14 MB

Dependency sizes

name	version	self size	total size
@datadog/native-appsec	8.0.1	15.59 MB	15.6 MB
@datadog/native-iast-taint-tracking	2.1.0	14.91 MB	14.92 MB
@datadog/pprof	5.3.0	9.85 MB	10.22 MB
protobufjs	7.2.5	2.77 MB	6.56 MB
@datadog/native-iast-rewriter	2.3.1	2.15 MB	2.24 MB
@opentelemetry/core	1.14.0	872.87 kB	1.47 MB
@datadog/native-metrics	2.0.0	898.77 kB	1.3 MB
@opentelemetry/api	1.8.0	1.21 MB	1.21 MB
import-in-the-middle	1.7.4	70.19 kB	739.86 kB
msgpack-lite	0.1.26	201.16 kB	281.59 kB
opentracing	0.14.7	194.81 kB	194.81 kB
semver	7.5.4	93.4 kB	123.8 kB
pprof-format	2.1.0	111.69 kB	111.69 kB
@datadog/sketches-js	2.1.0	109.9 kB	109.9 kB
lodash.sortby	4.7.0	75.76 kB	75.76 kB
lru-cache	7.14.0	74.95 kB	74.95 kB
ignore	5.2.4	51.22 kB	51.22 kB
int64-buffer	0.1.10	49.18 kB	49.18 kB
shell-quote	1.8.1	44.96 kB	44.96 kB
istanbul-lib-coverage	3.2.0	29.34 kB	29.34 kB
tlhunter-sorted-set	0.1.0	24.94 kB	24.94 kB
limiter	1.1.5	23.17 kB	23.17 kB
dc-polyfill	0.1.4	23.1 kB	23.1 kB
retry	0.13.1	18.85 kB	18.85 kB
jest-docblock	29.7.0	8.99 kB	12.76 kB
crypto-randomuuid	1.0.0	11.18 kB	11.18 kB
path-to-regexp	0.1.7	6.78 kB	6.78 kB
koalas	1.0.2	6.47 kB	6.47 kB
module-details-from-path	1.0.3	4.47 kB	4.47 kB

_{🤖 This report was automatically generated by heaviest-objects-in-the-universe}

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 80.42%. Comparing base (60f10dd) to head (5f8b189).

❗ Current head 5f8b189 differs from pull request most recent head f9b981d

Please upload reports for the commit f9b981d to get more accurate results.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #4366      +/-   ##
==========================================
- Coverage   85.01%   80.42%   -4.59%     
==========================================
  Files         257        3     -254     
  Lines       11361      373   -10988     
  Branches       33       33              
==========================================
- Hits         9658      300    -9358     
+ Misses       1703       73    -1630     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pr-commenter]

Benchmarks

Benchmark execution time: 2024-06-03 14:15:02

Comparing candidate commit f9b981d in PR branch sabrenner/openai-streamed-token-metrics with baseline commit 60f10dd in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 259 metrics, 7 unstable metrics.

[Yun-Kim]

LGTM from the ml-obs team, but open to the Node JS team's thoughts regarding the performance implications of try/catch for tiktoken.

[sabrenner]

APM folks - let me know if you have thoughts on this implementation. explained more in the PR description, but TL;DR in the Python tracer, we do something similar where we try/catch importing this tiktoken library. I included it here for consistency, and from customer ask. I'd prefer to have it as an option, but didn't want to explicitly make it an optional dependency for dd-trace-js, so if there are any objections/other ways of doing this, happy to implement!

[bengl]

This is totally fine.

[bengl]

This is totally fine.