Fix binary data serialization in dynamic instrumentation

[p-datadog]

What does this PR do?

Fixes binary data serialization in dynamic instrumentation snapshots. Binary strings (ASCII-8BIT encoding) and invalid UTF-8 strings are now escaped to a JSON-safe format, preventing JSON encoding failures when snapshots contain binary data. Also fixes exception handling for custom serializer conditions.

Motivation:

Binary data in DI snapshots causes JSON serialization to fail. When dynamic instrumentation captures variables containing binary strings (database records, protocol buffers, network payloads, etc.), the snapshot serialization fails during JSON encoding, preventing the snapshot from being sent to the backend. This implementation escapes binary data to b'\xHH' format (matches dd-trace-py output), handles invalid UTF-8 strings, applies truncation efficiently (limit original bytes, not escaped output), and preserves valid text in other encodings (Latin1, Windows-1252).

Additionally, strings with invalid UTF-8 encoding were causing exceptions during serialization, and custom serializer conditions using regex could fail on invalid UTF-8 strings.

Change log entry

Yes. Dynamic Instrumentation: Fix JSON serialization failures when snapshots contain binary data and invalid UTF-8 strings.

Additional Notes:

Implementation escapes binary bytes to b'\x80\xff' format with printable ASCII preserved, special characters (\n, \t, \r, \\, \') escaped, and other bytes as \xHH hex escapes. Truncation is applied to original binary data before escaping (e.g., 300-byte binary with limit=255 → truncate to 255 bytes, escape → 1023 char output, size=300). Output format matches dd-trace-py for consistency across language tracers. Escaping overhead is 2-6x (vs 40x for Java's approach).

Fixes:

• Invalid UTF-8 strings now use .bytesize and .byteslice instead of .length and [0...max] to avoid encoding exceptions
• Custom serializer conditions that raise exceptions (e.g., regex match against invalid UTF-8) are logged and skipped to prevent breaking the entire serialization process
• Added documentation in docs/DynamicInstrumentation.md explaining custom serializer exception handling behavior

Added comprehensive tests covering all byte values, truncation behavior, invalid UTF-8, valid non-UTF8 encodings, edge cases, custom serializer exception handling, and integration through transport layer.

How to test the change?

Run all binary data related tests:

bundle exec rspec spec/datadog/di/serializer_spec.rb spec/datadog/di/transport/input_spec.rb spec/datadog/di/integration/instrumentation_spec.rb

All tests pass (125 serializer tests including 2 new custom serializer exception tests).

Tested manually:

[github-actions]

Thank you for updating Change log entry section 👏

^{Visited at: 2026-03-11 20:09:57 UTC}

[pr-commenter]

Benchmarks

Benchmark execution time: 2026-03-12 02:37:52

Comparing candidate commit dfd176a in PR branch di-binary-data-serialization-tests with baseline commit 7a9046d in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 46 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

• 🟩 = significantly better candidate vs. baseline
• 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

[github-actions]

Typing analysis

Note: Ignored files are excluded from the next sections.

steep:ignore comments

This PR introduces 1 steep:ignore comment.

steep:ignore comments (+1-0)

❌ Introduced:

lib/datadog/di/serializer.rb:228

[datadog-official]

✅ Tests

🎉 All green!

❄️ No new flaky tests detected
🧪 All tests passed

🎯 Code Coverage (details)
• Patch Coverage: 96.06%
• Overall Coverage: 95.14% (-0.02%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: dfd176a | Docs | Datadog PR Page | Was this helpful? React with 👍/👎 or give us feedback!}

[Strech]

LGTM with few suggestions

1. The comment about working with strings and encodings gets repeated multiple times with a big multiline comment, maybe it could be trimmed or just referenced to a single place
2. A small suggestion for test