Add RUM SDK injection for servlet based web servers #9110

PerfectSlayer · 2025-07-07T07:58:26Z

What Does This Do

This PR introduces RUM SDK injection for servlet APIs 3, 4 and 5.

In addition to the RUM SDK injection itself, the PR also contains:

micro-benchmark for the OutputStream wrapper in charge of the SDK script injection
injection and its config unit tests (arbitrary config key/map unsupported)
web server instrumentation tests for Jetty 11 and 12, Tomcat 5.5
smoke tests for Tomcat 9, 10 and 11

Motivation

Brings parity with ngnix and apache support.

Additional Notes

Co-authored by @amarziali and @PerfectSlayer

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, move, or deletion
Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

RUM injection remote config is not dynamic as other products

PerfectSlayer

Left few questions.
Looking good to me 👍

...quest-3/src/main/java/datadog/trace/instrumentation/servlet3/WrappedServletOutputStream.java

...st-5/src/main/java/datadog/trace/instrumentation/servlet5/RumHttpServletResponseWrapper.java

...st-3/src/main/java/datadog/trace/instrumentation/servlet3/RumHttpServletResponseWrapper.java

internal-api/src/main/java/datadog/trace/api/rum/RumInjector.java

internal-api/src/main/java/datadog/trace/api/Config.java

internal-api/src/main/java/datadog/trace/api/rum/RumInjectorConfig.java

.../src/main/java/datadog/trace/bootstrap/instrumentation/buffer/InjectingPipeOutputStream.java

mcculls · 2025-07-08T09:53:53Z

.../src/main/java/datadog/trace/bootstrap/instrumentation/buffer/InjectingPipeOutputStream.java

+        found = true;
+        drain();
+        out.write(b, off, idx);
+        out.write(contentToInject);


How resilient is this code to a misbehaving out stream?

In other words if out.write suddenly threw an IOException in the middle of one of our buffering methods, would the state kept here remain consistent - so that if the application code attempted to write again, everything would resume as normal...

One way to confirm this would be to run tests where the out stream being wrapped threw an IOException on every other call. Ideally you should end up with the same behaviour from the application's perspective with or without this wrapper.

It is not today. I need kind of checkpoint the action on the buffer in order to make it resilient

...st-5/src/main/java/datadog/trace/instrumentation/servlet5/RumHttpServletResponseWrapper.java

mcculls · 2025-07-08T11:34:15Z

...st-5/src/main/java/datadog/trace/instrumentation/servlet5/RumHttpServletResponseWrapper.java

+      return super.getWriter();
+    }
+    if (printWriter == null) {
+      printWriter = new PrintWriter(getOutputStream());


This could result in changed behaviour if the underlying servlet framework has a getWriter() implementation that does more than just delegate to getOutputStream()

For example: https://github.com/undertow-io/undertow/blob/main/servlet/src/main/java/io/undertow/servlet/spec/HttpServletResponseImpl.java#L381

We could add a WrappedPrintWriter and pass it the result of super.getWriter() - while it means more code IMHO it would be safer

Yes I had the same concern indeed. I wanted to safe few wrapping but I'm also incline to do that

...st-3/src/main/java/datadog/trace/instrumentation/servlet3/RumHttpServletResponseWrapper.java

mcculls · 2025-07-08T11:48:38Z

...st-3/src/main/java/datadog/trace/instrumentation/servlet3/RumHttpServletResponseWrapper.java

+      return super.getWriter();
+    }
+    if (printWriter == null) {
+      printWriter = new PrintWriter(getOutputStream());


As in the servlet-5 instrumentation, I think a WrappedPrintWriter approach would be safer.

.../src/main/java/datadog/trace/bootstrap/instrumentation/buffer/InjectingPipeOutputStream.java

mcculls · 2025-07-08T12:31:34Z

.../src/main/java/datadog/trace/bootstrap/instrumentation/buffer/InjectingPipeOutputStream.java

+      }
+    } else {
+      // use slow path because the length to write is small and within the lookbehind buffer size
+      super.write(b, off, len);


This assumes that super.write(b, off, len) will repeatedly call write(int b) on this instance.

If it instead calls a shared internal method used by both write methods then we wouldn't detect the marker.

EDIT: to be sure we probably want to call write(int b) ourselves in a loop...

I assumed that because of FilterOutputStream does it but I'm not sure that the implementation will stay stable throughout the jdk versions

mcculls

Some things to be confirmed:

Is it feasible to add a PrintWriter wrapper?
How does InjectingPipeOutputStream behave with an unreliable stream?
Slow path should repeatedly call write(int b) rather than assume that super.write(b, off, len) will do it
Can we document somewhere the effect of RUM injection on Content-Length headers (i.e. they won't appear because the size cannot be pre-determined)

Since this is under an opt-in feature flag I'm ok with this being merged while these issues are investigated

| Package | Type | Package file | Manager | Update | Change | |---|---|---|---|---|---| | [com.google.errorprone:error_prone_annotations](https://errorprone.info) ([source](https://github.com/google/error-prone)) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `2.39.0` -> `2.40.0` | | [org.apache.commons:commons-lang3](https://commons.apache.org/proper/commons-lang/) ([source](https://gitbox.apache.org/repos/asf/commons-lang.git)) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `3.17.0` -> `3.18.0` | | [org.jetbrains.kotlinx.binary-compatibility-validator](https://github.com/Kotlin/binary-compatibility-validator) | plugin | misk/gradle/libs.versions.toml | gradle | patch | `0.18.0` -> `0.18.1` | | [com.datadoghq:dd-trace-api](https://github.com/datadog/dd-trace-java) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `1.50.1` -> `1.51.0` | | [software.amazon.awssdk:sdk-core](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.77` -> `2.31.78` | | [software.amazon.awssdk:sqs](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.77` -> `2.31.78` | | [software.amazon.awssdk:dynamodb-enhanced](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.77` -> `2.31.78` | | [software.amazon.awssdk:dynamodb](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.77` -> `2.31.78` | | [software.amazon.awssdk:aws-core](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.77` -> `2.31.78` | | [software.amazon.awssdk:bom](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.77` -> `2.31.78` | | [software.amazon.awssdk:auth](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.77` -> `2.31.78` | --- ### Release Notes <details> <summary>google/error-prone (com.google.errorprone:error_prone_annotations)</summary> ### [`v2.40.0`](https://github.com/google/error-prone/releases/tag/v2.40.0): Error Prone 2.40.0 Changes: - Bug fixes and improvements - Releases (including snapshots) have migrated from [OSSRH to the Central Publisher Portal](https://central.sonatype.org/pages/ossrh-eol/#process-to-migrate) Full changelog: google/error-prone@v2.39.0...v2.40.0 </details> <details> <summary>Kotlin/binary-compatibility-validator (org.jetbrains.kotlinx.binary-compatibility-validator)</summary> ### [`v0.18.1`](https://github.com/Kotlin/binary-compatibility-validator/releases/tag/0.18.1) [Compare Source](Kotlin/binary-compatibility-validator@0.18.0...0.18.1) #### What's Changed - Fixed a bug preventing use of cross-compilation support during KLIB dump validation \[[#304](https://github.com/Kotlin/binary-compatibility-validator/issues/304)]\[[#306](https://github.com/Kotlin/binary-compatibility-validator/issues/306)] </details> <details> <summary>datadog/dd-trace-java (com.datadoghq:dd-trace-api)</summary> ### [`v1.51.0`](https://github.com/DataDog/dd-trace-java/releases/tag/v1.51.0): 1.51.0 ### Components #### Application Security Management (IAST) - 🐛 Fix verify error when ctor params are used after a call site ([#9083](DataDog/dd-trace-java#9083) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - 🐛 Limit the maximum size of the location path in IAST vulnerabilities ([#9028](DataDog/dd-trace-java#9028) - [@jandro996](https://github.com/jandro996)) - 🐛 Fix IAST gRPC handler with null superclass ([#8984](DataDog/dd-trace-java#8984) - [@smola](https://github.com/smola)) - ✨ Optimize IAST Vulnerability Detection ([#8885](DataDog/dd-trace-java#8885) - [@jandro996](https://github.com/jandro996)) #### Application Security Management (WAF) - ✨ Upgrade libddwaf-java to 15.0.0 ([#9022](DataDog/dd-trace-java#9022) - [@sezen-datadog](https://github.com/sezen-datadog)) - ✨ Extract RestEasy json body response schemas ([#9015](DataDog/dd-trace-java#9015) - [@jandro996](https://github.com/jandro996)) - ✨ Extract Jersey json body response schemas ([#9014](DataDog/dd-trace-java#9014) - [@jandro996](https://github.com/jandro996)) - ✨ Extract Ratpack json body response schemas ([#9013](DataDog/dd-trace-java#9013) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - ✨ Enable API Security by default and make it lazy loading ([#9009](DataDog/dd-trace-java#9009) - [@smola](https://github.com/smola)) - ✨ Extract Vert.x json body response schemas ([#9001](DataDog/dd-trace-java#9001) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - ✨ Extract Play json body response schemas ([#8995](DataDog/dd-trace-java#8995) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - 🐛 Fix Jackson nodes introspection for request/response schema extraction ([#8980](DataDog/dd-trace-java#8980) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - ✨ Extract Spring json body response schemas ([#8938](DataDog/dd-trace-java#8938) - [@sezen-datadog](https://github.com/sezen-datadog)) - ✨ Default obfuscation regexp update ([#8937](DataDog/dd-trace-java#8937) - [@sezen-datadog](https://github.com/sezen-datadog)) #### Build & Tooling - ✨ Cancel GitLab running pipeline on new PR push ([#9023](DataDog/dd-trace-java#9023) - [@PerfectSlayer](https://github.com/PerfectSlayer)) - ✨ Migrate publishing to Maven Central Portal ([#8807](DataDog/dd-trace-java#8807) - [@sarahchen6](https://github.com/sarahchen6)) #### Continuous Integration Visibility - 🐛 Fix Test Optimization to work with JDK 24 ([#9114](DataDog/dd-trace-java#9114) - [@nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Add repo root as safe directory on git client creation ([#9033](DataDog/dd-trace-java#9033) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - ✨ Add PR number tag and improve PR information building ([#8990](DataDog/dd-trace-java#8990) - [@daniel-mohedano](https://github.com/daniel-mohedano)) - ✨ Update impacted tests logic ([#8923](DataDog/dd-trace-java#8923) - [@daniel-mohedano](https://github.com/daniel-mohedano)) #### Data Streams Monitoring - 🧹 Clean up DSM context injection ([#8776](DataDog/dd-trace-java#8776) - [@PerfectSlayer](https://github.com/PerfectSlayer)) #### Database Monitoring - 🐛 Set trace\_injected in try block ([#9025](DataDog/dd-trace-java#9025) - [@natashadada](https://github.com/natashadada)) #### Dynamic Instrumentation - 🐛 Add source file tracking enable option ([#9115](DataDog/dd-trace-java#9115) - [@jpbempel](https://github.com/jpbempel)) - ✨ Add java.util.Date support ([#9111](DataDog/dd-trace-java#9111) - [@jpbempel](https://github.com/jpbempel)) - ✨ Update file probe format ([#9047](DataDog/dd-trace-java#9047) - [@jpbempel](https://github.com/jpbempel)) - ✨ add safe local var hoisting ([#9034](DataDog/dd-trace-java#9034) - [@jpbempel](https://github.com/jpbempel)) - 🧹 Add new config for debugger upload interval ([#8959](DataDog/dd-trace-java#8959) - [@jpbempel](https://github.com/jpbempel)) - ✨ Enable Code Origin with Dynamic instrumentation ([#8940](DataDog/dd-trace-java#8940) - [@jpbempel](https://github.com/jpbempel)) #### ML Observability (LLMObs) - 💡 LLM Observability SDK ([#8781](DataDog/dd-trace-java#8781) - [@gary-huang](https://github.com/gary-huang), [@nayeem-kamal](https://github.com/nayeem-kamal)) #### Metrics - 🐛 Ensure client stat reporter is started when the agent is not available at bootstrap ([#9082](DataDog/dd-trace-java#9082) - [@amarziali](https://github.com/amarziali)) - ✨ Create metric: appsec.waf.config\_errors ([#8394](DataDog/dd-trace-java#8394) - [@sezen-datadog](https://github.com/sezen-datadog)) #### Platform components - ✨ Introduce environment component ([#9071](DataDog/dd-trace-java#9071) - [@PerfectSlayer](https://github.com/PerfectSlayer)) #### Profiling - 🐛 Remove annoying warning for smap event parsing ([#9119](DataDog/dd-trace-java#9119) - [@jbachorik](https://github.com/jbachorik)) - 🐛 Fix ByteCountingInputStream when reading past EOF ([#8988](DataDog/dd-trace-java#8988) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) #### Realtime User Monitoring - ✨ Add RUM SDK injection for servlet based web servers ([#9110](DataDog/dd-trace-java#9110) - [@PerfectSlayer](https://github.com/PerfectSlayer) [@amarziali](https://github.com/amarziali)) #### Telemetry - ✨ Update the config origin metric to match what it's mapping ([#9045](DataDog/dd-trace-java#9045) - [@sezen-datadog](https://github.com/sezen-datadog)) #### Testing - ✨ Add testing for latest stable version (JDK 24) ([#8875](DataDog/dd-trace-java#8875) - [@sarahchen6](https://github.com/sarahchen6)) #### Trace context propagation - 🐛 Fix bug with dropping baggage when `TracePropagationBehaviorExtract=IGNORE` ([#9037](DataDog/dd-trace-java#9037) - [@mhlidd](https://github.com/mhlidd)) - 🐛 Fix ArrayIndexOutOfBoundsException in PercentEscaper ([#9032](DataDog/dd-trace-java#9032) - [@mhlidd](https://github.com/mhlidd)) #### Tracer core - 🐛 Fix `Error` handling for trace interceptors ([#9097](DataDog/dd-trace-java#9097) - [@AlexeyKuznetsov-DD](https://github.com/AlexeyKuznetsov-DD)) - 💡 Add wildcard feature for `DD_TRACE_HEADER_TAGS` and enabling for Http Response headers ([#9067](DataDog/dd-trace-java#9067) - [@mhlidd](https://github.com/mhlidd)) #### Tracer public API - 💡 Add LLM Observability SDK ([#8781](DataDog/dd-trace-java#8781) - [@gary-huang](https://github.com/gary-huang)) ### Instrumentations #### Akka instrumentation - 🐛 Fix NPE in akka-http and pekko-http integrations ([#9019](DataDog/dd-trace-java#9019) - [@mcculls](https://github.com/mcculls)) #### Eclipse Vert.x instrumentation - ✨ Extract Vert.x json body response schemas ([#9001](DataDog/dd-trace-java#9001) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) - ✨ Write http.route tag as soon as possible in vert.x ([#8952](DataDog/dd-trace-java#8952) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) #### JAX-WS instrumentation - 💡⚠️ Enable jax-ws integration by default ([#9030](DataDog/dd-trace-java#9030) - [@bm1549](https://github.com/bm1549)) - ✨ Extract Jersey json body response schemas ([#9014](DataDog/dd-trace-java#9014) - [@jandro996](https://github.com/jandro996)) #### Mule instrumentation - 🐛 Propagate grizzly http span in filters if nothing is active ([#9016](DataDog/dd-trace-java#9016) - [@amarziali](https://github.com/amarziali)) #### Play Framework instrumentation - ✨ Extract Play json body response schemas ([#8995](DataDog/dd-trace-java#8995) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) #### Ratpack instrumentation - ✨ Extract Ratpack json body response schemas ([#9013](DataDog/dd-trace-java#9013) - [@manuel-alvarez-alvarez](https://github.com/manuel-alvarez-alvarez)) #### Spring instrumentation - ✨ Extract Spring json body response schemas ([#8938](DataDog/dd-trace-java#8938) - [@sezen-datadog](https://github.com/sezen-datadog)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - "after 6pm every weekday,before 2am every weekday" in timezone Australia/Melbourne, Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Never, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). GitOrigin-RevId: 649b690d4c9d7dcb572c457f0802b42b8e3e682e

PerfectSlayer and others added 23 commits July 3, 2025 17:45

feat(rum): Add initial config and API

257f167

feat(rum): Iterating on initial config and API

5e734e3

fix(rum): Fix config print

dde7164

fix(rum): Remove content type support as only "text/html" is supported

bb22e1a

feat(rum): Add smoke tests

1bf99a1

feat(rum): Add smoke tests

8814d86

feat(rum): Add smoke tests

a1f0a77

feat(rum): Add smoke tests

f941b4a

Add RUM injection for servlet 3

fe2006c

fix rum injection smoke test

2ac167a

fix javadoc

c389b77

Add benchmark

01f6aae

avoid linkage issues with earlier servlet specs

486ee57

feat(rum): Improve smoke test to add more cases

2a55b69

feat(rum): Add remote config and fix json encoding

ce2c141

feat(rum): Add more smoke tests

9d43968

fix(rum): Fix config

a809044

fix(rum): Fix smoke tests

a0bae01

feat(rum): Add more smoke tests

42e3e6b

feat(rum): Simplify config to remove dynamic init

4f0b646

RUM injection remote config is not dynamic as other products

Use runnable for callback

1c49a79

improve pipe perfs

24c817e

feat(rum): Add injector and config unit tests

4beb3a8

PerfectSlayer added type: enhancement Enhancements and improvements comp: rum Realtime User Monitoring labels Jul 7, 2025

PerfectSlayer changed the title ~~Add RUM SDK injection for servlet based web server~~ Add RUM SDK injection for servlet based web servers Jul 7, 2025

amarziali and others added 3 commits July 7, 2025 11:48

Add rum injection for jakarta servlet

23f65fb

codenarc

dc3beab

Merge branch 'master' into project/rum-injection

78ce365

PerfectSlayer commented Jul 7, 2025

View reviewed changes

mcculls reviewed Jul 7, 2025

View reviewed changes

internal-api/src/main/java/datadog/trace/api/rum/RumInjector.java Outdated Show resolved Hide resolved

mcculls reviewed Jul 7, 2025

View reviewed changes

internal-api/src/main/java/datadog/trace/api/rum/RumInjector.java Outdated Show resolved Hide resolved

mcculls reviewed Jul 7, 2025

View reviewed changes

internal-api/src/main/java/datadog/trace/api/Config.java Show resolved Hide resolved

mcculls reviewed Jul 7, 2025

View reviewed changes

internal-api/src/main/java/datadog/trace/api/rum/RumInjectorConfig.java Show resolved Hide resolved

mcculls reviewed Jul 7, 2025

View reviewed changes

internal-api/src/main/java/datadog/trace/api/rum/RumInjectorConfig.java Outdated Show resolved Hide resolved

mcculls reviewed Jul 7, 2025

View reviewed changes

internal-api/src/main/java/datadog/trace/api/rum/RumInjectorConfig.java Show resolved Hide resolved

amarziali and others added 3 commits July 8, 2025 09:44

Apply suggestions

b1ea903

feat(rum): Improve config related to PR review feedback

6fb58b4

Improve and fix circular buffer

1e6faf9