Allow sampling integrations when global rate is 1

[ngraef]

I have set DD_TRACE_SAMPLE_RATE=1.0 to enable Tracing Without Limits. However, I'd like to impose some limits. 😁

We're now ingesting about 30 GB per hour per environment from the dns integration alone, which far exceeds all of our other trace data combined by at least an order of magnitude. We have no reason to ingest 100% of those spans. I can disable the dns integration with DD_TRACE_DISABLED_PLUGINS=dns, but then we lose the metrics generated from those spans.

Is there a middle ground here? I tried setting DD_TRACE_DNS_ANALYTICS_SAMPLE_RATE=0.01 to configure the integration, but that doesn't seem to change the ingestion rate — perhaps because sampling decisions are made at the parent trace level? Is there a way to scale back ingestion of a subset of spans?

[rochdev]

The analytics sample rate is a legacy feature that existed before tracing without limits and doesn't work the same as the ingestion rate, so that can't really be used here. The ingestion rate works on entire traces, and not individual spans. Do you have any traces that contain only the DNS span? Otherwise I'm not sure I understand what you mean by having most of the data from DNS alone.

If it is the case that you have traces with only DNS that you don't care about and only want the spans and metrics for the ones that are part of an existing trace, we could add an option to the integration to allow or not DNS spans to be their own root. To be honest, when I think about it, this should even be disabled by default anyway since there are almost no use cases for this type of trace.

If I misunderstood and your use case is different, please provide more information about the traces you want to keep/drop and we should be able to come up with a way to do what you're trying to do.

[ngraef]

Thanks for the explanation. That makes sense.

Do you have any traces that contain only the DNS span?

Yes, most of the DNS spans are standalone traces for dns.lookup where the resource/hostname is the IP of the kubernetes node. I did some more digging and determined the source is the StatsD metrics exporter of the framework we're using. It's sending metrics data to the Datadog agent on the node using dgram socket.send, which does a dns lookup internally.

you have traces with only DNS that you don't care about and only want the spans and metrics for the ones that are part of an existing trace

I think this sums it up perfectly!

[rochdev]

What StatsD client are you using? I know that some of them allow caching the DNS lookup result which may resolve this issue, at least for your use case.

[jfirebaugh]

On https://app.datadoghq.com/apm/traces/ingestion-control, the "Configure Service Ingestion" UI suggests a way to configure sampling for individual sub-services for languages other than JS. For instance, I have a service named "multiplayer" which generates "multiplayer-dns" service traces. The "Configure Service Ingestion" UI suggests this for Python:

I think the request here is equivalent functionality for dd-trace-js. The code that this UI suggests for Node is:

const tracer = require('dd-trace')

// Provided sample rate overrides the default sample rate dynamically calculated by the agent
tracer.init({
  ingestion: {
    // Any traces started will be sampled at 10.00% with a rate limit of 100 per second
    sampleRate: 0.1000
  }
})

Which would obviously affect any multiplayer-* trace, not just multiplayer-dns.

[rochdev]

@brettlangdon For Python, does this work only for root services or any service at any level? Is the sampling decision shared across the trace or individually set on each span?