subscriber: implement per-layer filtering

[hawkw]

Motivation

Currently, filtering with Layers is always global. If a Layer
performs filtering, it will disable a span or event for all layers
that compose the current subscriber. In some cases, however, it is often
desirable for individual layers to see different spans or events than
the rest of the Layer stack.

Issues in other projects, such as tokio-rs/console#64 and
tokio-rs/console#76, linkerd/linkerd2-proxy#601,
influxdata/influxdb_iox#1012 and influxdata/influxdb_iox#1681,
jackwaudby/spaghetti#86, etc; as well as tracing feature requests like
#302, #597, and #1348, all indicate that there is significant demand for
the ability to add filters to individual Layers.

Unfortunately, doing this nicely is somewhat complicated. Although a
naive implementation that simply skips events/spans in Layer::on_event
and Layer::new_span based on some filter is relatively simple, this
wouldn't really be an ideal solution, for a number of reasons. A proper
per-layer filtering implementation would satisfy the following
desiderata:

• If a per-layer filter disables a span, it shouldn't be present for
  the layer that filter is attached to when iterating over span
  contexts (such as Context::event_scope, SpanRef::scope, etc), or
  when looking up a span's parents.
• When all per-layer filters disable a span or event, it should be
  completely disabled, rather than simply skipped by those particular
  layers. This means that per-layer filters should participate in
  enabled, as well as being able to skip spans and events in
  new_span and on_event.
• Per-layer filters shouldn't interfere with non-filtered Layers.
  If a subscriber contains layers without any filters, as well as
  layers with per-layer filters, the non-filtered Layers should
  behave exactly as they would without any per-layer filters present.
• Similarly, per-layer filters shouldn't interfere with global
  filtering. If a Layer in a stack is performing global filtering
  (e.g. the current filtering behavior), per-layer filters should also
  be effected by the global filter.
• Per-layer filters should be able to participate in Interest
  caching, but only when doing so doesn't interfere with
  non-per-layer-filtered layers.
• Per-layer filters should be tree-shaped. If a Subscriber consists
  of multiple layers that have been Layered together to form new
  Layers, and some of the Layered layers have per-layer filters,
  those per-layer filters should effect all layers in that subtree.
  Similarly, if Layers in a per-layer filtered subtree have their
  own per-layer filters, those layers should be effected by the union
  of their own filters and any per-layer filters that wrap them at
  higher levels in the tree.

Meeting all these requirements means that implementing per-layer
filtering correctly is somewhat more complex than simply skipping
events and spans in a Layer's on_event and new_span callbacks.

Solution

This branch has a basic working implementation of per-layer filtering
for tracing-subscriber v0.2. It should be possible to add this in a
point release without a breaking change --- in particular, the new
per-layer filtering feature doesn't interfere with the global
filtering behavior that layers currently have when not using the new
per-layer filtering APIs, so existing configurations should all behave
exactly as they do now.

Implementation

The new per-layer filtering API consists of a Filter trait that
defines a filtering strategy and a Filtered type that combines a
Filter with a Layer. When enabled is called on a Filtered layer,
it checks the metadata against the Filter and stores whether that
filter enabled the span/event in a thread-local cell. If every per-layer
filter disables a span/event, and there are no non-per-layer-filtered
layers in use, the Registry's enabled will return false.
Otherwise, when the span/event is recorded, each Filtered layer's
on_event and new_span method checks with the Registry to see
whether it enabled or disabled that span/event, and skips passing it
to the wrapped layer if it was disabled. When a span is recorded, the
Registry which filters disabled it, so that they can skip it when
looking up spans in other callbacks.

A new on_layer method was added to the Layer trait, which allows
running a callback when adding a Layer to a Subscriber. This allows
mutating both the Layer and the Subscriber, since they are both
passed by value when layering a Layer onto a Subscriber, and a new
register_filter method was added to LookupSpan, which returns a
FilterId type. When a Filtered layer is added to a subscriber that
implements LookupSpan, it calls register_filter in its on_layer
hook to generate a unique ID for its filter. FilterId can be used to
look up whether a span or event was enabled by the Filtered layer.

Internally, the filter results are stored in a simple bitset to reduce
the performance overhead of adding per-layer filtering as much as
possible. The FilterId type is actually a bitmask that's used to set
the bit corresponding to a Filtered layer in that bitmap when it
disables a span or event, and check whether the bit is set when querying
if a span or event was disabled. Setting a bit in the bitset and testing
whether its set is extremely efficient --- it's just a couple of bitwise
ops. Additionally, the "filter map" that tracks which filters disabled a
span or event is just a single u64 --- we don't need to allocate a
costly HashSet or similar every time we want to filter an event. This
does mean that we are limited to 64 unique filters per
Registry...however, I would be very surprised if anyone ever
builds a subscriber with 64 layers, let alone 64 separate per-layer
filters.

Additionally, the Context and SpanRef types now also potentially
carry FilterIds, so that they can filter span lookups and span
traversals for Filtered layers. When Filtered layers are nested
using the Layered, the Context combines the filter ID of the inner
and outer layers so that spans can be skipped if they were disabled by
either.

Finally, an implementation of the Filter trait was added for
LevelFilter, and new FilterFn and DynFilterFn structs were added
to produce a Filter implementation from a function pointer or closure.

Notes

Some other miscellaneous factors to consider when reviewing this change
include:

• I did a bit of unrelated refactoring in the layer module. Since we
  were adding more code to the Layer module (the Filter trait), the
  layer.rs file got significantly longer and was becoming somewhat
  hard to navigate. Therefore, I split out the Context and Layered
  types into their own files. Most of the code in those files is the
  same as it was before, although some of it was changed in order to
  support per-layer filtering. Apologies in advance to reviewers for
  this...
• In order to test this change, I added some new test-support code in
  tracing-subscriber. The tracing test support code provides a
  Subscriber implementation that can be configured to expect a
  particular set of spans and events, and assert that they occurred as
  expected. In order to test per-layer filtering, I added a new Layer
  implementation that does the same thing, but implements Layer
  rather than Subscriber. This allows testing per-layer filter
  behavior in the same way we test global filter behavior. Reviewers
  should feel free to just skim the test the test support changes, or
  skip reviewing them entirely.

Future Work

There is a bunch of additional stuff we can do once this change lands.
In order to limit the size of this PR, I didn't make these changes yet,
but once this merges, we will want to consider some important follow
up changes:

• Currently, only tracing-subscriber's Registry is capable of
  serving as a root subscriber for per-layer filters. This is in
  order to avoid stabilizing a lot of the per-layer filtering
  design as public API without taking the time to ensure there are
  no serious issues. In the future, we will want to add new APIs to
  allow users to implement their own root Subscribers that can
  host layers with per-layer filtering.
• The EnvFilter type doesn't implement the Filter trait and thus
  cannot currently be used as a per-layer filter. We should add an
  implementation of Filter for EnvFilter.
• The new FilterFn and DynFilterFn types could conceivably also
  be used as global filters. We could consider adding Layer
  implementations to allow them to be used for global filtering.
• We could consider adding new Filter implementations for performing
  common filtering predicates. For example, "only enable spans/events
  with a particular target or a set of targets", "only enable spans",
  "only enable events", etc. Then, we could add a combinator API to
  combine these predicates to build new filters.
• It would be nice if the Interest system could be used to cache
  the result of multiple individual per-layer filter evaluations, so
  that we don't need to always use enabled when there are several
  per-layer filters that disagree on whether or not they want a
  given span/event. There are a lot of unused bits in Interest
  that could potentially be used for this purpose. However, this
  would require new API changes in tracing-core, and is therefore
  out of scope for this PR.

Closes #302
Closes #597
Closes #1348

[davidbarsky]

(initial review, sorry. will add more comments later.)

[hawkw]

whew, okay, this finally has enough docs that i'm marking it as ready...there's room for improvement and expanding on some docs stuff, but i'd like to start the review process now that i'm not failing the "missing rustdoc" lint anymore.

[davidbarsky]

I'm maybe a fourth or fifth of the way through my review, but I'll leave some comments in the meantime.

[hawkw]

additional examples changes a la david's suggestions

[davidbarsky]

i love this passive-aggressive smiley that says "stay out of my territory"

[hawkw]

it wasn't intended to be passive aggressive, it was supposed to be reassuring! the intended message was "it's okay if this stuff is complicated, it won't be on the test"... ._.

[davidbarsky]

i think it's hilarious!

[Sushisource]

Thanks so much for landing this! I would vote a huge +1 to making it work with EnvFilter -- or even just having an easy way to parse env-filter Directives and use them with the Target filter, ignoring the things it doesn't support would make a huge difference.