Skip to content

subscriber: change registry exit to decrement local span ref only#3331

Merged
hds merged 4 commits intotokio-rs:mainfrom
davidhewitt:registry-exit
Nov 21, 2025
Merged

subscriber: change registry exit to decrement local span ref only#3331
hds merged 4 commits intotokio-rs:mainfrom
davidhewitt:registry-exit

Conversation

@davidhewitt
Copy link
Contributor

@davidhewitt davidhewitt commented Jul 3, 2025

Motivation

Fixes #3223

Solution

From my understanding, it's sufficient for Registry::exit just to call self.try_close() and do local bookkeeping:

  • The attempt to use the thread dispatcher leads directly to the issues observed in tracing & tracing-subscriber have a design defect I believe #3223, and
  • I don't think it's correct to call try_close on the whole Layer stack at this point anyway, a span being exited is not yet ready to close. All that is needed is to decrement the reference count within the current registry.

I've added a test which spawns a thread and enters (and exits, and drops) a span created on a dispatcher not registered to that thread.

Before this patch, the test fails with the span never being closed (because there is no thread dispatcher when the span is exited, and so a reference is leaked in Registry::exit).

With this patch, the bookkeeping demonstrated in the test seems correct to me.

@davidhewitt davidhewitt requested review from a team and hawkw as code owners July 3, 2025 15:17
@davidhewitt davidhewitt mentioned this pull request Jul 3, 2025
Closed
@davidhewitt
Copy link
Contributor Author

davidhewitt commented Jul 3, 2025

Note that changing Registry::exit to just call self.try_close() is also presumably a performance gain, because there is no need to dispatch through TLS on each span exit.

@vlovich
Copy link

vlovich commented Aug 1, 2025

FWIW I can also confirm this fixes the issue I've seen in real-world situations so this seems like a very simple fix that also improves performance. Great job!

@kevinji
Copy link

kevinji commented Sep 9, 2025

@jplatte are you able to review this change?

@jplatte
Copy link
Contributor

jplatte commented Sep 9, 2025

Hey, I'm sorry but I don't actually have a good understanding of tracing internals at all. I've just been reviewing and merging PRs that do not need much knowledge of the internals. I'm afraid this will have to wait for @hds.

@hds
Copy link
Contributor

hds commented Nov 19, 2025

Hey everyone. Just to let you know that I'm looking at this PR now. Hopefully get through it in the next couple of days.

@hds hds self-assigned this Nov 19, 2025
@hds hds added kind/bug Something isn't working crate/subscriber Related to the `tracing-subscriber` crate labels Nov 19, 2025
hds
hds requested changes Nov 20, 2025
View reviewed changes
Copy link
Contributor

@hds hds left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you very much for your PR! I think it makes sense, I a couple of comments on the test.

@davidhewitt
Copy link
Contributor Author

davidhewitt commented Nov 20, 2025

Thanks will try to get feedback addressed later today 👍

@davidhewitt davidhewitt requested a review from hds November 20, 2025 21:21
hds
hds approved these changes Nov 21, 2025
View reviewed changes
Copy link
Contributor

@hds hds left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your work on this PR!

@hds hds merged commit c287c84 into tokio-rs:main Nov 21, 2025
55 checks passed
@hds hds changed the title registry exit: decrement local span ref only subscriber: change registry exit to decrement local span ref only Nov 21, 2025
hds added a commit that referenced this pull request Nov 25, 2025
@hds
chore: prepare tracing-subscriber 0.3.21
f6ab6b2 
# 0.3.21 (November 30, 2025)

### Fixed

- Change registry exit to decrement local span ref only ([#3331])
- Make Layered propagate `on_register_dispatch` ([#3379])

### Performance

- Remove `clone_span` on enter (#3289)

### Documented

- Fix a few small things in the format module ([#3339])
- Fix extra closing brace in layer docs ([#3350])
- Fix link in `FmtSpan` docs ([#3411])

[#3331]: https://github.com/tokio-rs/tracing/pull/#3331
[#3339]: https://github.com/tokio-rs/tracing/pull/#3339
[#3350]: https://github.com/tokio-rs/tracing/pull/#3350
[#3379]: https://github.com/tokio-rs/tracing/pull/#3379
[#3411]: https://github.com/tokio-rs/tracing/pull/#3411
hds added a commit that referenced this pull request Nov 25, 2025
@hds
chore: prepare tracing-subscriber 0.3.21
5f6c48c 
# 0.3.21 (November 30, 2025)

### Fixed

- Change registry exit to decrement local span ref only ([#3331])
- Make Layered propagate `on_register_dispatch` ([#3379])

### Performance

- Remove `clone_span` on enter ([#3289])

### Documented

- Fix a few small things in the format module ([#3339])
- Fix extra closing brace in layer docs ([#3350])
- Fix link in `FmtSpan` docs ([#3411])

[#3289]: https://github.com/tokio-rs/tracing/pull/#3289
[#3331]: https://github.com/tokio-rs/tracing/pull/#3331
[#3339]: https://github.com/tokio-rs/tracing/pull/#3339
[#3350]: https://github.com/tokio-rs/tracing/pull/#3350
[#3379]: https://github.com/tokio-rs/tracing/pull/#3379
[#3411]: https://github.com/tokio-rs/tracing/pull/#3411
hds added a commit that referenced this pull request Nov 25, 2025
@hds
chore: prepare tracing-subscriber 0.3.21
75b8d01 
# 0.3.21 (November 30, 2025)

### Fixed

- Change registry exit to decrement local span ref only ([#3331])
- Make Layered propagate `on_register_dispatch` ([#3379])

### Performance

- Remove `clone_span` on enter ([#3289])

### Documented

- Fix a few small things in the format module ([#3339])
- Fix extra closing brace in layer docs ([#3350])
- Fix link in `FmtSpan` docs ([#3411])

[#3289]: https://github.com/tokio-rs/tracing/pull/#3289
[#3331]: https://github.com/tokio-rs/tracing/pull/#3331
[#3339]: https://github.com/tokio-rs/tracing/pull/#3339
[#3350]: https://github.com/tokio-rs/tracing/pull/#3350
[#3379]: https://github.com/tokio-rs/tracing/pull/#3379
[#3411]: https://github.com/tokio-rs/tracing/pull/#3411
hds added a commit that referenced this pull request Nov 26, 2025
@hds
chore: prepare tracing-subscriber 0.3.21
6c8c83b 
# 0.3.21 (November 26, 2025)

### Fixed

- Change registry exit to decrement local span ref only ([#3331])
- Make Layered propagate `on_register_dispatch` ([#3379])

### Changed

- `tracing`: updated to 0.1.42 ([#3418])

### Performance

- Remove `clone_span` on enter ([#3289])

### Documented

- Fix a few small things in the format module ([#3339])
- Fix extra closing brace in layer docs ([#3350])
- Fix link in `FmtSpan` docs ([#3411])

[#3289]: https://github.com/tokio-rs/tracing/pull/#3289
[#3331]: https://github.com/tokio-rs/tracing/pull/#3331
[#3339]: https://github.com/tokio-rs/tracing/pull/#3339
[#3350]: https://github.com/tokio-rs/tracing/pull/#3350
[#3379]: https://github.com/tokio-rs/tracing/pull/#3379
[#3411]: https://github.com/tokio-rs/tracing/pull/#3411
[#3418]: https://github.com/tokio-rs/tracing/pull/#3418
hds added a commit that referenced this pull request Nov 26, 2025
@hds
chore: prepare tracing-subscriber 0.3.21 (#3419)
a9eeed7 
# 0.3.21 (November 26, 2025)

### Fixed

- Change registry exit to decrement local span ref only ([#3331])
- Make Layered propagate `on_register_dispatch` ([#3379])

### Changed

- `tracing`: updated to 0.1.42 ([#3418])

### Performance

- Remove `clone_span` on enter ([#3289])

### Documented

- Fix a few small things in the format module ([#3339])
- Fix extra closing brace in layer docs ([#3350])
- Fix link in `FmtSpan` docs ([#3411])

[#3289]: https://github.com/tokio-rs/tracing/pull/#3289
[#3331]: https://github.com/tokio-rs/tracing/pull/#3331
[#3339]: https://github.com/tokio-rs/tracing/pull/#3339
[#3350]: https://github.com/tokio-rs/tracing/pull/#3350
[#3379]: https://github.com/tokio-rs/tracing/pull/#3379
[#3411]: https://github.com/tokio-rs/tracing/pull/#3411
[#3418]: https://github.com/tokio-rs/tracing/pull/#3418
jkachmar added a commit to MercuryTechnologies/hs-temporal-sdk that referenced this pull request Dec 3, 2025
@jkachmar
bump core sdk to include span drop fix
c094980 
* also ensure that 'tracing-subscriber' has been updated to 0.3.21,
  which should include their attempt at fixing this bug
  * cf. tokio-rs/tracing#3331
jkachmar added a commit to MercuryTechnologies/hs-temporal-sdk that referenced this pull request Dec 3, 2025
@jkachmar
bump core sdk to include span drop fix
0586274 
* also ensure that 'tracing-subscriber' has been updated to 0.3.21,
  which should include their attempt at fixing this bug
  * cf. tokio-rs/tracing#3331
jkachmar added a commit to MercuryTechnologies/hs-temporal-sdk that referenced this pull request Dec 3, 2025
@jkachmar
bump core sdk to include span drop fix
4e4ec2b 
* also ensure that 'tracing-subscriber' has been updated to 0.3.21,
  which should include their attempt at fixing this bug
  * cf. tokio-rs/tracing#3331
9999years added a commit to MercuryTechnologies/ghciwatch that referenced this pull request Dec 4, 2025
@9999years
cargo update
8cf955e 
This updates dependencies within semver bounds. In particular, it pulls
in this change which unblocks support for exporting OpenTelemetry spans
from `tracing`:
- tokio-rs/tracing#3331
@9999years 9999years mentioned this pull request Dec 4, 2025
Merged
jkachmar added a commit to MercuryTechnologies/hs-temporal-sdk that referenced this pull request Dec 4, 2025
@jkachmar
bump core sdk to include span drop fix
9bb3b02 
* also ensure that 'tracing-subscriber' has been updated to 0.3.21,
  which should include their attempt at fixing this bug
  * cf. tokio-rs/tracing#3331
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hds hds hds approved these changes

@hawkw hawkw Awaiting requested review from hawkw hawkw is a code owner

Assignees

@hds hds

Labels

crate/subscriber Related to the `tracing-subscriber` crate kind/bug Something isn't working

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

tracing & tracing-subscriber have a design defect I believe

5 participants

@davidhewitt @vlovich @kevinji @jplatte @hds