fix: Memory leak from streams rooted in the global registry

[hoxbro]

Description

Closes #6945
Closes #6988

Alternative to #6970

The problem

Stream.registry used to be a WeakKeyDictionary, so an entry should disappear once the element does. It never did, because only the keys were weak. The values were ordinary references [self.registry[source] = [self], and a stream reaches its own source back through its subscribers:

    Stream.registry                <- module level, never goes away
    └─ [stream]                    <- the value, held strongly
       └─ subscriber
          └─ your app object
             └─ source element     <- and this is the weak key

holoviews/holoviews/streams.py

Line 172 in fff801d

registry = weakref.WeakKeyDictionary()

holoviews/holoviews/streams.py

Line 376 in fff801d

self.registry[source] = [self]

The key was reachable from its own value, so the entry never expired. Any element ever passed as source= stayed alive until the process exited, along with its data, its streams and whatever those subscribers were bound to.

#6875 tried to fix this by holding subscribers weakly, in _WeakSubscriber. That cut the wrong link, and it only did anything for bound methods:

• check required inspect.ismethod, so closures and lambdas were never
  wrapped and kept leaking
• a functools.partial passed the check but fell through to
  weakref.ref(subscriber). Nothing else referenced the partial, so it was
  collected straight away and the subscriber never fired ([Regression] Stream subscribers based on partial functions no longer work #6945)
• __bool__ reported param methods as dead, so calling cleanup() on one plot
  removed subscribers belonging to another (Plot.cleanup() severs other plots' subscribers on shared streams: owner filter defeated by _WeakSubscriber.__bool__ #6988)

What changed in this PR

Ownership was turned around. The source element now keeps its own streams in LabelledData._streams, and Stream.registry was reduced to a weak index that only exists so a clone with the same _plot_id can still find them. An element, its streams and their subscribers are now just a cycle, which the garbage collector frees once nothing else refers to it.

With the global reference gone, _WeakSubscriber was deleted and subscribers went back to being ordinary strong references. #6945 and #6988 stopped being possible rather than being worked around.

A second leak with the same effect turned up while measuring the first. Every plot registers on_session_destroyed on its document, and the document keeps that callback, and the whole plot with it. BokehPlot.cleanup set plot._document = None directly instead of going through the setter, so nothing ever removed the callback.

Outside a server that document is curdoc(), which lives for the whole process, so every plot ever rendered was kept. The deregistration was pulled out into Plot._unwatch_session and is now called from the setter and from both cleanup implementations.

Also fixed

• Renderer.last_plot held the last plot for the life of the process. It is now a weak reference.
• attach_streams compared plot.refresh against the (precedence, subscriber) pairs, so its guard never matched and an overlay subscribed the same plot four times. It now compares against stream.subscribers.
• _link_dimensioned_streams registered _stream_update at precedence exactly one, inside the range Stream.clear treats as belonging to the user, so clear("user") dropped it. It registers at 1.05 now.

AI Disclosure

Tool & Model: Claude Code:claude-opus-5
Usage:

• I have tested all AI-generated content in my PR.
• I take responsibility for all AI-generated content in my PR.

Checklist

• Pull request title follows the conventional format
• Tests added and are passing

[codecov]

Codecov Report

❌ Patch coverage is 97.04433% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.35%. Comparing base (39582e2) to head (b90ee61).

Files with missing lines	Patch %	Lines
holoviews/streams.py	90.24%	4 Missing ⚠️
holoviews/plotting/plot.py	90.00%	1 Missing ⚠️
holoviews/plotting/renderer.py	87.50%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6990      +/-   ##
==========================================
+ Coverage   89.33%   89.35%   +0.01%     
==========================================
  Files         344      344              
  Lines       74689    74790     +101     
==========================================
+ Hits        66727    66826      +99     
- Misses       7962     7964       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[hoxbro]

Asked Opus to review this branch

Review

Review — PR #6990: "fix: Memory leak from streams rooted in the global registry"

Overview

Inverts stream ownership: the source element owns its streams (LabelledData._streams), Stream.registry becomes a weak index (_StreamRegistry, weak keys and weak values) used only for _plot_id lookups. _WeakSubscriber is deleted and subscribers go back to strong refs. Plus four side fixes: Plot._unwatch_session, weak Renderer.last_plot, the attach_streams dedup guard, and _stream_update precedence 1 → 1.05.

The core design is right, and the claimed fixes were verified empirically against main:

• _link_dimensioned_streams: on main clear("user") drops _stream_update; on this branch it survives. Confirmed.
• attach_streams: overlay went from 4 subscribers to 1, and the Layout case still keeps two distinct plots' refresh (1.1 + 2) — the dedup is per-plot, not over-eager. Confirmed no behavioral loss (Stream.trigger already deduped via unique_iterator, so this is pure cleanup).
• Document reassignment now drops the old hook (plot.document = doc2 → hook gone from doc1). Confirmed.
• Pickle/deepcopy/__dask_tokenize__ of an element with _streams attached: clean, thanks to the __getstate__ pop (dimension.py:851). Good catch — without it, attaching a stream would have changed a Dataset's dask token.
• Tests: test_streams.py + test_dynamic.py (259 passed), bokeh/test_plot.py + test_server.py + test_callbacks.py (66 passed, 1 skipped).

Findings

1. Non-LabelledData sources now raise AttributeError (regression)

streams.py:_attach does streams = source._streams unconditionally. Any source that isn't a LabelledData (or has __slots__ / blocks setattr) now blows up at construction:

main:   Tap(source=Foo())  -> OK
branch: Tap(source=Foo())  -> AttributeError: 'Foo' object has no attribute '_streams'

hvplot/geoviews/lumen/panel and every in-tree call site were checked — all sources are HoloViews objects, so real-world impact is low, but AttributeError deep inside _attach is a bad failure mode for user code that worked before. Suggest an explicit guard:

streams = getattr(source, "_streams", _MISSING)
if streams is _MISSING:
    raise TypeError(f"Stream source must be a HoloViews object, got {type(source).__name__}.")

2. The session-destroy leak is only half fixed

The PR body says "outside a server that document is curdoc() … so every plot ever rendered was kept". _unwatch_session only runs on cleanup() or document reassignment — nothing unregisters when a plot is simply dropped. Measured, identical on branch and main:

5 × renderer.get_plot(hv.Curve(...)):
  curdoc destroy callbacks: 5
  live Curve elements: 5      live CurvePlots: 5   (after gc.collect())

So hv.render(obj) / renderer.get_plot(obj) without an explicit cleanup() still pins every plot and its data for the process lifetime — Panel-driven paths are fine because the pane calls cleanup. Worth closing here, since it's the same leak class and the description implies it's gone. Two options:

• Skip registration entirely when the document isn't a real session doc (getattr(doc, "session_context", None) is None) — on_session_destroyed on the global curdoc() never fires anyway; or
• register a weak wrapper (weakref.WeakMethod) so the document doesn't own the plot.

3. Renderer.last_plot semantics change silently

last_plot was added as a debugging affordance ("easily debugging or exporting the last displayed plot", doc/releases.md:4146). It now returns None as soon as the caller drops the plot. In practice it survived every path tried (__call__, html, get_plot) because something else holds the plot — which makes it non-deterministic rather than clearly changed. Fine as a fix, but it deserves a line in the release notes, since a user's renderer.last_plot.state can now be an AttributeError on None.

4. Ownership doesn't follow link=True clones

clone(link=True) propagates _plot_id (dimension.py:683) but not _streams, so stream lifetime is bound to the original source object. If a linked clone outlives its original, the registry lookup finds nothing. This is not a regression (main behaves identically when no subscriber closes the cycle back to the source), but with the leak removed the window is real and previously masked in the cycle case. Consider carrying the list on linked clones — one line in clone, and it makes lifetime match the _plot_id linkage semantics the registry already assumes. At minimum, worth a comment in _attach noting the constraint.

Smaller points

• annotators.py:301 — self._stream.add_subscriber(self._update_object, precedence=0.1) is exactly the bug fixed at 1.05, mirrored: an internal subscriber sitting in the documented user range, so clear("user") drops it. Not fixed here; changing it flips its ordering relative to user callbacks, so it may be deliberate — worth a comment either way.
• plot.py:135-141 — doc.callbacks._session_destroyed_callbacks is private Bokeh API and now reached from more call sites (including via Plot.cleanup for mpl/plotly plots). A getattr(doc.callbacks, "_session_destroyed_callbacks", None) guard would cost nothing.
• _StreamRegistry — __getitem__/get/items/values resolve refs, but pop/setdefault/update/copy are inherited and return raw weakref.ref objects. Stream.registry is effectively public; either override them or document the resolved-vs-raw split (the internal super().get comment is good, extend it).
• plot.py:204 vs bokeh/plot.py:277 — generic Plot.cleanup doesn't null _document while BokehPlot.cleanup does. Harmless now that the hook is dropped, but the asymmetry invites confusion.

Test coverage gaps

Everything added is good; these claimed fixes are untested:

1. Precedence 1.05 — no test that clear("user") preserves _stream_update. It regresses cleanly on main, so it's a cheap, high-value test (dmap + hv.Curve([...]) with a dimensioned stream, then clear("user")).
2. Document reassignment — test_shared_stream_survives_one_session_teardown covers the destroy path, not plot.document = doc2 dropping doc1's hook.
3. __getstate__ pop — no test that an element with _streams pickles, or that its dask token is unaffected by attaching a stream.
4. Stream.registry removal — test_source_remap_releases_previous_source asserts the _streams lists; adding assert points not in Stream.registry would cover unregister's pop-when-empty branch.

[TheoMathurin]

Great! I'm looking forward to seeing this included in a release.

[hoxbro]

Is it possible for you to give this a go before merging?

[TheoMathurin]

It's all good: the subscriber is executed AND the memory is freed!

Thanks a bunch

[philippjfr]

Looking at the code everything looks good to me, but it's hard to reason about so I've also done a bunch of testing.