hold() leaves events queued on the Document when the session is not connected

[SimonHeybrock]

ALL software version info

Software Version Info

Panel: main @ c97e8098
Bokeh: 3.9.1
Python: 3.11
OS: Linux

Description of expected behavior and the observed behavior

When a hold() block exits while the Document is not connected, the finally chain in panel/io/document.py takes

elif not state._connected.get(doc):
    doc.callbacks._hold = None

which clears the hold policy but leaves doc.callbacks._held_events populated. Bokeh's unhold() returns early when _hold is None, so nothing drains them there. The events sit on the Document until some later, unrelated hold()/unhold() pair flushes them — by which point the session may be connected, and they are dispatched well outside the context they were recorded in.

Expected: leaving a hold() block leaves nothing queued behind it, whether the events are dispatched or deliberately discarded.

Complete, minimal, self-contained example code that reproduces the issue

import time

import panel as pn
import requests

from panel.io.state import state

info, docs = {}, []

def app():
    md = pn.pane.Markdown("initial")
    docs.append(state.curdoc)

    def before_ready():
        doc = state.curdoc
        with pn.io.hold():
            md.object = "changed"
        info['connected'] = state._connected.get(doc)
        info['hold'] = doc.callbacks.hold_value
        info['queued'] = [type(e).__name__ for e in doc.callbacks._held_events]

    state.curdoc.add_next_tick_callback(before_ready)
    return md

pn.serve(app, port=5099, show=False, threaded=True)
time.sleep(1)
requests.get("http://localhost:5099")
time.sleep(1)

print("on exiting the hold:", info)
print("still queued afterwards:", [type(e).__name__ for e in docs[0].callbacks._held_events])

Output:

on exiting the hold: {'connected': None, 'hold': None, 'queued': ['SessionCallbackAdded']}
still queued afterwards: ['SessionCallbackAdded']

The hold is gone but the event it collected is not, and nothing on the Document will drain it until an unrelated hold happens to.

The same applies to events queued by a hold() inside the app callable. Those currently take the threaded branch instead, because state._thread_id is still unset while the app is being built (#8691); with that fixed, a build-time hold strands TitleChangedEvent and RootAddedEvents, which are then flushed by the first hold() after the session connects. Build-time holds behaved that way before #8619 reordered this chain, so this is long-standing rather than new — but it is not obvious to me whether the not-connected behaviour is deliberate, or whether the intent was for the events to be discarded outright.

Stack traceback and/or browser JavaScript console output

None — this is silent.

Screenshots or screencasts of the bug in action

None. I have not established what a connected client actually sees when these events are replayed; the environment I reproduced this in blocks the CDN, so I could not drive a real browser against it. Reporting it on the server-side behaviour alone, since events outliving their hold looks unintended regardless of the client-visible impact.

• I may be interested in making a pull request to address this

---

_{Investigation assisted by Claude Code + Opus 5; the reproducer above was run and verified.}

[ebarkhordar]

On whether the not-connected branch is deliberate: it is, and the intent was to defer rather than to discard.

The branch arrived in 701e698b (#7972, "Do not dispatch events on (un)hold when Document is not initialized"), whose description gives the reason: "The hold decorator should not try to dispatch events when the Document is not yet fully initialized (since this likely leads to the dreaded pending writes error)." At that point the test was state._loaded; 3f0cf119 (#8101) swapped it for state._connected, and 905b44d9 (#8619) reordered the chain around it while leaving the branch itself untouched. So doc.callbacks._hold = None is chosen over doc.unhold() on purpose, precisely because unhold() would dispatch.

What seems to be missing is the other half of that deferral. state._on_load sets _connected[doc] = True (panel/io/state.py:454) and then either returns early or runs the onload callbacks; it never unholds, so the events that were parked waiting for the connection survive the moment they were waiting for. The Pyodide path does the opposite and calls pydoc.unhold() immediately before state._connected[pydoc] = True (panel/io/pyodide.py:393-395).

Your reproducer still holds at f1c0bb4c. Clean python:3.11-slim container, --network none, panel 1.9.3 wheel for the compiled assets with the checkout shadowing it on PYTHONPATH (panel.io.document.__file__ printed as /src/panel/io/document.py):

next-tick hold : {'connected': None, 'hold_after': None, 'queued_after': ['SessionCallbackAdded']}
queued on the Document at the end: ['SessionCallbackAdded']
hold_value at the end: None

One note on the build-time half, since #8693 landed after you filed: state._thread_id is now set before the app callable runs, so a hold() there takes the non-threaded path as you predicted. I could not get it to strand anything, though. With the hold wrapped around a pane mutation inside the callable, _held_events came back empty on exit, because the TitleChangedEvent and RootAddedEvents you name are emitted after the callable returns rather than inside it. If you have a placement that does strand them, that would be worth adding here, since it is a different entry into the same branch.

_{Investigation assisted by Claude Code + Opus 5; the commands and output above were run and verified.}

[SimonHeybrock]

Thanks, that makes the intent unambiguous.

Your framing is better than mine: the bug isn't that hold() leaks, it's that nothing completes what was deferred. _on_load is where the condition the events were parked on becomes true, and it's the one path that doesn't act on it.

On the build-time half: you are right, and I withdraw it. _eval_panel calls the app callable and only then runs _modify_doc, so the title and root events are emitted after the callable has returned; a hold() inside it has nothing attached to the Document yet to collect. That claim was extrapolation from the pre-#8619 behaviour.

One caveat [from Claude] on mirroring the Pyodide path on the server, which may be why the other half was never written: _schedule_on_load submits _on_load to state._thread_pool when config.nthreads is set, so a bare doc.unhold() there dispatches off the Document's thread — the pending-writes hazard #7972 was avoiding, relocated rather than removed. Pyodide has no thread pool, so it gets away with it. Separately, doc.unhold() drops whatever policy is currently set, including a hold owned by someone else, and takes no _HOLD_LOCK[doc], so it races the threaded unhold in panel/io/document.py. So the completion likely belongs in _schedule_on_load — already a document_ready handler, so on the right thread — or on a next-tick callback, and should go through _HOLD_LOCK.

Severity looks low, since once connected the next unrelated hold() takes the doc.unhold() branch and flushes the strays. Late and out-of-context dispatch rather than lost events, except in a session that never holds again.

[philippjfr]

Thanks for the thoughtful discussion here. As background, the idea was that model changes occurring during startup don't need to be synced, since they'll be picked up anyway when the whole document is serialized. So the intent was indeed that the events would be discarded, but it isn't sufficient as is, and the reason is a bit worse than late dispatch.

That premise only holds if the change reaches the model. Pre-connect, state._unblocked(doc) is False, so Reactive._apply_update schedules the update via doc.add_next_tick_callback, which emits a SessionCallbackAdded. That isn't a DocumentPatchedEvent, so it's stranded by this branch and the callback is never registered. The single SessionCallbackAdded in your reproducer is the pane update itself: md.object stays "initial" on the model at the end, and the full serialization on connect then sends the stale state. Same class of loss for add_periodic_callback, state.execute(schedule=True), and _send_event's scheduled branch.

So the split shouldn't be on DocumentPatchedEvent. SessionCallbackAdded/Removed have to be dispatched, and so does MessageSentEvent, which is a patched event but carries no model state (custom bokeh_events, ipywidgets_bokeh comms) and can't be regenerated. On CDS stream/patch: _stream/_patch write straight through to the underlying dict, so .data is locally correct and the rows do survive a full serialize; those events are transport hints rather than the state itself.

The threading side is where this gets messier, and it cuts against putting the completion in _on_load. The threaded branch never checks _connected, so it unconditionally schedules doc.unhold() on a next tick: the discard policy isn't implemented there at all, and the dispatch it does instead is the pending-writes hazard #7972 was avoiding, reached by a different door. With config.nthreads that's the common path, since _schedule_on_load submits _on_load to the thread pool. Note that branch already has the _hold = None / add_next_tick_callback / _hold = policy dance for exactly this trap, so that the SessionCallbackAdded for its own _unhold isn't swallowed.

I'd rather resolve this at hold exit than complete it at connect time: drain under _HOLD_LOCK[doc], clear _hold before dispatching so events emitted by the dispatch aren't re-collected, dispatch the non-recoverable ones, drop the rest, and apply the same split in the threaded branch so the two agree. Then nothing is parked and there's no second drain site to get right.

[ebarkhordar]

You wrote that branch, so this settles the one thing I answered wrong: I read #7972's description as deferring the dispatch, and told @SimonHeybrock the intent was to defer rather than discard. It was discard, and the premise behind it is not in the commit message.

I also left the SessionCallbackAdded in the reproducer unexplained instead of tracing it back to _apply_update, which is the part that actually matters here. Thanks for laying the resolution out in this much detail.

_{Written with Claude Code + Opus 5.}

[philippjfr]

@SimonHeybrock, I've now merged that fix. I'd still appreciate if you could give that a spin and confirm it addresses the issue(s).

[SimonHeybrock]

I can confirm this works with our real app.

Details of verification ran with Claude

Verified end-to-end, both halves. Panel checkout and our tree are back to clean.

How I verified

A/B of the same scripted session on f1c0bb4c (your #8693, pre-fix) vs dc785972 (#8698), plus upstream/main: real Bokeh server + real Chromium session (bundled assets, CDN blocked), a hold() inside a pre-connect next-tick callback that mutates a Markdown pane and emits a MessageSentEvent, with a doc.on_change recorder to see what actually gets dispatched. Script: .scratch/verify_8692.py.

	at hold exit (pre-connect)	after connect
f1c0bb4	queued ['SessionCallbackAdded', 'MessageSentEvent'], dispatched nothing	flushed late by an unrelated hold
dc78597	queued []; SessionCallbackAdded dispatched immediately; MessageSentEvent parked in _UNCONNECTED_EVENTS	MessageSentEvent dispatched, _UNCONNECTED_EVENTS empty

Pane update reached the model and the DOM in both, and no new browser console errors. Caveat worth stating in the reply: I could not demonstrate permanent loss pre-fix — in my reproducer a later hold flushed the strays, so the visible difference is determinism, not a repaired render.

Our workaround

Yes: ReductionApp.create_layout (src/ess/livedata/dashboard/dashboard.py:246) takes doc.hold('combine') around the whole build and unholds in finally, so widget pn.io.hold() calls nest instead of each leaking one. Guarded by tests/dashboard/layout_build_test.py.

I backported both commits onto installed panel 1.9.3 (they apply cleanly) and removed the workaround:

• stock 1.9.3, no workaround — real server logs hold_after_build=combine: the leak is live in our app today. Unit test fails.
• 1.9.3 + both fixes, no workaround — hold_after_build=None. Unit test passes; full browser suite 16 passed.

(First browser run against the ../panel checkout on PYTHONPATH failed 12/16 — stale JS bundle, mdc is not defined / Notyf is not defined. Environment artifact, not the fix; hence the backport-onto-1.9.3 approach.)

Our browser suite passes with the workaround removed on stock panel too, so it doesn't catch the leak — only the unit test does.

Recommendation

Keep the workaround until we can bump the panel pin (>=1.9.3) to the release carrying #8693 + #8698, then delete it together with layout_build_test.py's premise. Want me to draft the confirmation comment for the issue?

Verification script

```python """Verify #8692: hold() exiting before the session connects.

Runs a real server + real browser session (bundled assets, no CDN) and reports
what a pre-connect hold leaves behind and whether the update it collected
actually reaches the client.
"""

from future import annotations

import sys
import time

import panel as pn
from bokeh.document.events import MessageSentEvent
from panel.io.state import state

PORT = int(sys.argv[1]) if len(sys.argv) > 1 else 5099

def deferred_events(doc) -> list[str]:
try:
from panel.io.document import _UNCONNECTED_EVENTS
except ImportError:
return [""]
return [type(e).name for e in _UNCONNECTED_EVENTS.get(doc, [])]

report: dict = {}
docs: list = []
panes: list = []
dispatched: list[str] = []

def app():
md = pn.pane.Markdown("initial", css_classes=["probe"])
doc = state.curdoc
docs.append(doc)
panes.append(md)
doc.on_change(lambda event: dispatched.append(type(event).name))

def before_ready():
    with pn.io.hold():
        md.object = "changed"
        doc.callbacks.trigger_on_change(
            MessageSentEvent(doc, "probe-msg", {"n": 1})
        )
        report["inside_hold_queued"] = [
            type(e).__name__ for e in doc.callbacks._held_events
        ]
    report["connected_at_exit"] = state._connected.get(doc)
    report["hold_value_at_exit"] = doc.callbacks.hold_value
    report["queued_at_exit"] = [
        type(e).__name__ for e in doc.callbacks._held_events
    ]
    report["deferred_at_exit"] = deferred_events(doc)
    report["dispatched_at_exit"] = list(dispatched)

doc.add_next_tick_callback(before_ready)
return md

pn.serve(app, port=PORT, show=False, threaded=True)
time.sleep(2)

from playwright.sync_api import sync_playwright # noqa: E402

with sync_playwright() as p:
browser = p.chromium.launch(executable_path="/usr/bin/chromium")
page = browser.new_page()
console: list[str] = []
page.on('console', lambda m: console.append(f'{m.type}: {m.text}') if m.type in ('error', 'warning') else None)
page.on('pageerror', lambda e: console.append(f'pageerror: {e}'))
page.goto(f"http://localhost:{PORT}", wait_until="load")
page.wait_for_selector(".probe", timeout=30000)
time.sleep(3)
dom = page.locator(".probe").first.evaluate(
"""el => {
const walk = node => {
let text = node.shadowRoot ? walk(node.shadowRoot) : '';
for (const child of node.childNodes) {
if (child.nodeType === 3) text += child.textContent;
else if (child.nodeType === 1 && child.tagName !== 'STYLE') text += walk(child);
}
return text;
};
return walk(el).trim();
}"""
)
browser.close()

doc = docs[0]
md = panes[0]
models = list(md._models.values())
model = models[0][0] if models else None

print(f"\n=== panel {pn.version} from {pn.file} ===")
print("=== at hold exit (pre-connect) ===")
for key in (
"connected_at_exit",
"hold_value_at_exit",
"inside_hold_queued",
"queued_at_exit",
"deferred_at_exit",
"dispatched_at_exit",
):
print(f" {key:22} {report.get(key)}")

print("\n=== after the browser connected ===")
print(f" connected {state._connected.get(doc)}")
print(
f" still queued "
f"{[type(e).name for e in doc.callbacks._held_events]}"
)
print(f" still deferred {deferred_events(doc)}")
print(f" all dispatched events {dispatched}")
print(f" pane.object {md.object!r}")
print(f" model.text {getattr(model, 'text', None)!r}")
print(f" DOM text in browser {dom!r}")
print(f" browser console {console}")

sys.stdout.flush()
import os
os._exit(0)

</details>

[philippjfr]

Thanks for confirming, I'll cut a 1.9.4 this week.