`_dispatch_msgs` write task leaks the `Document` on a stalled/half-open websocket (never cancelled on session destroy)

[SimonHeybrock]

Note: This bug was found by an AI agent (Claude Code) as a drive-by while debugging another issue. Reproduction and analysis were also performed by the agent. I reviewed the below but do not have enough in-depth knowledge to truly say whether it is correct beyond the superficial.

ALL software version info

• Panel: 1.9.2 (same code shape on main at time of writing)
• Bokeh: server WSHandler.write_lock (tornado.locks.Lock)
• Tornado: locks.Lock / BoundedSemaphore
• Python: 3.11
• OS: Linux

Description of expected behavior and the observed behavior

When Panel must defer a document patch because a connection's websocket write_lock is held, panel.io.document.schedule_write_events starts an async task _dispatch_msgs(doc) that re-schedules itself every 10 ms until the lock frees (panel/io/document.py:167-204). The scheduled Task is stored in the module-global _write_tasks list and the coroutine keeps a strong reference to doc.

If the in-flight write never resolves — as on a half-open / blackholed TCP connection where the peer is silently gone and the socket neither completes nor promptly errors — the lock stays held and _dispatch_msgs re-arms indefinitely. Because nothing cancels _write_tasks (and nothing clears _WRITE_MSGS / _WRITE_BLOCK) on session/document teardown, the task:

1. Pins the session's Document (and its model graph and views) in memory, defeating the WeakKeyDictionary storage the _WRITE_* state relies on for cleanup.
2. Survives _destroy_document — panel/io/document.py:213-261 clears state's weak dicts and stops periodic callbacks but never touches _write_tasks or the module-global _WRITE_* dicts, and _write_tasks is never cancel()-ed anywhere in the module.

Expected: on session/document destroy, any pending _dispatch_msgs task for that document is cancelled and its _WRITE_MSGS / _WRITE_BLOCK / _WRITE_FUTURES entries cleared, so the Document becomes collectable and the ~100 Hz task stops.

Observed: the Document stays alive after destroy + gc.collect(), and a self-rescheduling task keeps running.

Mechanism (file:line)

• _dispatch_write_task keeps a strong ref to the task: _write_tasks.append(task) (panel/io/document.py:161-163); the add_done_callback(_cleanup_task) only removes a task once it completes.
• _dispatch_msgs defers any connection whose lock is held (socket.write_lock._block._value == 0, BoundedSemaphore value 0 = held) into remaining, then await asyncio.sleep(0.01); _dispatch_write_task(doc, _dispatch_msgs, doc) (panel/io/document.py:178-204). No termination is tied to connection/document liveness.
• _destroy_document does not cancel _write_tasks nor clear _WRITE_MSGS/_WRITE_BLOCK/_WRITE_FUTURES (panel/io/document.py:213-261).

The lock is held by Bokeh's own WSHandler.write_message(..., locked=True) (bokeh/server/views/ws.py:290-298), which awaits super().write_message inside with await self.write_lock.acquire():; a never-resolving write leaves it held.

Clean close is fine: the write raises WebSocketClosedError, the lock releases, and the next cycle hits ws_conn.is_closing() in dispatch_tornado → empty futures → loop terminates and clears _WRITE_BLOCK.

Complete, minimal, self-contained example (no real network)

Holds the lock directly via a fake connection and shows the Document surviving destroy + GC; a lock-free control terminates cleanly and is collected.

import asyncio, gc, weakref
from functools import partial
import tornado.locks
from bokeh.document import Document
import panel.io.document as pdoc

class P:
    def create(self, msgtype, events): return object()
class S:
    def __init__(self, held):
        self.write_lock = tornado.locks.Lock()
        if held: self.write_lock._block._value = 0   # 0 == lock held (in-flight write)
        self.ws_connection = type("W", (), {"is_closing": lambda s: True})()
class C:
    def __init__(self, held): self._socket, self.protocol = S(held), P()

pdoc.extra_socket_handlers[S] = lambda conn, msg=None: []   # no-op for the lock-free control

def destroy_like_panel(doc):
    doc.destroy = partial(pdoc._destroy_document, doc)       # io/application.py:176
    doc.destroy(None)

async def leak():
    base = len(pdoc._write_tasks)
    doc, conn = Document(), C(held=True); ref = weakref.ref(doc)
    pdoc.schedule_write_events(doc, [conn], [object()])
    await asyncio.sleep(0.06)
    assert len(pdoc._write_tasks) == base + 1               # task keeps re-arming
    assert doc in pdoc._WRITE_MSGS                           # weak entry re-populated each cycle
    destroy_like_panel(doc); del doc, conn
    gc.collect(); await asyncio.sleep(0.03); gc.collect()
    print("LEAK - document alive after destroy+gc:", ref() is not None)   # True

async def clean():
    base = len(pdoc._write_tasks)
    doc, conn = Document(), C(held=False); ref = weakref.ref(doc)
    pdoc.schedule_write_events(doc, [conn], [object()])
    await asyncio.sleep(0.06)
    assert len(pdoc._write_tasks) == base                    # loop terminated
    destroy_like_panel(doc); del doc, conn; gc.collect()
    print("control - document collected:", ref() is None)    # True

asyncio.run(leak()); asyncio.run(clean())

Output (Panel 1.9.2):

LEAK - document alive after destroy+gc: True
control - document collected: True

Severity / scope

Per stalled connection: a ~100 Hz asyncio task (allocating a Task + Future every 10 ms) plus a retained Document and its model graph, for as long as the in-flight write stays pending. On a real half-open TCP socket that is typically the kernel's retransmission window (often minutes) after which the write errors and the loop self-terminates; it becomes effectively permanent only when keepalive/ping is disabled or retransmit is unbounded. Multiple concurrent stalls multiply the CPU cost, and the pending tasks also complicate clean server shutdown. This matches a realistic deployment scenario (a long-lived dashboard whose client drops its network/VPN without closing the socket).

Suggested fix

In _destroy_document (or the session-destroy path), cancel any _write_tasks associated with the document and clear its _WRITE_MSGS / _WRITE_BLOCK / _WRITE_FUTURES entries. Alternatively, bound _dispatch_msgs by connection liveness (e.g. stop re-arming when ws_connection.is_closing() or the connection is no longer subscribed). Note PR #7641 cancelled the refresh-token task in state.py but did not touch _write_tasks.

[philippjfr]

As always @SimonHeybrock, I really appreciate your detailed investigations of these hairy issues.

I've pushed a fix in #8623. Was hoping to release soon but if you get a chance to test that would be super.

[SimonHeybrock]

Thanks for the quick fix!

Unfortunately, I think I probably won't be able to test easily before a release, because we only found this looking at some multi-day logs in production deployment.

[philippjfr]

Okay, no problem, please reopen if you uncover anything with 1.9.3.