RFC: DataFrame backend parity for Panel's tabular components via Narwhals

[ghostiee-11]

Summary

Panel already depends on narwhals >=2, and Bokeh already accepts Polars and PyArrow frames as ColumnDataSource input. Panel's tabular components accept neither. I would like to close that gap, and I have done enough analysis to know where the hard part is, which is not rendering.

I am asking for agreement on three design questions before writing anything substantial.

Current state

Run against main today:

import pandas as pd, polars as pl, pyarrow as pa, panel as pn
from bokeh.models import ColumnDataSource

data = {'a': [1, 2, 3], 'b': ['x', 'y', 'z']}
for backend, df in [('pandas', pd.DataFrame(data)),
                    ('polars', pl.DataFrame(data)),
                    ('pyarrow', pa.table(data))]:
    for label, fn in [('bokeh CDS.from_df',    ColumnDataSource.from_df),
                      ('panel Tabulator',      pn.widgets.Tabulator),
                      ('panel DataFrame pane', pn.pane.DataFrame),
                      ('panel Perspective',    pn.pane.Perspective)]:
        try:
            fn(df); print(f'OK    {label:22} {backend}')
        except Exception as e:
            print(f'FAIL  {label:22} {backend}  {type(e).__name__}')

Entry point	pandas	polars	pyarrow	Fails at
ColumnDataSource.from_df	OK	OK	OK
widgets.Tabulator	OK	FAIL	FAIL	widgets/tables.py:757
widgets.DataFrame	OK	FAIL	FAIL	widgets/tables.py:757
pane.DataFrame	OK	FAIL	FAIL	pane/base.py:211
pane.Perspective	OK	FAIL	FAIL	pane/perspective.py:102

Bokeh passes 4 of 4 non pandas cases. Panel passes 0 of 8. Identical on the released 1.8.9, so this is not a recent regression.

PyArrow fails in exactly the same places as Polars, which suggests these are general pandas API assumptions rather than anything Polars specific.

Why this seems worth doing now

• Bokeh completed its side in bokeh#13780 (closed as completed, February 2025). ColumnDataSource._data_from_df already imports narwhals.stable.v1 and branches on is_pandas_like_dataframe, so the serialization layer beneath Panel is already backend neutral.
• narwhals >=2 is already a hard dependency in pyproject.toml, currently used in one file, panel/pane/vega.py.
• The roadmap has a section on adopting Narwhals as the DataFrame compatibility layer.
• Add support for serializing polars DataFrame #7475 landed as an explicit temporary fix converting Polars to pandas, and that conversion is still what happens.
• Support Polars #7812 asks for this directly.

Scope of the coupling

Counting pandas specific operations (.index, .iloc, .loc[, reset_index, MultiIndex, .dtypes), excluding comments and docstrings:

Module	Count
panel/widgets/tables.py	87
panel/pane/perspective.py	22
panel/reactive.py	16

The main finding: read and write need different solutions

Narwhals covers the read path completely. I checked the operations the filter and sort code actually needs, and all of them behave identically on pandas, Polars and PyArrow under narwhals.stable.v2: equality, is_in, range comparisons, str.contains case insensitive, starts_with, ends_with, to_lowercase, multi key descending sort, nulls_last, with_row_index, slicing, schema, and fill_null.

Narwhals does not help with the write path, because it has no mutation API. It is a read and transform layer, and nw.DataFrame exposes no equivalent of .loc[...] = .... Panel's editing model mutates self.value in place, so backend parity for editing needs a small explicit dispatch rather than more Narwhals.

The backends also are not equally writable:

pl.DataFrame({'a': [1, 2, 3]})[0, 'a'] = 99      # works
pa.table({'a': [1, 2, 3]}).column('a')[0] = 99   # TypeError, ChunkedArray does not support item assignment

So I think the honest shape is a capability table rather than uniform parity:

Backend	Render	Filter / sort	Positional write	Label write
pandas	yes	yes	yes	yes, unchanged
polars	yes	yes	yes	not applicable, no index
pyarrow	yes	yes	no, immutable	not applicable

Proposed approach

Two internal modules, kept separate because they have different natures:

panel/_dataframe.py       read, Narwhals backed, pure functions
  is_dataframe, to_narwhals, index_columns, schema,
  filter_frame, sort_frame, slice_rows, to_cds

panel/_dataframe_mut.py   write, explicit backend dispatch
  writable(obj) -> LABEL | POSITIONAL | IMMUTABLE
  set_cell, set_column, concat_rows, replace

Nothing else in Panel would touch pandas directly. panel/pane/vega.py already does something similar on a smaller scale, so I would follow its idioms.

to_cds should mostly be a delegation. Since Bokeh's from_df already handles these backends, Panel's job is to stop doing its own index flattening first, not to write a new serializer.

One compatibility rule I want to state explicitly

BaseTable.add_filter() accepts a user supplied callable that receives the DataFrame and returns either a filtered frame or a boolean mask (tables.py:516-518, type(res) is type(df)). Those callables are user code that expects a concrete frame.

My proposal is that user callables always receive the native frame of whatever backend the user supplied, never a Narwhals wrapper. A pandas user's callable keeps receiving pandas. I would add a test asserting this so it cannot regress.

Questions

1. Is panel.widgets the right place for this? The roadmap deprecates panel.widgets, panel.pane and panel.layout at 3.0 in favour of the Material UI components. If backend parity should be built in panel-material-ui instead, or in whatever the panel.ui table becomes, I would rather know before starting. This is my main question.

2. What should PyArrow editing do? Tables are immutable, so the options are to raise for edits, or to reassign .value with a new frame. The second preserves parity but changes .value identity on every edit, which existing watchers may not expect.

3. Does the staging below look right, with the read path landing on its own first so it is useful even if the write half takes longer?

Proposed staging

Read path first, and it is a complete deliverable on its own:

1. panel/_dataframe.py plus a backend parametrised test matrix, non pandas cases xfail(strict=True). No behaviour change.
2. Index abstraction in the table base class. No behaviour change.
3. Column definitions and dtype mapping via Narwhals schema. No behaviour change.
4. Serialization routed through one function. No behaviour change.
5. Filter, using Narwhals expressions.
6. Sort and pagination slicing.
7. Perspective, pane.DataFrame, docs. Read parity complete, Support Polars #7812 closable.

Then the write path, which is where the risk is:

8. Mutation dispatch, _update_column, _map_indexes.
9. stream() and patch().

Steps 1 to 4 are pure refactors, which should make them fast to review. Each step leaves main green for pandas users and is independently revertible.

On sort specifically: _sort_df is a reimplementation rather than a translation, since it currently sorts with kind='mergesort' and a key= callable that lowercases string columns to match Tabulator's client side ordering, and Narwhals has no sort(key=). I intend to match the current pandas output exactly, including case handling, null placement and tie breaking. If I cannot, I will report the deviation rather than ship approximate ordering.

Out of scope

• Removing pandas as a dependency. That is compat: Remove direct dependency on pandas  #8236, and pandas >=1.2 would stay exactly where it is. This is only about accepting other frame types.
• param.DataFrame. param#975 is not a blocker here, because BaseTable.value is a plain param.Parameter (tables.py:157), so Panel never routes through that validation.
• LazyFrames. Eager frames only.
• Tabulator does not automatically update the screen content when using streaming and remote pagination #7721. That is a pre existing streaming and pagination bug on pandas. I do not want to absorb it into this work, and if the refactor makes it worse I would revert rather than push through.
• Performance. There is no speed claim here. ColumnDataSource.from_df is about 19 ms for 500k rows and Tabulator's paginated path stays under a millisecond regardless of row count, so serialization is not a bottleneck. The case is correctness and ecosystem reach.

I have a reproduction script and the measurements behind all of the above and can share them if useful.

[ghostiee-11]

I started on the read path to test the proposal against the code. Polars and PyArrow frames construct now, six changes in tables.py, all behind a has_index() check so the pandas paths are untouched.

Two corrections to what I wrote above.

index_columns() was the wrong primitive. The code only needs has_index(). The index naming (level_0, the nlevels padding) is Tabulator presentation rather than something generic about a frame, so it stays in tables.py.

The show_index=True synthesis is still open. Today indexes returns nothing for a frame without one regardless of show_index, and I am not convinced synthesising a positional column beats just ignoring the parameter for those backends.

Also worth knowing if anyone else digs in: pa.Table.columns is the column data, not the names, so column names now go through narwhals.