Make it easy to use dataclass like models using familiar apis

[MarcSkovMadsen]

Superseedes #6892.

Also motivated by me trying to demonstrate that you can just as well use Panel for geospatial applications as Solara by creating apps similar to https://github.com/opengeos/solara-geospatial/tree/main/pages. But currently Panel is harder to use because it requires adding more code for using observer pattern.

Scope: Currently ipywidgets, Pydantic models

Easy to view docs

• How-To ipywidgets
• How-To pydantic
• Reference Guide ipywidgets

Todo

• Create design proposal
• Get initial design feedback
• Update design proposal
• Implement design
• Clean Up
  • names
  • docstrings
  • Add examples to docstring
  • missing tests
• Make functionality covers functionality needed to wrap and share ipywidget as reusable component with Viewer interface.
• Support ipywidgets
• Support pydantic
• Document
  • Update IPyWidget reference notebook
    • remove example files.
  • Create how-to guide for wrapping ipywidget using functionality in panel.ipywidget.
  • Philipp?: Fix broken dev docs: TypeError: Cannot read properties of undefined (reading 'loader').
  • Philipp?: Fix test error. I don't know how to fix the markdown docs test failure.
  • Philipp?: Fix self.param.add_parameter(parameter, param.Parameter())
  • Philipp?: Fix JSON pane updating before model issue.
  • Build documentation and test
• Fix issues identified
  • Fix not working create_parameter of ipywidgets.
  • Fix missing pydantic docs dependency
  • Add create_parameter for pydantic to add appropriate types of parameters.
  • Fix ipywidget tuple length issue
  • Philipp?: Add sqlite as a dependency for running the ipywidgets docs in pyodide
• Test complex use cases with multiple models and sessions
• Final review and update

Maybe later

• Take inspiration from https://github.com/jmosbacher/pydantic-panel/blob/master/pydantic_panel/__init__.py.
  • Most features supported
  • BaseModel field supported
• Take inspiration from https://github.com/LukasMasuch/streamlit-pydantic
• Take inspiration from https://pydf-docs.onrender.com/ or https://community.plotly.com/t/dash-pydantic-form/84435 for inspiration.
• Improve ModelForm.
  • Support construction using instance as an alternative to class.
• Align traitlets type support with pydantic type support.

Promotion

Note: Features have been moved to panel.dataclass module since this video was made. WidgetViewer has been renamed to ModelViewer.

wrapping_ipywidgets.mp4

Design Principles

• I've tried to create small pieces of functionality that build up to the simple to use ModelViewer class and create_rx function such that there are no dead ends and its testable.
• I've tried to use naming conventions that would be general enough to accommodate similar functionality for dataclasses, Pydantic etc. one day.
• I've not gone into creating parameters of similar types to the traits. For now that is something the developer must do if that is needed.

[MarcSkovMadsen]

Hi @philippjfr . Would you take a look at the design spec, i.e. the current files? Thanks.

[codecov]

Codecov Report

Attention: Patch coverage is 94.89144% with 40 lines in your changes missing coverage. Please review.

Project coverage is 81.94%. Comparing base (7267b38) to head (d364cfe).
Report is 7 commits behind head on main.

Files	Patch %	Lines
panel/_dataclasses/base.py	86.71%	19 Missing ⚠️
panel/_dataclasses/ipywidget.py	86.66%	8 Missing ⚠️
panel/_dataclasses/pydantic.py	89.65%	6 Missing ⚠️
panel/tests/dataclasses/test_dataclass.py	99.00%	4 Missing ⚠️
panel/dataclass.py	97.39%	3 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6912      +/-   ##
==========================================
+ Coverage   81.71%   81.94%   +0.22%     
==========================================
  Files         326      331       +5     
  Lines       48082    48861     +779     
==========================================
+ Hits        39292    40040     +748     
- Misses       8790     8821      +31     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[MarcSkovMadsen]

Fixed by adding ModelForm

Our code instantiates pydantic models. Often they don't have default values. Instead initial values are required.

This makes it a bit hard to use our features. Especially for creating forms with validation which a popular use case (c.f. pydantic-panel, streamlit-pydantic, dash-pydantic-form).

For example the below test will currently raise an exception

def test_to_parameterized_no_defaults():
    from pydantic import BaseModel
    class ExampleModel(BaseModel):
        some_text: str
        some_number: int
    
    class ExampleModelParameterized(ModelParameterized):
        _model_class = ExampleModel

    ExampleModelParameterized()

Something like the default_values of the below code is currently required to work around this

import panel as pn
from pydantic import BaseModel
import param
from panel._dataclasses.pydantic import PydanticUtils

pn.extension()

class ModelForm(pn.viewable.Viewer):
    value = param.ClassSelector(class_=BaseModel, allow_None=True)

    submit_button_visible = param.Boolean(default=True, label="Show Submit Button")

    def __init__(self, model_class, submit_button_visible: bool=True, **params):
        self._model_class = model_class
        self._fields = list(model_class.model_fields.keys())
                
        super().__init__(**params)

        fields = model_class.model_fields
        default_values = {field: PydanticUtils.create_parameter(model_class, field).default for field, info in fields.items() if info.is_required()}
        model=model_class(**default_values)
        self._model = model=model_class(**default_values)
        parameters = list(ExampleModel.model_fields.keys())
        parameterized = pn.dataclass.to_viewer(model)
        parameterized.param.watch(self._update_value_on_parameter_change, parameters)
        submit = pn.widgets.Button(name="Submit", button_type="primary", on_click=self._update_value, visible=self.param.submit_button_visible)
        
        self._form = pn.Column(
            pn.Param(parameterized, parameters=parameters),
            submit)

    def _update_value(self, *args):
        self.value = self._model.copy(deep=True)

    def _update_value_on_parameter_change(self, *args):
        if not self.submit_button_visible:
            self.value = self._model.copy(deep=True)

    def __panel__(self):
        return self._form

    @param.depends("value")
    def value_as_dict(self):
        if not self.value:
            return {}
        return self.value.dict()

class ExampleModel(BaseModel):
    some_text: str
    some_number: int
    some_boolean: bool

form = ModelForm(model_class=ExampleModel)
    

pn.Column(form, pn.pane.JSON(form.value_as_dict), form.param.submit_button_visible).servable()

[MarcSkovMadsen]

I added the ModelForm to support the form use case. This is the use case that pydantic-panel, streamlit-pydantic and dash-pydantic-form all support.

Its really a more general request as in #3687. We can solve this is several ways:

• not solve :-)
• document how to solve
• make general Panel form functionality
• add this dataclass specific form functionality.

What do you recommend @philippjfr ?

[MarcSkovMadsen]

Its worth noting that I achieve the form functionality a bit differently than the packages mentioned above. They translate pydantic fields to widgets while I translate pydantic fields to Parameters which Panel can translate to widgets. I.e. in my version we instantiate a Parameterized and then use Param. In the packages they can avoid instantiating a model before the user has filled out the form including required values.

Is my version the right way to go @philippjfr ?

[Coderambling]

Great work! And the form use case should definitely be supported.

[MarcSkovMadsen]

I've tried to achieve almost feature parity with pydantic-panel.

What is missing is Pydantic BaseModel attributes and pandas intervals.

• BaseModels will be added later
• I did not understand the pandas intervals support.

pip install pydantic-panel

import pydantic
import panel as pn
from typing import List
from pydantic_panel.dispatchers import infer_widget
from datetime import datetime, date
import numpy as np
import pandas as pd

pn.extension("tabulator")

class ChildModel(pydantic.BaseModel):
    name: str = "child"

class SomeModel(pydantic.BaseModel):
    name: str = "some model"
    
    child_field: ChildModel = ChildModel()
    date_field: date = date(2024,1,2)
    dateframe: pd.DataFrame = pd.DataFrame({"x": [1], "y": ["a"]})
    datetime_field: datetime = datetime(2024,1,1)
    dict_field: dict = {"a": 1}
    float_field: float = 42
    int_field: int = pydantic.Field(default=2, lt=10, gt=0, multiple_of=2)
    list_field: list = [1, "two"]
    nparray_field: np.ndarray = np.array([1, 2, 3])
    str_field: str = pydantic.Field(default = "to", min_length=2, max_length=10)
    tuple_field: tuple = ("a", 1)

    class Config:
        arbitrary_types_allowed = True # to allow np.array

model = SomeModel()

pydantic_panel_editor = pn.panel(model, sizing_mode="fixed") # Pydantic(model).layout[0]
print(type(pydantic_panel_editor))
panel_editor = pn.Param(pn.dataclass.to_parameterized(model))

pn.Row(
    pydantic_panel_editor,
    panel_editor,
).servable()

[MarcSkovMadsen]

What else should be done here?

[philippjfr]

I'm still fully on board with the aims of this PR but it's simply too large a PR to make it into 1.5.0 at this point.

[maximlt]

Amusingly, given the PR title and module name, I was expecting support for Python's stdlib dataclass :)

[maximlt]

I think we use reactive expressions in the documentation elsewhere (panel, param).

[maximlt]

It's best to add a minimum pin, as the code is likely not to be compatible with all the versions of pydantic. And yes, most of the dependencies there have no pins, it doesn't mean that's the right way :)