Release 0.19.0: Polars validation support

[cosmicBboy]

✨ Highlights ✨

📣 Pandera now supports validation of polars.DataFrame and polars.LazyFrame 🐻‍❄️!

You can now do this:

import pandera.polars as pa
import polars as pl


class Schema(pa.DataFrameModel):
    state: str
    city: str
    price: int = pa.Field(in_range={"min_value": 5, "max_value": 20})


lf = pl.LazyFrame(
    {
        'state': ['FL','FL','FL','CA','CA','CA'],
        'city': [
            'Orlando',
            'Miami',
            'Tampa',
            'San Francisco',
            'Los Angeles',
            'San Diego',
        ],
        'price': [8, 12, 10, 16, 20, 18],
    }
)
Schema.validate(lf).collect()

And of course you can do functional validation with decorators like so:

from pandera.typing.polars import LazyFrame

@pa.check_types
def function(lf: LazyFrame[Schema]) -> LazyFrame[Schema]:
    return lf.filter(pl.col("state").eq("CA"))

function(lf).collect()

You can read more about the integration here. Not all pandera features are supported at this point, but depending on community demand/contributions we'll slowly add them. To learn more about what's currently supported, check out this table.

Special shoutout to @AndriiG13 and @FilipAisot for their contributions on the built-in checks and polars datatypes, respectively, and to @evanrasmussen9, @baldwinj30, @obiii, @Filimoa, @philiporlando, @r-bar, @alkment, @jjfantini, and @robertdj for their early feedback and bug reports during the 0.19.0 beta.

What's Changed

• Support polars DataFrames, LazyFrames by @cosmicBboy, @AndriiG13, and @FilipAisot in Support polars DataFrames, LazyFrames #1373
• bugfix: optional columns in polars schema should no longer raise errors when not present by @cosmicBboy in bugfix: optional columns in polars schema should no longer raise errors when not present #1532
• check_nullable does not uselessly compute isna() anymore in pandas backend by @smarie in check_nullable does not uselessly compute isna() anymore in pandas backend #1538
• Polars LazyFrames are validated at the schema-level by default by @cosmicBboy in Polars LazyFrames are validated at the schema-level by default #1534
• Enable from_format_kwargs for dict format by @ektar in Enable from_format_kwargs for dict format #1539
• Convert docs to myst by @cosmicBboy in Convert docs to myst #1542
• fix README(tab to space) by @np-yoe in fix README(tab to space) #1544
• pandas DataFrameModel accepts python generic types by @cosmicBboy in pandas DataFrameModel accepts python generic types #1547
• Backend registration happens at schema initialization by @cosmicBboy in Backend registration happens at schema initialization #1548
• do not format if test is not necessary by @mattB1989 in do not format if test is not necessary #1530
• Register default backends when restoring state by @alkment in Register default backends when restoring state #1550
• Bump actions/setup-python from 4 to 5 by @dependabot in Bump actions/setup-python from 4 to 5 #1452
• fix: prevent environment pollution when importing pyspark by @sam-goodwin in fix: prevent environment pollution when importing pyspark #1552
• use rst to speed up api docs generation by @cosmicBboy in use rst to speed up api docs generation #1557
• Add _GenericAlias.call patch by @cosmicBboy in Add _GenericAlias.__call__ patch #1561
• support typeguard < 3 for better compatability by @cosmicBboy in support typeguard < 3 for better compatability #1563
• Add parse function to DataFrameModel in Add parse function to DataFrameModel #1181
• localize GenericAlias patch to DataFrameBase subclasses by @cosmicBboy in localize GenericAlias patch to DataFrameBase subclasses #1571
• Bump idna from 3.4 to 3.7 by @dependabot in Bump idna from 3.4 to 3.7 #1569
• docs: fix typo in env var name by @alekseik1 in docs: fix typo in env var name #1562
• polars: fix element-wise checks, register backends by @cosmicBboy in polars: fix element-wise checks, register backends #1572
• remove pytest ignore on modin, dask. pyspark tests with pandas >= 2 by @cosmicBboy in remove pytest ignore on modin, dask. pyspark tests with pandas >= 2 #1573
• make sure check name is propagated to error report by @cosmicBboy in make sure check name is propagated to error report #1574
• update ci to run pyspark, modin, dask with pandas >= v2 by @cosmicBboy in update ci to run pyspark, modin, dask with pandas >= v2 #1575
• use sphinx-design instead of sphinx-panels by @cosmicBboy in use sphinx-design instead of sphinx-panels #1581
• Update bug_report.md by @philiporlando in Update bug_report.md #1585
• bugfix: polars column core checks now return check output by @cosmicBboy in bugfix: polars column core checks now return check output #1586
• make pandera.typing.Series[TYPE] error in polars DataFrameModel more readable by @cosmicBboy in make pandera.typing.Series[TYPE] error in polars DataFrameModel more readable #1588
• implement timezone agnostic polars_engine.DateTime type by @cosmicBboy in implement timezone agnostic polars_engine.DateTime type #1589
• fix pyspark import error by @cosmicBboy in fix pyspark import error #1591
• fix pyspark tests when run on full test suite by @cosmicBboy in fix pyspark tests when run on full test suite #1593
• Bugfix/1580 by @cosmicBboy in Bugfix/1580 support list[str] in DataFrameModel #1596
• Set pandas_io.from_frictionless_schema to use a raw string for docs by @mark-thm in Set pandas_io.from_frictionless_schema to use a raw string for docs #1597
• Add a generic Series type for polars by @baldwinj30 in Add a generic Series type for polars #1595
• Add StructType and DDL extraction from Pandera schemas by @filipeo2-mck in Add StructType and DDL extraction from Pandera schemas #1570
• Clean up typing for pandas GenericDtype by @cosmicBboy in Clean up typing for pandas GenericDtype #1601
• Adding warning for unique in pyspark field and a test showing the issue as well as config when it works. by @zippeurfou in Raise SchemaInitError when for unique field is specified in pyspark field and a test showing the issue as well as config when it works. #1592
• bugfix/1607: coercion error should correctly report relevant failure cases by @cosmicBboy in bugfix/1607: coercion error should correctly report relevant failure cases #1608
• Create a common DataFrameSchema class, update mypy used in pre-commit by @cosmicBboy in Create a common DataFrameSchema class, update mypy used in pre-commit #1609
• Dataframe column schema by @cosmicBboy in Dataframe column schema #1611
• bugfix: column-level coercion is properly implemented by @cosmicBboy in bugfix: column-level coercion is properly implemented #1612
• update docs for polars by @cosmicBboy in update docs for polars #1613
• fix: properly coerce dtypes for columns with regex=True by @tesslinden in fix: properly coerce dtypes for columns with regex=True #1602
• rewrite Check class docstrings to remove pandas assumption by @cosmicBboy in rewrite Check class docstrings to remove pandas assumption #1614
• add tests for polars decorators by @cosmicBboy in add tests for polars decorators #1615

New Contributors

• @smarie made their first contribution in check_nullable does not uselessly compute isna() anymore in pandas backend #1538
• @ektar made their first contribution in Enable from_format_kwargs for dict format #1539
• @np-yoe made their first contribution in fix README(tab to space) #1544
• @alkment made their first contribution in Register default backends when restoring state #1550
• @sam-goodwin made their first contribution in fix: prevent environment pollution when importing pyspark #1552
• @alekseik1 made their first contribution in docs: fix typo in env var name #1562
• @philiporlando made their first contribution in Update bug_report.md #1585
• @mark-thm made their first contribution in Set pandas_io.from_frictionless_schema to use a raw string for docs #1597
• @baldwinj30 made their first contribution in Add a generic Series type for polars #1595
• @zippeurfou made their first contribution in Raise SchemaInitError when for unique field is specified in pyspark field and a test showing the issue as well as config when it works. #1592
• @tesslinden made their first contribution in fix: properly coerce dtypes for columns with regex=True #1602

Full Changelog: v0.18.3...v0.19.0

---

This discussion was created from the release Release 0.19.0: Polars validation support.