pandas-select

[jeffzi]

I'm the author of pandas-select, a collection of DataFrame selectors that facilitates indexing and selecting data. The main goal is to bring the power of R's tidyselect to pandas.

I added support for pandera in the latest version:

import pandas as pd
import pandera as pa
from pandas_select import SchemaSelector

df = pd.DataFrame(data=[[1, 2, 3, 4]], columns=["a", "abc", "b", "c"])
print(df)
#>    a  abc  b  c
#> 0  1    2  3  4
schema = pa.DataFrameSchema({"a": pa.Column(int, regex=True, required=False)})

# df = df.pandera.add_schema(schema) # add schema manually
df = schema.validate(df)
df[SchemaSelector(required=False) | "c"]
#>    a  abc  c
#> 0  1    2  4

^{Created on 2021-01-03 by the reprexpy package}

With pandas_select.SchemaSelector, you can filter any columns using attributes of the pandera schema attached to a dataframe.

Generally speaking, I've been thinking about a proposal to extend DataFrameSchema.select_columns and adding arbitrary metadata to a schema. For example, adding a tag for roles (target, feature) in machine learning datasets (inspired from the R library Recipe).

pa.DataFrameSchema(
    {
        "x1": pa.Column(float, role="feature"),
        "x2": pa.Column(float, role="feature"),
        "y": pa.Column(float, role="target"),
    }
)
...
df[SchemaSelector(role="target")] # or built-in pandera method?

^ This is already possible by simply subclassing pandera.Column.

[cosmicBboy]

This is neat! Btw the role metadata seems really nice, I feel like this is a more elegant and flexible solution than the sketch proposal currently in #179, and it extends beyond ML use cases.