Validation of pandas dataframe column of "string[pyarrow]" dtype fails

[ClauPet]

Describe the bug
Using "string[pyarrow]" string alias for both the DataFrame and the DataFrameSchema result in a failing validation. This is very unintuitive.

• I have checked that this issue has not already been reported.
• I have confirmed this bug exists on the latest version of pandera.
• (optional) I have confirmed this bug exists on the main branch of pandera.

Code Sample, a copy-pastable example

df = pd.DataFrame({"col1": ["a", "b"]}, dtype = "string[pyarrow]")
schema = pa.DataFrameSchema(columns={"col1": pa.Column("string[pyarrow]")})
schema.validate(df)
# SchemaError: expected series 'col1' to have type string[pyarrow], got string

Expected behavior

I expect the validation to pass.

Desktop (please complete the following information):

• OS: Windows
• Browser: NA
• Version: 0.24.0

Additional context

My guess is that while in Pandera the "string[pyarrow]" alias points at the actual Arrow datatype, in pandas "string[pyarrow]" maps to pd.StringDtype("pyarrow"). See this from the Pandas documentation (https://pandas.pydata.org/docs/user_guide/pyarrow.html):

The string alias "string[pyarrow]" maps to pd.StringDtype("pyarrow") which is not equivalent to specifying dtype=pd.ArrowDtype(pa.string()). Generally, operations on the data will behave similarly except pd.StringDtype("pyarrow") can return NumPy-backed nullable types while pd.ArrowDtype(pa.string()) will return [ArrowDtype](https://pandas.pydata.org/docs/reference/api/pandas.ArrowDtype.html#pandas.ArrowDtype).

[m-farooqui]

Hi! Thanks for reporting this and for the clear example.

What's happening:
Pandera follows Pandas' distinction between different string dtypes. In Pandas, "string[pyarrow]" (i.e., pd.StringDtype('pyarrow')) and pd.ArrowDtype(pa.string()) are not the same type, and are not treated as equivalent. This is described in the Pandas documentation:

The string alias "string[pyarrow]" maps to pd.StringDtype("pyarrow") which is not equivalent to specifying dtype=pd.ArrowDtype(pa.string()).

Why validation fails:
Pandera expects the DataFrame column dtype and the schema column dtype to match exactly. If your DataFrame uses "string[pyarrow]" and your schema also specifies "string[pyarrow]", this should work, but if there is any mismatch (e.g., one side is "string" or ArrowString), validation will fail.

Code Example:

import pandas as pd
import pandera as pa
from pandera import Column, DataFrameSchema

# This will FAIL because the DataFrame dtype is 'string[pyarrow]' but the schema expects 'string'
df_fail = pd.DataFrame({"col1": ["a", "b"]}, dtype="string[pyarrow]")
schema_fail = pa.DataFrameSchema({"col1": pa.Column("string")})
try:
    schema_fail.validate(df_fail)
except pa.errors.SchemaError as e:
    print("Validation failed as expected:", e)

# This will PASS because both the DataFrame and schema use 'string[pyarrow]'
df_pass = pd.DataFrame({"col1": ["a", "b"]}, dtype="string[pyarrow]")
schema_pass = pa.DataFrameSchema({"col1": pa.Column("string[pyarrow]")})
schema_pass.validate(df_pass)  # No error

Workaround:

• Make sure both your DataFrame and your schema use the same dtype. For example:

  df = pd.DataFrame({"col1": ["a", "b"]}, dtype="string[pyarrow]")
  schema = pa.DataFrameSchema({"col1": pa.Column("string[pyarrow]")})
  schema.validate(df)  # This should pass if both dtypes match

• If you want to use Arrow-backed strings, use pd.ArrowDtype(pa.string()) in both the DataFrame and the schema.

Note:
If you think Pandera should treat these types as equivalent, or if you have a specific use case in mind, please let us know! This could be considered as a feature request for more flexible dtype equivalence or improved error messages.

Hope this helps clarify the behavior!

[cosmicBboy]

This looks like a bug @ClauPet, looking

[m-farooqui]

Thanks for looking into this, @cosmicBboy!

Just to clarify, my investigation found that the current behavior is consistent with how Pandas distinguishes between "string[pyarrow]" (pd.StringDtype("pyarrow")) and pd.ArrowDtype(pa.string()), but I agree it’s not very intuitive for users expecting these to be interchangeable in Pandera schemas.

If there’s anything I can do to help (e.g., provide more examples, test a fix, or clarify use cases), please let me know. I’m happy to assist!

Thanks again for your attention to this!

[ClauPet]

@cosmicBboy, thanks a lot for developing a fix for this so quickly!