DataFrameSchema with column of type pa.Float will raise SchemaError on validating DataFrame with float32 column

[daavidstein]

Bug description
DataFrameSchema with column of type pandera.Float will raise SchemaError on validating DataFrame with float32 column

Minimal Failing Example

schema = DataFrameSchema(

    columns={
        "A": Column(
              pa.Float,
        ),

    },
)
df =pd.DataFrame({"A":[0.1]}, dtype="float32")
schema.validate(df)
#SchemaError: expected series 'A' to have type 'float64', got 'float32'

Expected behavior

Given that the pandera documentation states that pandera.Float corresponds to numpy float the expectation is that a SchemaError is not raised for columns with type float16, float32, float64, or float128

Additional context

Our dataset columns go through a transformation from float128 to float32 and we would like to use the same schema for validation before and after the transformation.

[spencervoorend]

Thanks for raising this, been encountering it myself.

[jeffzi]

Technically the documentation is correct, numpy.float_ is an alias for the default float size. Pandas behaves similarly.
Pandera follows pandas' defaults. On my system:

import numpy as np
import pandas as pd

print(np.float_)
#> <class 'numpy.float64'>
print(pd.Series([1.0], dtype="float").dtype)
#> float64

At the moment, Pandera only does strict dtype validation. Perhaps the documentation should be clearer?

That said, I do think it would be helpful to be allow size-agnostic float, int and complex dtypes. For context, we are finalizing a refactor of how pandera handles dtypes (#369), and it is now a good time to introduce that kind of feature.

I see multiple ways to go about it:

1. Make pandera.Int, pandera.Float and pandera.Complex size-agnostic, strict checking can be done with specific dtypes such as pandera.Float64. However, that would break current behavior.
2. Add a casting argument to pandera.Column similar to numpy.can_cast(). I tested this approach during the design of the new dtype system (see comment here). The default would be 'no', i.e current behavior. The behavior @daavidstein requested would be achieved with casting='same_kind'.

I favor 2. because it's more explicit and is not a breaking change.

@cosmicBboy Thoughts?

[cosmicBboy]

@daavidstein the approach pandera takes is to (a) err on the explicit side rather than the implicit (a validation library with too much magic seems like an oxymoron 🙂) and (b) to be consistent with pandas/numpy behavior, so pa.Float is basically an alias of float, "float", np.float_, which defaults to float64. This is behavior consistent with numpy and pandas.

In [26]: pd.Series([1,2,3], dtype=float).dtype
Out[26]: dtype('float64')

In [27]: pd.Series([1,2,3], dtype="float").dtype
Out[27]: dtype('float64')

Currently I'd recommend two approaches:

• explicitly specify the datatype bitsize.
• use coerce=True to coerce the data type to the default type.

1. Make pandera.Int, pandera.Float and pandera.Complex size-agnostic, strict checking can be done with specific dtypes such as pandera.Float64. However, that would break current behavior.
2. Add a casting argument to pandera.Column similar to numpy.can_cast(). I tested this approach during the design of the new dtype system (see comment here). The default would be 'no', i.e current behavior. The behavior @daavidstein requested would be achieved with casting='same_kind'.

For for the future state of things, I think casting would make sense, though can_cast might be a better name since it clearly gets across the behavior that the df.column.dtype can be casted into a particular datatype rather than actually being casted (which is the coerce option).