Skip to content

Conversation

@AndrejIring
Copy link
Contributor

Resolve issue #2035.

@codecov
Copy link

codecov bot commented Jun 10, 2025

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.53%. Comparing base (812b2a8) to head (0c5f814).
⚠️ Report is 358 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2036      +/-   ##
==========================================
- Coverage   94.28%   93.53%   -0.76%     
==========================================
  Files          91      135      +44     
  Lines        7013    10496    +3483     
==========================================
+ Hits         6612     9817    +3205     
- Misses        401      679     +278     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Copy link
Collaborator

@deepyaman deepyaman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be good to add a test for the failure case you mentioned in #2035.

Edit: And it looks like it is causing a lot of test cases to fail that weren't previously failing.

@AndrejIring
Copy link
Contributor Author

Sure, I will add a new test and investigate why the tests are failing.

@deepyaman
Copy link
Collaborator

Looks like this isn't passing some places where you have failure_cases of type np.ndarray, I think:

=========================== short test summary info ============================
FAILED tests/pyspark/test_schemas_on_pyspark_pandas.py::test_custom_checks - TypeError: string indices must be integers, not 'str'
FAILED tests/pyspark/test_schemas_on_pyspark_pandas.py::test_schema_model - assert {'float_field', 'AttributeError("\'numpy.ndarray\' object has no attribute \'apply\'")'} == {'float_field', 'd', '-1'}
  
  Extra items in the left set:
  'AttributeError("\'numpy.ndarray\' object has no attribute \'apply\'")'
  Extra items in the right set:
  'd'
  '-1'
  
  Full diff:
    {
  +     'AttributeError("\'numpy.ndarray\' object has no attribute \'apply\'")',
  -     '-1',
  -     'd',
        'float_field',
    }

@AndrejIring AndrejIring force-pushed the bugfix/2035 branch 2 times, most recently from 7221c5d to 45e7d17 Compare June 26, 2025 09:40
@cosmicBboy cosmicBboy changed the title Resolve #2035 bugfix: fix format_vectorized_error_message to properly format nested pyarrow failed cases Jun 26, 2025
AndrejIring and others added 22 commits June 30, 2025 15:01
Signed-off-by: Andrej Iring <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
…nionai-oss#2028)

* bugfix: PANDERA_VALIDATION_ENABLED=False should disable validation

Signed-off-by: Niels Bantilan <[email protected]>

* fix tests

Signed-off-by: Niels Bantilan <[email protected]>

---------

Signed-off-by: Niels Bantilan <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
* Expect Python slice index errors with pandas again

Seen in https://github.com/unionai-oss/pandera/actions/runs/15526426636/job/43706977542?pr=2030

Signed-off-by: Deepyaman Datta <[email protected]>

* Expect Python slice index errors after Python 3.10

Signed-off-by: Deepyaman Datta <[email protected]>

* Fix the version check to not include Python 3.10.x

Signed-off-by: Deepyaman Datta <[email protected]>

---------

Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
* Restore accidentally-deleted use of "breakpoint()"

Signed-off-by: Deepyaman Datta <[email protected]>

* Swap `types-pkg_resources` with `types-setuptools` (unionai-oss#1779)

* Swap `types-pkg_resources` with `types-setuptools`

Signed-off-by: Deepyaman Datta <[email protected]>

* Update the expected error message to name variable

Signed-off-by: Deepyaman Datta <[email protected]>

* Fix expected outputs for pandas-stubs 2.2.2.240807

Signed-off-by: Deepyaman Datta <[email protected]>

* Update CI configuration to filter right pandas ver

Signed-off-by: Deepyaman Datta <[email protected]>

---------

Signed-off-by: Deepyaman Datta <[email protected]>

* Do not test Ibis below py3.9, add DuckDB for tests

Signed-off-by: Deepyaman Datta <[email protected]>

---------

Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
* Test model schema equivalency with optional dtypes

Signed-off-by: Deepyaman Datta <[email protected]>

* Support specifying and validating optional columns

Signed-off-by: Deepyaman Datta <[email protected]>

* Add DuckDB for testing and regenerate requirements

Signed-off-by: Deepyaman Datta <[email protected]>

---------

Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
* Implement minimal `coerce` and corresponding tests

Signed-off-by: Deepyaman Datta <[email protected]>

* Do not exclude 3.9 in Ibis CI, and regenerate reqs

Signed-off-by: Deepyaman Datta <[email protected]>

---------

Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
deepyaman and others added 25 commits June 30, 2025 15:01
Signed-off-by: Deepyaman Datta <[email protected]>

Co-authored-by: cosmicBboy <[email protected]>

Signed-off-by: cosmicBboy <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
* Create element-wise checks automatically for users

Signed-off-by: Deepyaman Datta <[email protected]>

* Mark unsupported scenario with element-wise checks

Signed-off-by: Deepyaman Datta <[email protected]>

---------

Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
* Execute dataframe-level checks on the Ibis backend

Signed-off-by: Deepyaman Datta <[email protected]>

* Remove unnecessary, un-Pythonic array length check

Signed-off-by: Deepyaman Datta <[email protected]>

* Remove bit on separately running data-level checks

Signed-off-by: Deepyaman Datta <[email protected]>

---------

Signed-off-by: Deepyaman Datta <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
* fix generating pydantic core schema for generic types

Signed-off-by: Jarek-Rolski <[email protected]>

* test pytest for typed generic DataFrame

Signed-off-by: Jarek-Rolski <[email protected]>

* fix issue with docs gitactions test

Signed-off-by: Jarek-Rolski <[email protected]>

* fix pytest test_typed_generic_dataframe

Signed-off-by: Jarek-Rolski <[email protected]>

* fix pytest test_typed_generic_dataframe

Signed-off-by: Jarek-Rolski <[email protected]>

* improve changes

Signed-off-by: Jarek-Rolski <[email protected]>

* bug fix

Signed-off-by: Jarek-Rolski <[email protected]>

* fix pydantic tests for pydantic<2

Signed-off-by: Jarek-Rolski <[email protected]>

* add pytest test_typed_dataframe_model_json_schema

Signed-off-by: Jarek-Rolski <[email protected]>

---------

Signed-off-by: Jarek-Rolski <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
Signed-off-by: Andrej Iring <[email protected]>
Copy link
Collaborator

@cosmicBboy cosmicBboy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

@cosmicBboy cosmicBboy merged commit f9b0af5 into unionai-oss:main Jul 2, 2025
201 of 202 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants