Push predicates into Parquet scan

[roeap]

Is your feature request related to a problem?

The base implementation of a new table scan deliberately omitted pushing predicated into the parquet scan. There are at least two bigger sources of complexity that should be looked at:

• handling logical to physical translation of predicates
• handling row ids

In principle we can also push the deletion vector into the parquet scan which should also be considered carefully when dealing with row ids.

Describe the solution you'd like

AN optimized parquet scan that leverages a large spectrum of datafsuion's parquet implementation.

Describe alternatives you've considered

No response

Priority

Medium - Would be helpful

Additional context

No response

Contribution

• I'm willing to submit a pull request for this feature
• I can help with testing this feature
• I can help with documentation for this feature

[abhiaagarwal]

In principle we can also push the deletion vector into the parquet scan which should also be considered carefully when dealing with row ids.

This would be pretty fascinating to see in practice. In my case, I wanted to do merges on Postgres CDC data but gave up on it precisely because it required rewriting the whole file and blocked my real-time pipeline, opting instead for continuous appends, but with DVs this would make it doable