TableProvider via KernelScans

[roeap]

Is your feature request related to a problem?

Currently our datafusion table scans are build without accounting for several transformations that need to be applied on top of the data.

Additionally we can apply some optimisations for specific queries that can be evaluated on metadata only - e.g. some COUNT(*) cases.

Describe the solution you'd like

A new TableProvider implementation that is deigned with the kernel APIs in mind.

Describe alternatives you've considered

migrating the existing providers is quite cumbersome and error prone as they do many things that are now covered in kernel and are largely based on the older internal APIs.

Priority

High - Important for my use case

Additional context

No response

Contribution

• I'm willing to submit a pull request for this feature
• I can help with testing this feature
• I can help with documentation for this feature