Discussion: Switch DataFusion to using arrow2?

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
Datafusion currently relies on the https://github.com/apache/arrow-rs implementation of Apache Arrow. This also means any project that is built on DataFusion is likely to end up using that implementation as well

There has been various talk / discussion / work on switching to arrow2 - https://github.com/jorgecarleitao/arrow2 from @jorgecarleitao

Describe the solution you'd like
A consensus on if we want to switch datafusion to using arrow2

Additional context

• Arrow2 milestone; https://github.com/apache/arrow-datafusion/milestone/3
• PR with more discussion on this issue: Experimenting with arrow2 #68
• Code from @houqp and @yjshen https://github.com/houqp/arrow-datafusion/tree/arrow2_merge
• @Igosuki 's PR: https://github.com/Igosuki/arrow-datafusion/tree/arrow22r
• Roadmap: https://github.com/apache/arrow-datafusion/blob/0b8bffd6410ecdcfa29788f75fbc5ca15242a239/docs/source/specification/roadmap.md#runtime--infrastructure

[alamb]

I believe the current proposal is to make an official arrow branch in datafusion: #68 (comment), which is probably a step towards switching to arrow2

[thinkharderdev]

Will arrow-rs eventually support async file IO? Requiring a synchronous ChuckReader is currently a major limitation in supporting alternate ObjectStores

[hntd187]

I guess, what are the reasons switching would be a bad idea? Like what is the delta between what they both currently provide?

[houqp]

Thank you @alamb for bringing this up!

I believe the current proposal is to make an official arrow branch in datafusion: #68 (comment), which is probably a step towards switching to arrow2

Yes, this aligns with what I have in mind. The official arrow2 branch was proposed so we can close that long running PR and have a centralized location for folks to collaborate on the migration until we are happy with the master merge. If the community is happy with merging directly into master and iterate there, that would work as well.

Will arrow-rs eventually support async file IO? Requiring a synchronous ChuckReader is currently a major limitation in supporting alternate ObjectStores

I believe so. However, we could probably save this work and get it for free with the arrow2 switch.

I guess, what are the reasons switching would be a bad idea? Like what is the delta between what they both currently provide?

IMHO, the main downside is the switch cost and downstream impact. But I think it's a one time cost that's worth paying. I think arrow2 at this point should have covered most of all our needs in datafusion as demonstrated in #68. All unit and integration tests are passing at the moment.

[alamb]

Will arrow-rs eventually support async file IO? Requiring a synchronous ChuckReader is currently a major limitation in supporting alternate ObjectStores

I think someone just needs to put in the effort to make the current arrow-rs implementation async -- I don't know of any reason it can be done, nor do I know of anyone who plans to do so yet.

at this point should have covered most of all our needs in datafusion as demonstrated in #68. All unit and integration tests are passing at the moment.

If this is really true (I haven't checked) it certainly sounds compelling

I like the idea of making an official "arrow2" branch in DataFusion, getting some more 👀 on it, and then propose it as a PR to merge to datafusion master.

@houqp can you make a PR? Would you like me to? @yjshen ?

[houqp]

@houqp can you make a PR? Would you like me to? @yjshen ?

For sure, I can help create that PR 👍

[hntd187]

I guess, what are the reasons switching would be a bad idea? Like what is the delta between what they both currently provide?

IMHO, the main downside is the switch cost and downstream impact. But I think it's a one time cost that's worth paying. I think arrow2 at this point should have covered most of all our needs in datafusion as demonstrated in #68. All unit and integration tests are passing at the moment.

That's good enough for me

[Igosuki]

Btw the latest arrow2 commit that still has RecordBatch is jorgecarleitao/arrow2@ef7937d it would probably be good to use that as a starting point for a temporary arrow2 fork ? That would allow me to integrate the necessary patches for some features such as decimal, without having to switch to having RecordBatch in datafusion.

[jorgecarleitao]

Thank you for considering using arrow2, very excited about this!

To provide some selling points, the primary goals of the repo have been:

• be a place to innovate both on Arrow and on Rust
• be sound and use the least amount of unsafe, pass MIRI checks, have a curated selection of dependencies
• be panic-free on untrusted input to be usable in the context of web servers
• be idiomatic via iterators, std::Vec, MutableArray API, Scalar API, easy to follow structs, etc.
• be performant via simd, trusted len, and fast implementations
• support sync and async IO with APIs that decouple blocking from non-blocking tasks
• be interoperable with other formats such as Arrow, Avro, Parquet and ORC, including mandatory integration tests against corresponding reference implementations
• be modular / easy to compile via feature flags over almost all functionality
• support WASM
• be maintainable via macros to reduce code duplication, user guide, examples and general avoidance of unsafe

atm it is the fastest implementation of Apache Parquet IO and Apache Avro IO that I can find (both read and write), both supporting sync and async executions and implemented in safe Rust (all IO in the crate is unsafe-free).

The crate is under active development, both in volume (~800 commits in a year), and also exploring different ideas, such as

• switched to std::Vec
• Removed dict_id and dict_is_ordered from Field
• Replaced RecordBatch by Chunk
• investigate switching to portable simd
• investigate switching to a safer implementation of flatbuffers
• investigate enable copy on write semantics
• investigate mutable arrays in compute

(which is a major reason why it is 0.X, to allow space to try things out)

The crate has been adopted by Polars, databend, grafana's SDK for Rust and is interoperable with connectorx.

Releases have been happening about once a month (breaking), and on demand for bug fixes. The next is planned for end of this week.

I hope this offers a general idea of what is the crate and where it is heading.

[tustvold]

Will arrow-rs eventually support async file IO? Requiring a synchronous ChuckReader is currently a major limitation in supporting alternate ObjectStores

FWIW it would be relatively straightforward to support async IO within the context of arrow-rs. You need buffered fetching in order to get reasonable IO performance anyway, and so you just do an async fetch into a buffer and then use the sync decoders to decode it. I believe this is what arrow2 is doing anyway?? I quickly cobbled something together showing how this can be done with parquet here.

FWIW I have some optimisations to the arrow-rs parquet reader in flight that yield some pretty significant speedups apache/arrow-rs#1054, apache/arrow-rs#1082. And I am planning to work on dictionary preservation next which should yield orders of magnitude speedups for string dictionaries.

I would personally prefer an approach that sees the great work on arrow2 cherry-picked into arrow-rs, with arrow2 serving as an incubator for new ideas. I am happy to help out with this if there are things people would particularly like to see ported across? The current ecosystem fragmentation is just unfortunate for both users and contributors imo...

[alamb]

I am happy to help out with this if there are things people would particularly like to see ported across?

I have heard lots of excitement about async IO (for parquet, csv, avro, etc) and the performance of those readers.

[alamb]

So my summary of this ticket so far is that the next step is to get a PR up to datafusion with the most up to date code to get arrow2 working. In parallel, I will plan to start some discussions (hopefully later in the week) on the apache arrow dev list about potential ways to getarrow2 unified with arrow-rs

Looking forward to seeing a PR so we can assess how close/far we are from this goal.

[Igosuki]

My branch got merged into the fork so now we only need to address a few remaining issues that break tests.