feat: Support struct schema evolution matching by name

[rui-mo]

The default behavior of the schema evolution for row type is matching by index.
This PR supports matching by name for Parquet and ORC file formats. Missing
subfields are identified by matching the file type and requested type on the
names of subfileds, and 'null' occupies the position of the missing subfields.
Below table summarizes the results for difference cases and file formats.

Column schema	Requested output schema	Parquet result	ORC result
row({"a", "c"})	row({"a", "b", "c"})	row(a_val, null, c_val)	row(a_val, null, c_val)
row({"a", "c"})	row({"b"})	null	row(null)
row({"a", "c"})	row({"b", "d"})	null	row(null, null)
row({"a", "c"})	row({})	null	row()

[netlify]

✅ Deploy Preview for meta-velox canceled.

Name	Link
🔨 Latest commit	1ccda8f
🔍 Latest deploy log	https://app.netlify.com/projects/meta-velox/deploys/692fd2fd9160ce000821310b

[Yuhta]

Requested type is available as getSelector()->getSchemaWithId()->type. We may want to convert it to a type directly in the future, but for now let's not keep 2 copies of the same thing.

[Yuhta]

We need to decide what is the schema evolution strategy we want here. In our data warehouse, columns are not matched by name but by position, so any extra fields added need to be at the end of the children list. This allows column renaming. If we match by name here, we will lose the renaming functionality and this seems quite important in most data warehouse.

[rui-mo]

Thanks for your comment. Does that mean for a row(a, c) struct schema in parquet, the expected output can only be like row(a, c, xxx, ...)? In Spark, there is no such limitation to extra child fields.

[Yuhta]

Yes new subfields can only be appended. So in plain vanilla Spark, field renaming is not supported? There is also a third way to match by field ID (e.g. Iceberg), we need to start draft some design about this to cover all three cases.

[rui-mo]

How does field renaming is conducted in the data warehouse you mentioned? In Spark, for query like select a as b, it adds a projection node with Alias expression after scan.
And what do you suggest for the design, should I added some notes in this PR or something else?

[Yuhta]

With matching by name you need to know all the old field names (a in your query) in all old files, which is not practical in a normal data warehouse. I would suggest we pause this PR for a bit and design the right way to allow matching columns in different ways first.

[rui-mo]

Thanks. That looks good to me. Convert this PR to draft for now.

[rui-mo]

@majetideepak Understood. I will take a look, thanks.

[rui-mo]

@majetideepak @Yuhta The prior refactor has been merged. Would you please take another look? Thanks!

[rui-mo]

Hi @Yuhta @majetideepak, could you please spare some time to continue the review process of this change? Thanks!

[zhouyuan]

@Yuhta gentle ping

[zhouyuan]

@Yuhta gentle ping

[zhztheplayer]

May I understand what's blocking on merging this? @Yuhta

The patch could also remove a blocker against building a Velox Parqet reader for Spark + Delta for another workload.

[rui-mo]

cc: @pedroerp @FelixYBW This is essential for Gluten. I’d be glad to follow up on any additional comments. Thanks!

[majetideepak]

@Yuhta can you please take another look at this PR?

[Yuhta]

This is a very big and risky change and I don't think we can support it without an explicit flag to indicate we want this mode. Let's start by putting all the change behind the a flag, and also breaking down the changes to multiple PRs, along with proper tests for both formats.

[rui-mo]

Thanks @Yuhta. Let me follow up on that.