Change FileScanConfig.table_partition_cols from (String, DataType) to Fields

[NGA-TRAN]

Which issue does this PR close?

Closes #7875

Rationale for this change

Currently, FileScanConfig.table_partition_cols has data type Vec<(String, DataType)> to store only columns name and its data type. A column can include many more information such as nullable and extra meta data. Thus, when we convert table_partition_cols to Fields here, all other information of a field will either empty or default.

We want the data type of table_partition_cols a vector of Fields in the first place so when we need to store a Field, we won't lose any information.

FYI: IOx needs this requirement.

What changes are included in this PR?

Replace data type of FileScanConfig.table_partition_cols from Vec<(String, DataType)> to Vec`

Are these changes tested?

Yes

Are there any user-facing changes?

The API to create FileScanConfig needs a vector of Fields for table_partition_cols. Most of the places it is an empty vector means it is not used.

[NGA-TRAN]

@alamb and @crepererum
Can you help review this? I do not have permission to set the label to api-change

[NGA-TRAN]

Here is the key change

[tustvold]

You probably want to use FieldRef not Field

[NGA-TRAN]

And this where we convert table_partition_cols to Field

[alamb]

Thank you @NGA-TRAN -- I think the idea of passing a real Field as the partition column makes a lot of sense and that this PR does it very nicely 👍

I had a few code improvement suggestions, but nothing I think is required to merge this.

Thanks again

[alamb]

I find this name confusing given the three letter abbreviations and I don't think this is common elsewhere in the DataFusion codebase.

How about something like

Suggested change

	fn config_for_proj_with_field_tab_part(
	fn config_for_projection_with_partition_fields(

Or maybe instead you could change config_for_projection to take table_partition_cols: Vec<Field>, and make a function like

/// Convert all 
fn partition_cols(  table_partition_cols: Vec<(&str, DataType)>) -> Vec<Field> {
         table_partition_cols
            .iter()
            .map(|(name, dtype)| Field::new(name, dtype.clone(), false))
            .collect::<Vec<_>>()
}

And then convert the call sites of config_for_projection to be config_for_projection(.., partition_cols(..))

[NGA-TRAN]

I implemented your second suggestion @alamb . Thanks

[NGA-TRAN]

I have addressed all the comments. Thanks @alamb

[alamb]

Thanks @NGA-TRAN

[Dandandan]

Thanks @NGA-TRAN