Add Delete Files To Files metadata table

[szehon-ho]

I am thinking current "files" table suffices to add "delete file" information.

Today, "entries" table already shows some delete file information, though in struct form.
Example entries table "row" for positional delete file:

|status|snapshot_id        |sequence_number|data_file
|1     |3491280865879215816|2              |{1, file:/my_table_path/data/p=1/00129-2-47a2d786-6eaa-45c8-a7df-6bf1303553ec-00001.parquet, PARQUET, 0, {1}, 1, 1798, {2147483546 -> 186, 2147483545 -> 46}, {2147483546 -> 1, 2147483545 -> 1}, {2147483546 -> 0, 2147483545 -> 0}, {}, {2147483546 -> file:/my_table_path/data/p=1/00000-0-2043c59b-7a64-4c51-9a99-fb95f83ac076-00001.parquet, 2147483545 -> }, {2147483546 -> file:/my_table/data/p=1/00000-0-2043c59b-7a64-4c51-9a99-fb95f83ac076-00001.parquet, 2147483545 -> }, null, null, null, null}|

I think the current "files" table has an adequate schema to capture delete files metadata.

|content|   file_path|file_format|spec_id|partition|record_count|file_size_in_bytes|      column_sizes|    value_counts|null_value_counts|nan_value_counts|        lower_bounds|        upper_bounds|key_metadata|split_offsets|equality_ids|sort_order_id|

[szehon-ho]

@coolderli Have you started, or may I work on this one? (I saw on #4005 (comment)).

And any thought on the idea? My only doubt so far is that there is an "all" version of "files" table called "all_data_files", and if we put the delete files there then the name won't be clear anymore, I wonder if we can deprecate it and change it to "all_files", if this is the right approach.

[szehon-ho]

@jackye1995 @rdblue @chenjunjiedada (or sorry if missed someone also involved) wanted to check, has there been some discussion of the whether this is a good way to go (re-use existing data_files table), versus making a new metadata table for delete files? Thanks

I am thinking that the obstacle would be the naming, ie 'files' indicates data+delete files , and its current counterpart 'all_data_files' indicates just data files.

[rdblue]

The all_data_files metadata table was added after we were planning to add delete files, which is why we fixed the name. I'd prefer to add a separate delete_files metadata table since they're reasonably separate. I'm not sure whether we should add deletes to files or not, but I like the ability to read separate metadata tables.

What do you think, @szehon-ho?

[szehon-ho]

Got it, thanks for the background/reply, then it seems all_data_files was designed to only have data files and thus we need all_delete_files.

Was initially thinking the content field in 'files' should be enough to filter out data vs deletes, but I think a delete_files table is also reasonable too to keep it symmetric (content is still useful to distinguish between kinds of deletes).

[szehon-ho]

also fyi @RussellSpitzer @aokolnychyi @flyrain

[aokolnychyi]

Okay, it seems we are in agreement to add these two tables:

• DELETE_FILES
• ALL_DELETE_FILES

I also think we need to do something about files. Right now, it is inconsistent. I see two options:

• Option 1: Add an alias data_files and treat data_files as files.
• Option 2: Show both data and delete files in files and add a separate data_files table.

While Option 2 seems more consistent with what we have today, I am not sure there is a lot of value in it. Users can simply do a union of data_files and delete_files.

Any thoughts?

[rdblue]

I would expect that files produces all data and delete files. I would go with option 2 since it makes the most sense. That said, if we can think of a case where adding delete files to files may cause a problem then we should probably not do it. But I think it is more likely that there is confusion that it does not include delete files currently.

[aokolnychyi]

I believe only Flink heavily uses v2 tables at this point. Since we just added proper support for writing delete files in Spark and still have some gaps in maintenance, I think it should be safe. Let me know if anyone has any concerns.

cc @RussellSpitzer @flyrain @szehon-ho @pvary @rymurr @openinx @stevenzwu

[aokolnychyi]

The current plan:

• Add DELETE_FILES
• Add ALL_DELETE_FILES
• Make FILES report both data and delete files
• Add DATA_FILES

[pvary]

I am fine with having the new metadata tables.

[szehon-ho]

Makes sense to me, thanks guys. (or welcome to more opinions).

For completeness, I think we could also have ALL_FILES at some point (for user to conveniently see total reachability graph of Iceberg)

[rdblue]

I think all_files makes sense.

Unrelated but similar is that we should think about how to manage the reachability set as we add new structures. I think that we may want to have TableMetadata report a reachable set of files that are referenced there. Then we'd only need to union that with all_manifests and all_files.

[RussellSpitzer]

Do we have to do a deprecation warning for FILES? or is it ok to just switch the behavior in 0.14

[aokolnychyi]

Adding ALL_FILES makes sense to me too.

I think that we may want to have TableMetadata report a reachable set of files that are referenced there.

@rdblue, do you mean other json files reported by previousFiles in TableMetadata?

Do we have to do a deprecation warning for FILES? or is it ok to just switch the behavior in 0.14

I don't think we will mark FILES as deprecated. I'd just add delete files to it, which I consider rather a fix than a behavior change. We should give a warning in the release notes, though.