Iceberg Table Maintenance: Acceleration Opportunities

[Shekharrajak]

What is the problem the feature request solves?

We should explore and benchmark Iceberg table maintenance operations, while checking for any missing DataFusion implementations that could accelerate Spark Iceberg table maintenance.

Describe the potential solution

No response

Additional context

Iceberg table maintenance using Spark: https://iceberg.apache.org/docs/latest/maintenance/#delete-orphan-files

Medium blog compaction with datafusion + rust : https://medium.com/@amitgil87/building-a-distributed-compaction-engine-for-apache-iceberg-with-rust-datafusion-09faa63c1438

Some old discussion to refer : apache/iceberg-rust#624

[mbutrovich]

It's worth pursuing at some point. We'd need to start by characterizing the writer features in iceberg-rust. We started with the reader first since those gaps seemed more important to close first.

[Shekharrajak]

We'd need to start by characterizing the writer features in iceberg-rust

Can you please share reference links - discussion thread, issue tracker ? I will also look into.

[mbutrovich]

We'd need to start by characterizing the writer features in iceberg-rust

Can you please share reference links - discussion thread, issue tracker ? I will also look into.

I don’t have any. I’d be starting from the same iceberg-rust discussion issue you linked and reading the code.

[Shekharrajak]

Update :
Started a step towards supporting native compaction spark procedure : https://iceberg.apache.org/docs/latest/spark-procedures/#output , internally it uses replace data file action : apache/iceberg-rust#2106

[parthchandra]

@Shekharrajak this is a great idea. do you have any thoughts on how to create sufficiently representative the data sets for these operations?

[Shekharrajak]

@parthchandra , we do not have dataset as such but planning to have benchmarks something like CometIcebergMaintenanceBenchmark - we have bunch of benchmarks already here: https://github.com/apache/datafusion-comet/tree/main/spark/src/test/scala/org/apache/spark/sql/benchmark

2 options :

1. SQL insert queries to create certain number of datafiles, partitions and number of rows
2. Use TPCDS data - the way we have used in some of the benchmark like CometTPCHQueryBenchmark CometTPCDSQueryBenchmark

[parthchandra]

@Shekharrajak I don't know if TPC-DS static data sets would be sufficient to benchmark Iceberg table maintenance operations which typically involve a bunch of updates added periodically leading to a large number of update/delete files.
We could try to use a TPC-DS schema and the TPC-DS datagen utility to create the initial Iceberg tables; but the tricky part is going to be in deciding what kind of updates to the data reflect real world usage.

[Shekharrajak]

Actually we can use the TPCDS and create fragmented tables - insert in batches, such that we have enough number of rows, number of files created to analyse the benchmark.
Last few days I am playing with default spark vs comet native compaction using apache/iceberg-rust#2106 branch and I will share what I found after refactoring.