Python: add support to optimize, analyze tables and expire snapshots, remove orphan files

[alaturqua]

Feature Request / Improvement

As an pyiceberg user I would like to be able to do following with PyIceberg:

• expire snapshots
• remove orphan files
• optimize tables
• analyze tables

Regards.

Query engine

Other

[Fokko]

Thanks @alaturqua for raising this. It would be cool to add this. Can you elaborate on what you mean by analyze tables?

[alaturqua]

@Fokko

I mean letting it gather statistics. I don‘t know exactly if metadata files are holding statistics.

For example trino has this function:
analyze table_name
https://trino.io/docs/current/connector/iceberg.html#table-statistics

additionally letting it roll back snapshots or registering iceberg tables would be a nice thing as well.

[Fokko]

@alaturqua That's an interesting thought. I think that should be quite straightforward. We could collect all the column metrics, and combine them to show count, null_count, nan_count, min and max.

[Fokko]

@alaturqua which catalog are you using? Rollback of snapshots is also not that much work.

[alaturqua]

@Fokko

We are using hive metastore. But plan to switch to rest catalog, if it supports views in the future.

[alaturqua]

@Fokko

I have a use case, where we copy iceberg table folders from a blob storage location to another.

And updating metadata and file location on meta files is really cumbersome. Being able to update metadata location and file locations respectively plus registering the table via pyiceberg would be great improvement as well.

It can be used for migrations between vendors or storages etc.

[Fokko]

This would be quite straightforward. You would need to implement the _commit_table to update the table properties for the Hive catalog, this is already done for the REST catalog.

If you're interested in contributing, this might be an interesting PR, otherwise I can try to squeeze it in at some point

[onerishabh]

Hey @Fokko, happy to start looking into some features here one by one. Would be great if each task is an issue and has a brief description about how to tackle it. 😃

[netanelm-upstream]

@alaturqua Hi, Can you please also add the rewriteManifests operation?
It can be very useful.

[alaturqua]

Hi @netanelm-upstream,

I created the feature request but unfortunately I do not have time to work on these in the near future.

I was hoping @Fokko would take these into his backlog.

[Fokko]

Thanks everyone for jumping in here.

• Expire snapshots, and optimize table These require write support, which we are working on! This will also include rewriteManifest operations.
• Remove orphan files Is something that could be done today. This would require listing all the files from the table, and also listing all the files from a specific directory. We could do this in the CLI where you get a prompt of the files that need to be deleted.
• Analyze tables We currently have listing capabilities in the CLI, maybe it would be cool to also include the min/max/nulls in there.

I was hoping @Fokko would take these into his backlog.

Yes, I'm happy to, but we first need to get write support in :)

[Fokko]

I've migrated this issue to the new repository: apache/iceberg-python#31 We definitely don't want to lose track of this! 👍