Add interfaces for Action RemoveExpiredFiles #10643
Conversation
Co-authored-by: Yufei Gu <yufei@apache.org>
| } | ||
|
|
||
| /** Instantiates an action to remove expired files. */ | ||
| default RemoveExpiredFiles removeExpiredFiles(Table table) { |
There was a problem hiding this comment.
@huaxingao: Can you please provide more background about why this interface is needed and what spark action we are planning to build on top of it? As per me, expireSnapshots action itself can clean the expired data and metadata files. So, I am wondering why this is required?
There was a problem hiding this comment.
This is part of earlier PR #4705. I think this PR was separated out to get consensus on the interface for this action.
There was a problem hiding this comment.
Yes, this PR was separated out from #4705 to get consensus on the interface for this action.
The ExpireSnapshots action does delete the files associated with expired snapshots. However, the RemoveExpiredFiles action serves a different purpose. While ExpireSnapshots removes the expired snapshots from the top-level metadata, some files that should be deleted might still remain due to actions like table copying. Essentially, RemoveExpiredFiles allows the target table to simulate the ExpireSnapshots behavior in the source table, ensuring that the files are cleaned up appropriately.
There was a problem hiding this comment.
a) Let me go through the table copying PR and the changes looks like it is specifically needed only for table copy.
b) Can't the existing remove orphan files clean those left over files after table copy?
c) RemoveExpiredFiles, I definitely want the naming to be different. It is already confusing. Maybe as RemoveTableCopyOrphanFiles
There was a problem hiding this comment.
Yes, the existing remove orphan files can clean those left over files, but it is very expensive, especially for S3.
There was a problem hiding this comment.
Yes, the existing remove orphan files can clean those left over files, but it is very expensive, especially for S3.
Ack. I think instead of separate action, copy table action should atomically clean up its orphan files also. This way it will be less confusing for users and one less action to remember. Lets see what others think.
There was a problem hiding this comment.
I think the name and purpose here are a little confusing for a public API. I would agree with @ajantha-bhat that we should consider keeping this a private part of the copy table action. If we are going to make it public maybe it should be a sub-function of that action, rather than top level api here.
api/src/main/java/org/apache/iceberg/actions/RemoveExpiredFiles.java
Outdated
Show resolved
Hide resolved
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time. |
4 participants
Co-authored-by: Yufei Gu yufei@apache.org
Co-authored-by: Huaxin Gao huaxin_gao@apple.com
This PR adds interfaces for Action RemoveExpiredFiles. We will have another PR to implement the interfaces in Spark 3.4 and Spark 3.5.