feat(transaction): Add option to check added data files in FastAppendAction

[mitchellciupak]

Which issue does this PR close?

This PR does not close any existing issues. It addresses an optimization opportunity in the fast append workflow.

Use Case

I need to validate data files before committing them to the table. Currently, validate_added_data_files() is called internally during commit(), which means validation occurs on every commit attempt, including retries.

Enhancement

By disabling validate_added_data_files() in the commit method, I can perform validation once before attempting said commit. This allows for commit retries without re-running validation, reducing overhead in retry scenarios.

It's a performance optimization that provides more control over the validation/commit lifecycle.

What changes are included in this PR?

This commit adds an option to the FastAppendAction to disable the validation step snapshot_producer.validate_added_data_files() during commits. This is similar to the option to disable snapshot_producer.validate_duplicate_files()

• Adds an option/flag to FastAppendAction to perform or disable validation of added data files when appending.
• Wiring the option through relevant code paths in append.rs.

The change is implemented in crates/iceberg/src/transaction/append.rs.

Are these changes tested?

These changes have been manually tested outside the test framework. I noticed that the existing with_check_duplicate() method has no test coverage. I'm not sure if either feature is just too small to be considered in scope for the project's test strategy. If helpful, I can add tests for both with_check_duplicate() and the new validate_added_data_files() method here in this PR.

[CTTY]

Hi, thanks for contributing! the change looks good

If helpful, I can add tests for both with_check_duplicate() and the new validate_added_data_files() method here in this PR.

Yes, this would be nice!

[mitchellciupak]

added test_fast_append_with_check_duplicate_false and test_fast_append_with_check_added_data_files_false in crates/iceberg/src/transaction/append.rs! Passing locally and passing in CI!

@CTTY

[mitchellciupak]

@CTTY what does the approval and merge process look like in this project?

[liurenjie1024]

Hi, @mitchellciupak thanks for this pr, but I don't think this is the right way to go. Maybe adding some kind of cache is a better approach to optimize it.

[mitchellciupak]

Hey @liurenjie1024, thanks for taking a look, I was planning on handling the cache prior to the commit call, essentially validating the snapshot once beforehand. The flag would then allow for cheaper retries on the commit method with data file checks disabled.

Different users may implement their caching/validation strategies differently, but I am certainly happy to add a solution for it. Do you have any initial ideas on implementation?