API, Spark 4.0: Add create_file_list option to RewriteTablePathProcedure. #13837

slfan1989 · 2025-08-17T03:04:01Z

This is a minor feature improvement. The background is that we are using RewriteTablePathProcedure to convert Hive tables to Iceberg tables, as detailed in #12762. RewriteTablePathProcedure generates a file-list file, and I need to manually clean up this file after each conversion. I understand that the file-list is mainly used to check data integrity, but since it is not essential for metadata, I believe allowing users to decide whether to generate this file would offer greater flexibility.

… file-list generation.

slfan1989 · 2025-08-18T02:21:23Z

@szehon-ho @dramaticlly @manuzhang @huaxingao @nastra Could you help review this pr? This is a minor improvement. Currently, when using the RewriteTablePath stored procedure, a file named file-list is generated in the target path, even though it is unrelated to the actual data. We would like to have the option to skip creating this file. This PR introduces a new option that allows skipping the generation of the file-list file.

We discussed some aspects of this PR in #12844. Can we continue moving this PR forward?

manuzhang · 2025-08-18T03:01:10Z

api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java

+   * @param skipFileList true to skip saving the file list, false to include it
+   * @return this instance for method chaining
+   */
+  default RewriteTablePath saveFileList(boolean skipFileList) {


The method name and the parameter name have opposite meanings.

saveFileList(true) is actually not to save file list.

dramaticlly · 2025-08-18T21:16:06Z

api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java

+    /** count of rewrite metadata files involved, default value is 0 */
+    default int metadataFilesCount() {
+      return 0;


I think rewrite metadata files can be confused with metadata.json files, if this is count of rewritten manifest files. let's call it rewrittenManifestFilesCount? Same wise for rewrittenDeleteFilesCount above.

Also, do we expect other counts might need for path based rewrite? I am wondering how such information can be useful for programatic access in real world scenario

I think rewrite metadata files can be confused with metadata.json files, if this is count of rewritten manifest files. let's call it rewrittenManifestFilesCount? Same wise for rewrittenDeleteFilesCount above.

Your suggestions are very valuable, and I am working on making improvements based on your feedback.

dramaticlly · 2025-08-18T21:20:48Z

...k/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteTablePathSparkAction.java

+    if (skipFileList) {
+      return builder.fileListLocation("").build();


can you return builder.build() instead?

dramaticlly · 2025-08-18T21:27:26Z

...ensions/src/test/java/org/apache/iceberg/spark/extensions/TestRewriteTablePathProcedure.java

  }

+  @TestTemplate
+  public void testRewriteTablePathWithSkipFileList() {


Let's add another test where we rewritten delete and manifest file count is no zero and assert on the procedure output?

dramaticlly · 2025-08-18T21:32:16Z

...k/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteTablePathSparkAction.java

   * </ul>
   */
-  private String rebuildMetadata() {
+  private Result rebuildMetadata() {


I think we shall add a corresponding test in TestRewriteTablePathsAction

nastra · 2025-08-21T08:14:04Z

api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java

+   * @param skipFileList true to skip saving the file list, false to include it
+   * @return this instance for method chaining
+   */
+  default RewriteTablePath skipFileList(boolean skipFileList) {


why not call this createFileList that always defaults to true. Also without reading the javadoc it's not clear what skip actually refers to. Does it skip scanning that file or creating/deleting/x?

Thank you for your question! RewriteTablePathProcedure generates a CSV file, which is saved by default in the target directory. In #12762, I proposed a new migration approach that allows upgrading Hive tables to Iceberg tables in place. Using this method, we have already upgraded more than 2,000 tables.

Currently, after each migration, a file-list.csv file is created in the table’s metadata directory. Since this file is not actually part of the metadata, we need to clean it up manually. I suggest adding a parameter to the procedure so that when it is invoked, we can choose not to generate this file, thereby simplifying the migration process.

I think calling it createFileList is reasonable, and I will optimize this part accordingly.

nastra · 2025-08-21T08:14:12Z

api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java

    /** Name of latest metadata file version */
    String latestVersion();
+
+    /** count of rewrite delete files, default value is 0 */


Suggested change

/** count of rewrite delete files, default value is 0 */

/** count of rewritten delete files, default value is 0 */

nastra · 2025-08-21T08:14:22Z

api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java

+      return 0;
+    }
+
+    /** count of rewrite metadata files involved, default value is 0 */


Suggested change

/** count of rewrite metadata files involved, default value is 0 */

/** count of rewritten manifests involved, default value is 0 */

nastra · 2025-08-21T08:15:01Z

open-api/src/testFixtures/java/org/apache/iceberg/rest/RESTCatalogServer.java

      File tmp = java.nio.file.Files.createTempDirectory("iceberg_warehouse").toFile();
      tmp.deleteOnExit();
-      warehouseLocation = new File(tmp, "iceberg_data").getAbsolutePath();
+      warehouseLocation = new File(tmp, "iceberg_data").toURI().toString();


what error is this causing? Can you please paste a stack trace?

I provided a more detailed explanation in #13882. The issue arises because the path generated by REST uses getAbsolutePath(), while the current unit tests use targetTableDir.toFile().toURI().toString(). Since the two paths do not match, the unit tests fail.

Path file:/var/folders/2k/21gv5vmx6z7dlr1g0_jtm8r80000ks/T/iceberg_warehouse10689209309257803697/iceberg_data/default/table/data/deletes.parquet does not start with /var/folders/2k/21gv5vmx6z7dlr1g0_jtm8r80000ks/T/iceberg_warehouse10689209309257803697/iceberg_data/default/table/ java.lang.IllegalArgumentException: Path file:/var/folders/2k/21gv5vmx6z7dlr1g0_jtm8r80000ks/T/iceberg_warehouse10689209309257803697/iceberg_data/default/table/data/deletes.parquet does not start with /var/folders/2k/21gv5vmx6z7dlr1g0_jtm8r80000ks/T/iceberg_warehouse10689209309257803697/iceberg_data/default/table/ at org.apache.iceberg.RewriteTablePathUtil.relativize(RewriteTablePathUtil.java:686) at org.apache.iceberg.RewriteTablePathUtil.newPath(RewriteTablePathUtil.java:663) at org.apache.iceberg.RewriteTablePathUtil.writeDeleteFileEntry(RewriteTablePathUtil.java:492) at org.apache.iceberg.RewriteTablePathUtil.lambda$rewriteDeleteManifest$5(RewriteTablePathUtil.java:438) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1845) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)

can we remove the change from this patch and let's use #13882 to fix testFixture?

@dramaticlly Thank you for your message! I have carefully readed the relevant code and confirmed that this change is necessary. The reason is that the base path for the Hive library in RESTCatalog is inconsistent with other types of Catalogs. Currently, HIVE, HADOOP, and SPARK_SESSION all use URI format, so RESTCatalog needs to be adjusted to the same format. Additionally, since the TestRewriteTablePathProcedure test involves parameter type testing and simultaneously tests four different Catalogs, we must standardize the base path of the Hive library across all four Catalogs to URI format in order to conduct the test successfully. Otherwise, there could be issues with prefix validation failures.

Path file:/var/folders/2k/21gv5vmx6z7dlr1g0_jtm8r80000ks/T/iceberg_warehouse10689209309257803697/iceberg_data/default/table/data/deletes.parquet does not start with /var/folders/2k/21gv5vmx6z7dlr1g0_jtm8r80000ks/T/iceberg_warehouse10689209309257803697/iceberg_data/default/table/ java.lang.IllegalArgumentException: Path file:/var/folders/2k/21gv5vmx6z7dlr1g0_jtm8r80000ks/T/iceberg_warehouse10689209309257803697/iceberg_data/default/table/data/deletes.parquet does not start with /var/folders/2k/21gv5vmx6z7dlr1g0_jtm8r80000ks/T/iceberg_warehouse10689209309257803697/iceberg_data/default/table/ at org.apache.iceberg.RewriteTablePathUtil.relativize(RewriteTablePathUtil.java:686) .....

I have provided a detailed description in #13882 and hope someone from the team can assist in reviewing this issue.

cc: @nastra @manuzhang @huaxingao

@RussellSpitzer I initially intended to make this change in PR #13837, but since this change seems inconsistent with the core modifications of #13837, I have extracted it into PR #13882 for processing.

@slfan1989 I think this is actually because how position delete was written in unit test with File file = new File(removePrefix(table.location()) + "/data/deletes.parquet");. I suggested alternative way so that we can still get what we want without need to change this (at least for this PR).

If we want to update the behavior in REST test fixture, let's use #13882 instead with proper unit tests

@dramaticlly Thank you for your suggestion, which gave me some insights. I have added a logic in the unit tests for the REST Catalog. When encountering REST Catalog, it will use getAbsolutePath to obtain the path for deleting the file. After local testing, the solution works well, and I have added comments to explain the issue.

if (SparkCatalogConfig.REST.catalogName().equals(catalogName)) { // We applied this special handling because the base path for // matching the RESTCATALOG's Hive BaseLocation is represented // in the form of an AbsolutePath. filePath = file.getAbsolutePath().toString(); }

@dramaticlly Can I make this modification? I would like to hear your opinion. Thank you!

Given this is just for unit tests, apply special casing on REST catalog is fine to collect path is fine. We can later focus on #13882 to see if we need to resolve the differences between catalogs.

nastra · 2025-08-21T08:15:28Z

...ensions/src/test/java/org/apache/iceberg/spark/extensions/TestRewriteTablePathProcedure.java

  }

+  @TestTemplate
+  public void testRewriteTablePathWithSkipFileList() {


Suggested change

public void testRewriteTablePathWithSkipFileList() {

public void testRewriteTablePathWithoutFileList() {

nastra · 2025-08-21T08:15:49Z

...ensions/src/test/java/org/apache/iceberg/spark/extensions/TestRewriteTablePathProcedure.java

+
+  @TestTemplate
+  public void testRewriteTablePathWithManifestAndDeleteCounts() throws IOException {
+


unnecessary newline

nastra · 2025-08-21T08:16:56Z

...ensions/src/test/java/org/apache/iceberg/spark/extensions/TestRewriteTablePathProcedure.java

+    sql("INSERT INTO %s VALUES (3, 'c')", tableName);
+
+    Table table = validationCatalog.loadTable(tableIdent);
+    List<Pair<CharSequence, Long>> deletes =


this says deletes but it actually refers to data files

nastra · 2025-08-21T08:18:38Z

.../v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/RewriteTablePathProcedure.java

-            new StructField("file_list_location", DataTypes.StringType, true, Metadata.empty())
+            new StructField("file_list_location", DataTypes.StringType, true, Metadata.empty()),
+            new StructField(
+                "rewrite_metadata_files_count", DataTypes.IntegerType, true, Metadata.empty()),


Suggested change

"rewrite_metadata_files_count", DataTypes.IntegerType, true, Metadata.empty()),

"rewritten_manifests_count", DataTypes.IntegerType, true, Metadata.empty()),

nastra · 2025-08-21T08:18:48Z

.../v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/RewriteTablePathProcedure.java

+            new StructField(
+                "rewrite_metadata_files_count", DataTypes.IntegerType, true, Metadata.empty()),
+            new StructField(
+                "rewrite_delete_files_count", DataTypes.IntegerType, true, Metadata.empty())


Suggested change

"rewrite_delete_files_count", DataTypes.IntegerType, true, Metadata.empty())

"rewritten_delete_files_count", DataTypes.IntegerType, true, Metadata.empty())

nastra · 2025-08-21T08:31:08Z

api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java

    String latestVersion();
+
+    /** count of rewrite delete files, default value is 0 */
+    default int rewrittenDeleteFilesCount() {


I think rewrittenDeleteFilesCount might be a bit misleading? Aren't we just rewriting table paths and not actually compacting files together? Other Actions/Procedures use a similar naming but there it actually means that files have been compacted together

I think your suggestion is reasonable. Would it be fine to name them deleteFilesPathRewrittenCount and manifestFilesPathRewrittenCount? This naming would accurately reflect that only the paths are being rewritten, rather than the files themselves being merged or compacted.

I think that makes sense with a slight update to the naming, so maybe rewrittenDeleteFilePathsCount and rewrittenManifestFilePathsCount. @szehon-ho do you think that makes sense here?

nastra · 2025-08-21T08:38:12Z

...k/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteTablePathSparkAction.java

-    return saveFileList(copyPlan);
+    return builder
+        .stagingLocation(stagingDir)
+        .fileListLocation(fileListLocation)


I think another issue is actually that fileListLocation is as mandatory property in

iceberg/api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java

Line 121 in 26259d8

String fileListLocation();

(it's using Immutables and anything not marked as optional or nullable is a required field to be set when the object is being created). That being said, we can't make it optional/nullable at the API level, as that would break the API contract. Are we going to put a dummy string in there?

@nastra I am curious, does API contract allow String typed fileListLocation to have value null? I assume line 319 once built without fileListLocation will have its value = null

@dramaticlly

I checked the code based on nastra’s suggestion and confirmed that his analysis is correct. I also verified it locally: if fileListLocation is not set, an exception will indeed be thrown.

Cannot build Result, some of required attributes are not set [fileListLocation] java.lang.IllegalStateException: Cannot build Result, some of required attributes are not set [fileListLocation] at org.apache.iceberg.actions.ImmutableRewriteTablePath$Result$Builder.build(ImmutableRewriteTablePath.java:514)

I’ve been thinking—if fileListLocation is empty, can we simply treat it as N/A? My idea is to assign N/A directly during the check, without changing the original definition. Do you think this approach would work?

yep I think assigning a n/a string when building the result object makes sense to me, cc @szehon-ho

dramaticlly · 2025-08-21T18:16:46Z

...ensions/src/test/java/org/apache/iceberg/spark/extensions/TestRewriteTablePathProcedure.java

+        .isGreaterThan(0);
+    assertThat(rewrittenManifestFilesCount)
+        .as("Should rewrite at least one manifest file")
+        .isGreaterThan(0);


can we assert on equality of those rewritten file count? This is to ensure we have number correctly in procedure output

dramaticlly · 2025-08-21T18:19:35Z

...k/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteTablePathSparkAction.java

-    return saveFileList(copyPlan);
+    return builder
+        .stagingLocation(stagingDir)
+        .fileListLocation(fileListLocation)


@nastra I am curious, does API contract allow String typed fileListLocation to have value null? I assume line 319 once built without fileListLocation will have its value = null

slfan1989 · 2025-08-22T03:24:29Z

@nastra @dramaticlly Thank you for reviewing the code! Your suggestions have been very helpful to me. I have optimized the code based on both of your suggestions. Could you please take another look? Thank you so much!

dramaticlly · 2025-08-22T22:03:52Z

...k/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteTablePathSparkAction.java

  private String endVersionName;
  private String stagingDir;
-  private boolean skipFileList;
+  private boolean fileListEnabled = true;


can we call it createFileList so we can have consist naming in interface, action and procedure?

dramaticlly · 2025-08-22T22:13:39Z

...k/v4.0/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteTablePathsAction.java

  }
+
+  @Test
+  public void testRewritePathWithCreateFileListFalse() throws Exception {


nit: how about testRewritePathWithoutCreateFileList

dramaticlly · 2025-08-22T22:15:02Z

...k/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteTablePathSparkAction.java

+
+    // file list generation disabled
+    if (!fileListEnabled) {
+      return builder.fileListLocation("N/A").build();


nit: can we pull this up as a package local variable to represent not applicable and use it to assert in spark/v4.0/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteTablePathsAction.java

Thank you for the suggestion! I have extracted a variable to represent 'N/A'.

dramaticlly · 2025-08-22T22:19:55Z

api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java

+    /** Number of delete files whose paths were updated. Default value is 0. */
+    default int deleteFilesPathRewrittenCount() {
+      return 0;
+    }
+
+    /** Number of manifest files whose paths were updated. Default value is 0. */
+    default int manifestFilesPathRewrittenCount() {
+      return 0;
+    }


Suggested change

/** Number of delete files whose paths were updated. Default value is 0. */

default int deleteFilesPathRewrittenCount() {

return 0;

}

/** Number of manifest files whose paths were updated. Default value is 0. */

default int manifestFilesPathRewrittenCount() {

return 0;

}

/** Number of delete files with rewritten paths. */

default int deleteFilesPathRewrittenCount() {

return 0;

}

/** Number of manifest files with rewritten paths. */

default int manifestFilesPathRewrittenCount() {

return 0;

}

slfan1989 · 2025-09-08T06:24:06Z

@nastra Could you please take another look at this PR? Thank you very much!

slfan1989 · 2025-09-16T04:52:28Z

@nastra Could you please help review it? Thanks a lot! dramaticlly has already given a +1

nastra · 2025-09-16T14:23:50Z

api/src/main/java/org/apache/iceberg/actions/RewriteTablePath.java

    String latestVersion();
+
+    /** Number of delete files with rewritten paths. */
+    default int deleteFilesPathRewrittenCount() {


can we rename these two methods to what I suggested earlier in #13837 (comment)?

Thank you for reviewing the code. I’ve made improvements based on your suggestions.

nastra · 2025-09-16T14:27:17Z

...ensions/src/test/java/org/apache/iceberg/spark/extensions/TestRewriteTablePathProcedure.java

+      // in the form of an AbsolutePath.
+      filePath = file.getAbsolutePath().toString();
+    }
+    DeleteFile positionDeletes =


nit: please add a newline after the }

I have improved this part of the code.

...k/v4.0/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteTablePathsAction.java

nastra · 2025-09-16T15:17:30Z

.../v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/RewriteTablePathProcedure.java

          if (stagingLocation != null) {
            action.stagingLocation(stagingLocation);
          }
+          action.createFileList(createFileList);


nit: newline after }

nastra · 2025-09-16T15:18:14Z

.../v4.0/spark/src/main/java/org/apache/iceberg/spark/procedures/RewriteTablePathProcedure.java

-            new StructField("file_list_location", DataTypes.StringType, true, Metadata.empty())
+            new StructField("file_list_location", DataTypes.StringType, true, Metadata.empty()),
+            new StructField(
+                "rewritten_metadata_files_count", DataTypes.IntegerType, true, Metadata.empty()),


we would need to update the naming here as well to indicate that these are referring to paths

nastra · 2025-09-16T15:18:50Z

...k/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteTablePathSparkAction.java

+            .manifestFilesPathRewrittenCount(metaFiles.size())
+            .latestVersion(RewriteTablePathUtil.fileName(endVersionName));
+
+    // file list generation disabled


comment can be removed as it doesn't add much value imo

nastra · 2025-09-17T06:20:43Z

...k/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteTablePathSparkAction.java

+    return builder
+        .stagingLocation(stagingDir)
+        .fileListLocation(fileListLocation)
+        .latestVersion(RewriteTablePathUtil.fileName(endVersionName))


why are we setting latestVersion and stagingLocation again here?

Thanks for the question! The code isn’t duplicated — we moved the doExecute logic into rebuildMetadata so that fileListLocation and other Builder variables can be set in one place.

what I mean is: we are setting latestVersion and stagingLocation already in L309, so why do it here again?

Apologies for the misunderstanding, and thank you for your feedback. You are correct that this code is redundant, as the relevant variables are already set in line 309, and we don't need to set them again.

nastra

LGTM once the remaining comment has been applied. @szehon-ho can you also take a look please?

nastra · 2025-09-17T06:23:19Z

@slfan1989 can you please update the PR title to reflect the latest changes?

slfan1989 · 2025-09-17T16:39:40Z

@slfan1989 can you please update the PR title to reflect the latest changes?

@nastra Thanks a lot for the review! I’ve updated the title to Add create_file_list option to RewriteTablePathProcedure — Does this look okay to you?

slfan1989 · 2025-09-23T04:00:25Z

LGTM once the remaining comment has been applied. @szehon-ho can you also take a look please?

@nastra Could we consider moving this PR forward? Many thanks! I’d also like to contribute a new feature on top of it — in-place upgrades from Hive tables to Iceberg tables. Over the past four months, we’ve used this internally to upgrade more than 2,800 tables, delivering significant performance and cost benefits. I hope this feature can help more users perform in-place upgrades from Hive to Iceberg.

cc: @dramaticlly @szehon-ho

slfan1989 · 2025-09-24T22:26:34Z

@nastra @dramaticlly @manuzhang Thanks a lot for your help throughout the process of completing this PR!

szehon-ho · 2025-09-25T07:15:12Z

Sorry for delay in reply, late looks good to me, thanks @slfan1989

slfan1989 · 2025-09-25T07:43:32Z

Sorry for delay in reply, late looks good to me, thanks @slfan1989

@szehon-ho Thank you for helping with the review!

szehon-ho · 2025-09-25T07:50:20Z

...k/v4.0/spark/src/main/java/org/apache/iceberg/spark/actions/RewriteTablePathSparkAction.java


  private static final Logger LOG = LoggerFactory.getLogger(RewriteTablePathSparkAction.class);
  private static final String RESULT_LOCATION = "file-list";
+  static final String NOT_APPLICABLE = "N/A";


sorry just noticed one thing, this seems a bit awkward. Was there some discussion on just having an empty string, or does the result object not allow it? In any case, I feel it can be private here.

Thank you for your message. After discussion, it was ultimately decided to be a package-level private member. We also used this variable in the unit tests.

#13837 (comment)

The use of N/A has also been discussed. From a personal perspective, I believe it should be acceptable. The specific message link is as follows:

#13837 (comment)
#13837 (comment)

…ure. (apache#13837)

Add 'skip_file_list' option to RewriteTablePathProcedure for optional…

b138052

… file-list generation.

github-actions bot added API spark core labels Aug 17, 2025

slfan1989 changed the title ~~Add 'skip_file_list' option to RewriteTablePathProcedure for optional file-list generation.~~ Spark: Add 'skip_file_list' option to RewriteTablePathProcedure for optional file-list generation. Aug 17, 2025

slfan1989 marked this pull request as draft August 17, 2025 03:04

slfan1989 marked this pull request as ready for review August 18, 2025 02:02

manuzhang changed the title ~~Spark: Add 'skip_file_list' option to RewriteTablePathProcedure for optional file-list generation.~~ API, Spark 4.0: Add 'skip_file_list' option to RewriteTablePathProcedure for optional file-list generation. Aug 18, 2025

manuzhang reviewed Aug 18, 2025

View reviewed changes

dramaticlly reviewed Aug 18, 2025

View reviewed changes

Improve Some Code.

26259d8

github-actions bot added the OPENAPI label Aug 21, 2025

Improve Some Code.

815e092

slfan1989 mentioned this pull request Aug 21, 2025

Use URI string for default Iceberg warehouse location #13882

Closed

nastra reviewed Aug 21, 2025

View reviewed changes

dramaticlly reviewed Aug 21, 2025

View reviewed changes

Improve Some Code.

3f1ab0c

dramaticlly reviewed Aug 22, 2025

View reviewed changes

dramaticlly approved these changes Sep 3, 2025

View reviewed changes

slfan1989 and others added 2 commits September 15, 2025 13:10

Merge branch 'main' into add_skip_file_list2

1fb1407

Spark 4.0: Fix CheckStyle Issue.

8471bf1

nastra reviewed Sep 16, 2025

View reviewed changes

...k/v4.0/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteTablePathsAction.java Outdated Show resolved Hide resolved

nastra reviewed Sep 16, 2025

View reviewed changes

...k/v4.0/spark/src/test/java/org/apache/iceberg/spark/actions/TestRewriteTablePathsAction.java Outdated Show resolved Hide resolved

nastra reviewed Sep 16, 2025

View reviewed changes

slfan1989 added 2 commits September 17, 2025 12:03

Spark 4.0: Improve Some Code.

9904d5e

Spark 4.0: Fix CheckStyle Issue.

add19fc

nastra reviewed Sep 17, 2025

View reviewed changes

nastra approved these changes Sep 17, 2025

View reviewed changes

slfan1989 changed the title ~~API, Spark 4.0: Add 'skip_file_list' option to RewriteTablePathProcedure for optional file-list generation.~~ API, Spark 4.0: Add create_file_list option to RewriteTablePathProcedure. Sep 17, 2025

Spark 4.0: Improve Some Code.

c7909ca

nastra requested a review from szehon-ho September 19, 2025 13:55

nastra merged commit 5f30e4f into apache:main Sep 24, 2025
43 checks passed

slfan1989 mentioned this pull request Sep 25, 2025

Spark 3.4,3.5: Backport: Add create_file_list option to RewriteTablePathProcedure. #14180

Merged

szehon-ho reviewed Sep 25, 2025

View reviewed changes

gabeiglio pushed a commit to gabeiglio/iceberg that referenced this pull request Oct 1, 2025

API, Spark 4.0: Add create_file_list option to RewriteTablePathProced…

fe72180

…ure. (apache#13837)

thomaschow pushed a commit to thomaschow/iceberg that referenced this pull request Jan 19, 2026

API, Spark 4.0: Add create_file_list option to RewriteTablePathProced…

21862ac

…ure. (apache#13837)

		if (skipFileList) {
		return builder.fileListLocation("").build();

	/** count of rewrite delete files, default value is 0 */
	/** count of rewritten delete files, default value is 0 */

	/** count of rewrite metadata files involved, default value is 0 */
	/** count of rewritten manifests involved, default value is 0 */

	public void testRewriteTablePathWithSkipFileList() {
	public void testRewriteTablePathWithoutFileList() {


		@TestTemplate
		public void testRewriteTablePathWithManifestAndDeleteCounts() throws IOException {

	"rewrite_metadata_files_count", DataTypes.IntegerType, true, Metadata.empty()),
	"rewritten_manifests_count", DataTypes.IntegerType, true, Metadata.empty()),

	"rewrite_delete_files_count", DataTypes.IntegerType, true, Metadata.empty())
	"rewritten_delete_files_count", DataTypes.IntegerType, true, Metadata.empty())

API, Spark 4.0: Add create_file_list option to RewriteTablePathProcedure. #13837

API, Spark 4.0: Add create_file_list option to RewriteTablePathProcedure. #13837

Uh oh!

Conversation

slfan1989 commented Aug 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

slfan1989 commented Aug 18, 2025

Uh oh!

manuzhang Aug 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nastra Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

slfan1989 Aug 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

nastra Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

slfan1989 commented Aug 17, 2025 •

edited

Loading

manuzhang Aug 18, 2025 •

edited

Loading

nastra Aug 21, 2025 •

edited

Loading

slfan1989 Aug 29, 2025 •

edited

Loading

nastra Aug 21, 2025 •

edited

Loading