How to move Iceberg table from one location to another

[cccs-jc]

I'm using the directory based Iceberg catalogue (Hadoop based)

I've put my files into

/iceberg_src/db/table1

I would like to move table1 into another directory based Iceberg catalogue

/iceberg_dst/db2

I can't use linux mv command to move the table1 directory into /iceberg_dst/db2. Iceberg keeps full paths of the data files in it's metadata files and manifest avro files.

Is there an Iceberg command to move a table this way, which would update the metadata and manifests?
Preferably in pyspark or SQL.
Thanks

[anuragmantri]

There is an open issue to add support for relative paths in table metadata to make relocating tables easier. There is also a design doc if you are interested.

Unfortunately, until this support is added, you will have to rewrite the metadata files to reflect the new paths after moving your data.

[meicanhong]

@anuragmantri hi, I would like to know how to rewrite the path in the metadata file? We are now migrating the iceberg table to another cloud, and we have encountered the same problem.

[atifiu]

@anuragmantri How to rewrite the metadata after copying the files, I have tried using add_files but it does not support transformation on partition column

@meicanhong Are you able to solve this ?

[meicanhong]

@atifiu I didn't solve this problem. For a discussion on this issue, check out this issue #1617

[ksmatharoo]

@atifiu @meicanhong We encountered a similar issue when we switched S3 vendors, necessitating the copying of all data to another bucket, resulting in unreadable iceberg tables. Currently, we have two options: either attempt to rewrite metadata with custom code or inject custom file IO. We have experimented with both approaches, and I can provide you with our code for reference.
https://github.com/ksmatharoo/IcebergCustomFileIO
https://github.com/ksmatharoo/IcebergMetadataRewrite

[cccs-jc]

@ksmatharoo, thank you for providing your code to replace the paths within the iceberg metadata.

I've implemented a similar solution using Python's fastavro to handle Avro files. In my approach, I read the Avro files, replace the paths, and then save the modified files.

So far, everything seems to be working well. However, I noticed that your code is more sophisticated, particularly in how you manage lists of known paths. I'm curious if there's a specific reason for this. While my solution is straightforward, I'm wondering if there might be subtle issues that I haven't considered. Any insights you could provide would be appreciated!

[namangoel31]

@cccs-jc, how do you determine the schema for writing to avro? I'm not able to get anything useful. Looking for python-only solutions to modify location in manifest and snap files.

[anuragmantri]

There is now a PR in review which rewrites metadata with new location and also does some other checks. Please take a look at the PR for some ideas

#10024

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[forzagreen]

Still having the issue, in AWS S3.

[Hoeze]

The missing ability to "mv " to another location is still a main reason why I do not use Iceberg.