Skip to content

[iceberg][doc] add docs about iceberg migration #5097

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
JingsongLi merged 2 commits into from
Mar 3, 2025
Merged

[iceberg][doc] add docs about iceberg migration #5097

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
7d6ac93
[doc] first doc
LsomeYeah Jan 14, 2025
55a501d
[doc] second version
LsomeYeah Feb 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions docs/content/flink/procedures.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -578,6 +578,25 @@ All available procedures are listed below.
CALL sys.migrate_file(connector => 'hive', source_table => 'default.T', target_table => 'default.T2', delete_origin => true, parallelism => 6)
</td>
</tr>
<tr>
<td>migrate_iceberg_table</td>
<td>
-- Use named argument<br/>
CALL sys.migrate_iceberg_table(source_table => 'database_name.table_name', iceberg_options => 'iceberg_options', options => 'paimon_options', parallelism => parallelism);<br/><br/>
-- Use indexed argument<br/>
CALL sys.migrate_iceberg_table('source_table','iceberg_options', 'options', 'parallelism');
</td>
<td>
To migrate iceberg table to paimon. Arguments:
<li>source_table: string type, is used to specify the source iceberg table to migrate, it's required.</li>
<li>iceberg_options: string type, is used to specify the configuration of migration, multiple configuration items are separated by commas. it's required.</li>
<li>options: string type, is used to specify the additional options for the target paimon table, it's optional.</li>
<li>parallelism: integer type, is used to specify the parallelism of the migration job, it's optional.</li>
</td>
<td>
CALL sys.migrate_iceberg_table(source_table => 'iceberg_db.iceberg_tbl',iceberg_options => 'metadata.iceberg.storage=hadoop-catalog,iceberg_warehouse=/path/to/iceberg/warehouse');
</td>
</tr>
<tr>
<td>expire_snapshots</td>
<td>
Expand Down
173 changes: 173 additions & 0 deletions docs/content/migration/migration-from-iceberg.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,173 @@
---
title: "Migration From Iceberg"
weight: 1
type: docs
aliases:
- /migration/migration-from-iceberg.html
---
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->

# Iceberg Migration

Apache Iceberg data with parquet file format could be migrated to Apache Paimon.
When migrating an iceberg table to a paimon table, the origin iceberg table will permanently disappear. **So please back up your data if you
still need the original table.** The migrated paimon table will be an [append table]({{< ref "append-table/overview" >}}).

<span style="color: red; "> **We highly recommend to back up iceberg table data before migrating, because migrating action is not atomic. If been interrupted while migrating, you may lose your data.** </span>

## Migrate Iceberg Table
Currently, we can use paimon catalog with MigrateIcebergTableProcedure or MigrateIcebergTableAction to migrate the data used by **latest iceberg snapshot**
in an iceberg table to a paimon table.

Iceberg tables managed by hadoop-catalog or hive-catalog are supported to be migrated to paimon.
As for the type of paimon catalog, it needs to have access to the file system where the iceberg metadata and data files are located.
This means we could migrate an iceberg table managed by hadoop-catalog to a paimon table in hive catalog if their warehouses are in the same file system.

When migrating, the iceberg data files which were marked by DELETED will be ignored. Only the data files referenced by manifest entries with 'EXISTING' and 'ADDED' content will be migrated to paimon.
Notably, now we don't support migrating iceberg tables with delete files(deletion vectors, position delete files, equality delete files etc.)

Now only **parquet** format is supported in iceberg migration.

### MigrateIcebergTableProcedure
You can run the following command to migrate an iceberg table to a paimon table.
```sql
-- Use named argument
CALL sys.migrate_iceberg_table(source_table => 'database_name.table_name', iceberg_options => 'iceberg_options', options => 'paimon_options', parallelism => parallelism);

-- Use indexed argument
CALL sys.migrate_iceberg_table('source_table','iceberg_options', 'options', 'parallelism');
```
* `source_table`, string type, is used to specify the source iceberg table to migrate, it's required.
* `iceberg_options`, string type, is used to specify the configuration of migration, multiple configuration items are separated by commas. it's required.
* `options`, string type, is used to specify the additional options for the target paimon table, it's optional.
* `parallelism`, integer type, is used to specify the parallelism of the migration job, it's optional.

#### hadoop-catalog
To migrate iceberg table managed by hadoop-catalog, you need set `metadata.iceberg.storage=hadoop-catalog` and `iceberg_warehouse`. Example:
```sql
CREATE CATALOG paimon_catalog WITH ('type' = 'paimon', 'warehouse' = '/path/to/paimon/warehouse');

USE CATALOG paimon_catalog;

CALL sys.migrate_iceberg_table(
source_table => 'iceberg_db.iceberg_tbl',
iceberg_options => 'metadata.iceberg.storage=hadoop-catalog,iceberg_warehouse=/path/to/iceberg/warehouse'
);
```
If you want the metadata of the migrated paimon table to be managed by hive, you can also create a hive catalog of paimon for migration. Example:
```sql
CREATE CATALOG paimon_catalog WITH (
'type' = 'paimon',
'metastore' = 'hive',
'uri' = 'thrift://<host>:<port>',
'warehouse' = '/path/to/paimon/warehouse'
);

USE CATALOG paimon_catalog;

CALL sys.migrate_iceberg_table(
source_table => 'iceberg_db.iceberg_tbl',
iceberg_options => 'metadata.iceberg.storage=hadoop-catalog,iceberg_warehouse=/path/to/iceberg/warehouse'
);
```

#### hive-catalog
To migrate iceberg table managed by hive-catalog, you need set `metadata.iceberg.storage=hive-catalog`
and provide information about Hive Metastore used by the iceberg table in `iceberg_options`.

<table class="table table-bordered">
<thead>
<tr>
<th class="text-left" style="width: 20%">Option</th>
<th class="text-left" style="width: 5%">Default</th>
<th class="text-left" style="width: 10%">Type</th>
<th class="text-left" style="width: 60%">Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><h5>metadata.iceberg.uri</h5></td>
<td style="word-wrap: break-word;">none</td>
<td>String</td>
<td>Hive metastore uri for Iceberg Hive catalog.</td>
</tr>
<tr>
<td><h5>metadata.iceberg.hive-conf-dir</h5></td>
<td style="word-wrap: break-word;">none</td>
<td>String</td>
<td>hive-conf-dir for Iceberg Hive catalog.</td>
</tr>
<tr>
<td><h5>metadata.iceberg.hadoop-conf-dir</h5></td>
<td style="word-wrap: break-word;">none</td>
<td>String</td>
<td>hadoop-conf-dir for Iceberg Hive catalog.</td>
</tr>
<tr>
<td><h5>metadata.iceberg.hive-client-class</h5></td>
<td style="word-wrap: break-word;">org.apache.hadoop.hive.metastore.HiveMetaStoreClient</td>
<td>String</td>
<td>Hive client class name for Iceberg Hive Catalog.</td>
</tr>
</tbody>
</table>

Example:
```sql
CREATE CATALOG paimon_catalog WITH (
'type' = 'paimon',
'metastore' = 'hive',
'uri' = 'thrift://<host>:<port>',
'warehouse' = '/path/to/paimon/warehouse'
);

USE CATALOG paimon_catalog;

CALL sys.migrate_iceberg_table(
source_table => 'iceberg_db.iceberg_tbl',
iceberg_options => 'metadata.iceberg.storage=hive-catalog,metadata.iceberg.uri=thrift://<host>:<port>'
);
```

### MigrateIcebergTableAction
You can also use flink action for migration:
```bash
<FLINK_HOME>/bin/flink run \
/path/to/paimon-flink-action-{{< version >}}.jar \
migrate_iceberg_table \
--table <icebergDatabase.icebergTable> \
--iceberg_options <iceberg-conf [,iceberg-conf ...]> \
[--parallelism <parallelism>] \
[--catalog_conf <paimon-catalog-conf> [--catalog_conf <paimon-catalog-conf> ...]] \
[--options <paimon-table-conf [,paimon-table-conf ...]> ]
```

Example:
```bash
<FLINK_HOME>/bin/flink run \
/path/to/paimon-flink-action-{{< version >}}.jar \
migrate_iceberg_table \
--table iceberg_db.iceberg_tbl \
--iceberg_options metadata.iceberg.storage=hive-catalog, metadata.iceberg.uri=thrift://localhost:9083 \
--parallelism 6 \
--catalog_conf warehouse=/path/to/paimon/warehouse \
--catalog_conf metastore=hive \
--catalog_conf uri=thrift://localhost:9083
```