AWS Glue Catalog for Iceberg ingest extension

[Shekharrajak]

Description

The Iceberg extension to connect to Glue Iceberg catalog.

Motivation

We have Iceberg framework in AWS Glue, that would be helpful for Druid users to directly connect Iceberg table in Glue.

The Iceberg Ingest extension needs to be enhanced accordingly.

[Shekharrajak]

I'm currently exploring the feasibility of this feature and putting together a preliminary design and code draft for a PR. This will allow for a more concrete discussion. I'd appreciate any feedback or links to relevant discussions that could inform this process.

[abhishekrb19]

Hi @Shekharrajak, this would be a useful feature. Please see the REST-based catalog implementation for reference - #17124. The Glue API docs once you've the Glue dependencies added - https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-tables.html#aws-glue-api-catalog-tables-IcebergInput.

cc: @a2l007

[Shekharrajak]

Thanks, I am almost done with the basic version of it. I will share the PR soon for further discussion and testing.

[a2l007]

Thanks for working on this @Shekharrajak , looking forward to the PR!

[Shekharrajak]

I feel we should have modular API to choose catalog, something like (means every input source can have option to choose the catalog that input source supports):

"inputSource": {
     "type": "iceberg",
     "tableName": "iceberg_spr_1_test_23sep",
     "namespace": "test",
     "catalog":
       { 
        "type" : "iceberg-catalog",
         "catalog-type" : "glue", // or hive, HDFS, JDBC, ..  - iceberg support catalog
         "warehousePath": "<>", // and all other config which is needed for respective catalog connection. Ref: https://iceberg.apache.org/docs/nightly/kafka-connect/#catalog-configuration
         "catalogProperties" :
           {"<key>": "<value>"}
       },
      "icebergFilter":
       {
         ...
       },
   "warehouseSource": {
        ..
     }
   }

What do you guys think ? @a2l007 @abhishekrb19

[Shekharrajak]

Once we have such API then we can leverage Iceberg CatalogUtil.buildIcebergCatalog method that will returns the org.apache.iceberg.catalog.Catalog

[a2l007]

@Shekharrajak The catalog properties are specific to the iceberg catalog and doesn't apply to other input sources. Do you mind expanding on how this can be useful for other input sources?