Warehouse path is always required for Glue catalog, even when there isn't one

[omerhadari]

The warehouse config prop for Glue catalog is required, but it isn't always known or required. The Glue catalog ID is enough in order to resolve metadata locations.

Moreover - I don't think this path is used for anything but protocol (FileIO type) inference? I was able to work around this issue by passing the correct Glue catalog ID (which is the AWS account ID) and s3:// as the warehouse, with no issue.

In my opinion warehouse should be replaced with something like FileIO type and only the Glue catalog ID should be required, but for backwards compatibility not passing warehouse should at least be allowed.

[CTTY]

I think this makes sense and making the warehouse path optional matches the iceberg-java behavior. We should probably fix this while implementing the new catalog builder: #1259

[chasehaddleton]

I believe there is an entire class of bugs caused by the current behavior. For example, if you pass s3 as the warehouse, but the catalog contains tables with both s3 and s3a files, then it erroneously fails saying that protocols do not match (even though it is immaterial). Moving to a FileIOType could solve this, assuming it's paired with a set of "valid protocols for a type".

[lliangyu-lin]

Another issue is that when loading the table from glue, the table is read using the catalog file io. So if catalog file io is set to S3 with setting the warehouse, but the table loaded is stored in another storage, it will continue to use the S3 file io to read the metadata. I think the file io should be inferred during create table and load table instead of initialized with the catalog...

Reference in python

• https://github.com/apache/iceberg-python/blob/main/pyiceberg/io/__init__.py#L348

[github-actions]

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

[blackmwk]

Don't close this, it's a but.