Python: Add Google Cloud Storage support #6906

Buktoria · 2023-02-22T17:11:36Z

Adding Google Cloud Storage support to the Python Iceberg CLI

Added GCSFileIO support through the gcsfs python lib
Added PyArrow GCS support with PyArrows GcsFileSystem class
Added GCS test suite using a GCS emulator Docker Container (https://github.com/fsouza/fake-gcs-server)
Updated docs to include GCSFileIO configs options (full list of possible options can be found here)

Fokko · 2023-02-27T13:35:31Z

@Buktoria this is awesome, thanks for working on this. Let me know when it is ready for review!

Fokko · 2023-03-08T12:33:40Z

python/Makefile


 test:
-	poetry run coverage run --source=pyiceberg/ -m pytest tests/ -m "not s3 and not adlfs" ${PYTEST_ARGS}
+	poetry run coverage run --source=pyiceberg/ -m pytest tests/ -m "not s3 and not adlfs and not gcs" ${PYTEST_ARGS}


This will get prettier once #6398 has been merged

Buktoria · 2023-03-13T00:16:03Z

@Fokko This PR is ready to be reviewed

JonasJ-ap

Thank you for your great work!

JonasJ-ap · 2023-03-22T05:38:30Z

python/pyiceberg/io/pyarrow.py

+                }
+                return GcsFileSystem(**client_kwargs)
+            else:
+                return GcsFileSystem()


[Question] Is it possible to take out this if...else..., like:

elif scheme in {"gs", "gcs"}: client_kwargs = { "access_token": self.properties.get("gs.access-token"), "credential_token_expiration": self.properties.get("gs.credential-token-expiration"), } return GcsFileSystem(**client_kwargs)

?
Is there any problem if access_token is not given but credential_token_expiration is provided? Thank you in advance for your answer.

Fokko · 2023-04-10T22:28:45Z

@Buktoria thanks for raising this. I was out for a couple of weeks, could you resolve the merge conflicts? I'll review the PR asap.

Fokko

@Buktoria Sorry for pinging you again? Would you have any time to fix the merge conflicts? It would be great to get this into 0.4.0

python/pyiceberg/io/fsspec.py

Fokko · 2023-05-14T19:54:56Z

@Buktoria Gentle ping! :)

Buktoria · 2023-05-23T19:31:09Z

Hey Fokko. I have been away, and just got back over the weekend. I will make sure to wrap this up this week. So sorry for the delay.

Fokko · 2023-06-13T15:38:30Z

@Buktoria No problem, just checking if it is still on your list. Would be really cool to get this in.

Fokko · 2023-07-18T15:16:21Z

@Buktoria Gente ping from my side. Would be great to get this in.

danielcweeks · 2023-07-18T15:19:57Z

python/pyiceberg/io/pyarrow.py

            }
            return S3FileSystem(**client_kwargs)
+        elif scheme in {"gs", "gcs"}:
+            if access_token := self.properties.get("gs.access-token"):


I think this property should be gs.token based on the properties in the readme

danielcweeks · 2023-07-18T15:20:20Z

python/pyiceberg/io/pyarrow.py

            }
            return S3FileSystem(**client_kwargs)
+        elif scheme in {"gs", "gcs"}:
+            if access_token := self.properties.get("gs.access-token"):


Can we add gs.credential-token-expiration to the docs?

I missed it. I've added it to the docs now.

bryanck · 2023-07-19T15:11:12Z

python/mkdocs/docs/configuration.md

-| adlfs.client-secret     | oCA3R6P\*ka#oa1Sms2J74z...                                                                 | The client-secret                                                                                                                                                                                                                                                                    |
+| adlfs.client-secret     | oCA3R6P\*ka#oa1Sms2J74z...                                                                | The client-secret                                                                                                                                                                                                                                                                    |
+
+### Google Cloud Storage


It would be nice if the Python GCS properties could be consistent with the Java properties where possible. Java already has some properties defined so starting with those would help avoid any backwards compatibility issues (rather than changing them). There is also a PR to add OAuth2 access token properties on the Java side.

Co-authored-by: Fokko Driesprong <fokko@apache.org>

Fokko

Thanks for picking this up, it looks like the CI is failing. Can you run pip3 install pre-commit && pre-commit run --all-files?

Fokko · 2023-07-24T06:45:05Z

python/mkdocs/docs/configuration.md

 | Key                     | Example                                                                                   | Description                                                                                                                                                                                                                                                                            |
 | ----------------------- | ----------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
 | adlfs.connection-string | AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqF...;BlobEndpoint=http://localhost/ | A [connection string](https://learn.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string). This could be used to use FileIO with any adlfs-compatible object storage service that has a different endpoint (like [azurite](https://github.com/azure/azurite)). |
+


Can you remove this unrelated change?

Fokko · 2023-07-24T06:47:03Z

python/mkdocs/docs/index.md

 | s3fs     | S3FS as a FileIO implementation to interact with the object store    |
 | adlfs    | ADLFS as a FileIO implementation to interact with the object store   |
 | snappy   | Support for snappy Avro compression                                  |
+| gcs      | GCS as the FileIO implementation to interact with the object store   |


Fokko · 2023-08-14T12:44:57Z

Closing this one, since #8207 is in! Thanks for the PR @Buktoria 🙌🏻

github-actions bot added the python label Feb 22, 2023

Buktoria force-pushed the pyiceberg/accpet-file-paths-with-gs-prefix branch from 68cc990 to 5345bb9 Compare February 25, 2023 01:42

Buktoria force-pushed the pyiceberg/accpet-file-paths-with-gs-prefix branch 2 times, most recently from 2e59713 to aa58ac2 Compare March 8, 2023 01:48

Buktoria marked this pull request as ready for review March 8, 2023 01:59

Buktoria force-pushed the pyiceberg/accpet-file-paths-with-gs-prefix branch from aa58ac2 to 626aa7e Compare March 8, 2023 02:33

Fokko reviewed Mar 8, 2023

View reviewed changes

Buktoria force-pushed the pyiceberg/accpet-file-paths-with-gs-prefix branch 2 times, most recently from 9c5a826 to a3d430b Compare March 13, 2023 00:12

JonasJ-ap reviewed Mar 22, 2023

View reviewed changes

Fokko self-requested a review April 10, 2023 22:28

Fokko added this to the PyIceberg 0.4.0 release milestone Apr 18, 2023

Fokko reviewed Apr 24, 2023

View reviewed changes

python/pyiceberg/io/fsspec.py Outdated Show resolved Hide resolved

Fokko changed the title ~~pyiceberg: Add Google Cloud Storage support~~ Python: Add Google Cloud Storage support Apr 24, 2023

Fokko modified the milestones: PyIceberg 0.4.0 release, PyIceberg 0.5.0 release Jun 22, 2023

danielcweeks reviewed Jul 18, 2023

View reviewed changes

bryanck reviewed Jul 19, 2023

View reviewed changes

pyiceberg: Add Google Cloud Storage support

fa609f8

Buktoria force-pushed the pyiceberg/accpet-file-paths-with-gs-prefix branch 2 times, most recently from 71e7135 to d17d112 Compare July 24, 2023 00:05

updated project dependecies

8b2ba33

Buktoria force-pushed the pyiceberg/accpet-file-paths-with-gs-prefix branch from d17d112 to 8b2ba33 Compare July 24, 2023 00:08

Update python/pyiceberg/io/fsspec.py

3740c69

Co-authored-by: Fokko Driesprong <fokko@apache.org>

Fokko reviewed Jul 24, 2023

View reviewed changes

Fokko mentioned this pull request Aug 2, 2023

Python: GCS Support #8207

Merged

Fokko closed this Aug 14, 2023

	\| gcs \| GCS as the FileIO implementation to interact with the object store \|
	\| gcsfs \| GCS as the FileIO implementation to interact with the object store \|

Python: Add Google Cloud Storage support #6906

Python: Add Google Cloud Storage support #6906

Uh oh!

Conversation

Buktoria commented Feb 22, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Fokko commented Feb 27, 2023

Uh oh!

Fokko Mar 8, 2023

Choose a reason for hiding this comment

Uh oh!

Buktoria commented Mar 13, 2023

Uh oh!

JonasJ-ap left a comment

Choose a reason for hiding this comment

Uh oh!

JonasJ-ap Mar 22, 2023

Choose a reason for hiding this comment

Uh oh!

Fokko commented Apr 10, 2023

Uh oh!

Fokko left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Fokko commented May 14, 2023

Uh oh!

Buktoria commented May 23, 2023

Uh oh!

Fokko commented Jun 13, 2023

Uh oh!

Fokko commented Jul 18, 2023

Uh oh!

danielcweeks Jul 18, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

danielcweeks Jul 18, 2023

Choose a reason for hiding this comment

Uh oh!

Buktoria Jul 24, 2023

Choose a reason for hiding this comment

Uh oh!

bryanck Jul 19, 2023

Choose a reason for hiding this comment

Uh oh!

Fokko left a comment

Choose a reason for hiding this comment

Uh oh!

Fokko Jul 24, 2023

Choose a reason for hiding this comment

Uh oh!

Fokko Jul 24, 2023

Choose a reason for hiding this comment

Uh oh!

Fokko commented Aug 14, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Buktoria commented Feb 22, 2023 •

edited

Loading

danielcweeks Jul 18, 2023 •

edited

Loading