Skip to content

ARROW-16818: [Doc][Python] Document GCS filesystem for PyArrow #13681

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
AlenkaF merged 2 commits into from
Jul 22, 2022
Merged

ARROW-16818: [Doc][Python] Document GCS filesystem for PyArrow #13681

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
c497dfb
Initial commit
rok Jul 21, 2022
7490423
Apply suggestions from code review
rok Jul 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/source/python/api/filesystems.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ Filesystem Implementations

LocalFileSystem
S3FileSystem
GcsFileSystem
HadoopFileSystem
SubTreeFileSystem

Expand Down
35 changes: 35 additions & 0 deletions docs/source/python/filesystems.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ Pyarrow implements natively the following filesystem subclasses:

* :ref:`filesystem-localfs` (:class:`LocalFileSystem`)
* :ref:`filesystem-s3` (:class:`S3FileSystem`)
* :ref:`filesystem-gcs` (:class:`GcsFileSystem`)
* :ref:`filesystem-hdfs` (:class:`HadoopFileSystem`)

It is also possible to use your own fsspec-compliant filesystem with pyarrow functionalities as described in the section :ref:`filesystem-fsspec`.
Expand Down Expand Up @@ -183,6 +184,40 @@ Example how you can read contents from a S3 bucket::
for the different ways to configure the AWS credentials.


.. _filesystem-gcs:

Google Cloud Storage File System
--------------------------------

PyArrow implements natively a Google Cloud Storage (GCS) backed file system
for GCS storage.

If not running on Google Cloud Platform (GCP), this generally requires the
environment variable ``GOOGLE_APPLICATION_CREDENTIALS`` to point to a
JSON file containing credentials.

Example showing how you can read contents from a GCS bucket::

>>> from datetime import timedelta
>>> from pyarrow import fs
>>> gcs = fs.GcsFileSystem(anonymous=True, retry_time_limit=timedelta(seconds=15))

# List all contents in a bucket, recursively
>>> uri = "gcp-public-data-landsat/LC08/01/001/003/"
>>> file_list = gcs.get_file_info(fs.FileSelector(uri, recursive=True))

# Open a file for reading and download its contents
>>> f = gcs.open_input_stream(file_list[0].path)
>>> f.read(64)
b'GROUP = FILE_HEADER\n LANDSAT_SCENE_ID = "LC80010032013082LGN03"\n S'

.. seealso::

The :class:`GcsFileSystem` constructor by default uses the
process described in `GCS docs <https://google.aip.dev/auth/4110>`__
to resolve credentials.


.. _filesystem-hdfs:

Hadoop Distributed File System (HDFS)
Expand Down