[Python][R] GcsFilesystem can appear to hang for non permanent errors

[asfimport]

GcsFileSystem will attempt to retry if it gets a non-permanent error (such as couldn't connect to server). That's fine, except: (1) the sleep call used by the retry doesn't seem to check for interrupts and (2) the default retry timeout is 15 minutes!

The following snippets will hang for 15 minutes if you run them and wait about 5 seconds before trying to do a keyboard interrupt (CTRL+C):

Rscript -e 'library(arrow); fs <- GcsFileSystem$create(endpoint_override="localhost:1234", anonymous=TRUE); fs$CreateDir("x")'

python -c 'from pyarrow.fs import GcsFileSystem; fs = GcsFileSystem(endpoint_override="localhost:1234", anonymous=True); fs.create_dir("x")'

 

Reporter: Will Jones / @wjones127

Related issues:

• [R][C++] Allow cancelling long-running commands (relates to)

_{Note: This issue was originally created as ARROW-17020. Please see the migration documentation for further details.}

[asfimport]

Will Jones / @wjones127:
FYI @coryan

[asfimport]

Ian Cook / @ianmcook:
As of the 9.0.0 release, R package docs contain a reference to this 15-minute default and an example showing how to override it (added in ARROW-16887) and the PyArrow docs contain an example showing how to override it (added in ARROW-16818). If we choose to implement a different default retry timeout to solve this issue, these docs should be modified accordingly.