[C++] S3 FileSystem doesn't seem to handle redirects

[asfimport]

In some conditions AWS S3 seems to respond with a redirect, but Arrow seems to consider it an error instead of following the redirect.

For example see

    s3, bucket = fs.FileSystem.from_uri("s3://ursa-labs-taxi-data/?region=us-east-1")
    print(s3.get_file_info(fs.FileSelector(bucket+"/2011", recursive=True)))

The error that you get is

 OSError: When listing objects under key '2011' in bucket 'ursa-labs-taxi-data': AWS Error [code 100]: Unable to parse ExceptionName: PermanentRedirect Message: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.

It should probably follow the PermanentRedirect instead of choking over it

IT is also possible to reproduce it using

    from pyarrow import fs

    s3 = fs.SubTreeFileSystem("ursa-labs-taxi-data", fs.S3FileSystem())
    print(s3.get_file_info(fs.FileSelector("2011", recursive=True)))

Reporter: Alessandro Molina / @amol-

Related issues:

• [Python] Expose S3 region resolver (relates to)

_{Note: This issue was originally created as ARROW-13237. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
Do other S3 toolkits follow the redirect automatically in this particular case? I wonder what we're missing exactly.

[asfimport]

Alessandro Molina / @amol-:
If we don't want to follow the redirect (which I think we should, I guess it's a reasonable expectation that HTTP clients handle redirects), it would at least be convenient to print the target URL in the exception, so that the user has a way to know how it should change the code.

[asfimport]

Antoine Pitrou / @pitrou:
Ok, it seems you can use (on recent AWS SDK versions perhaps?) a pseudo-region named "aws-global" that will automatically handle redirects:

>>> f = fs.S3FileSystem(region='aws-global')
>>> f.get_file_info(fs.FileSelector('ursa-labs-taxi-data'))
[<FileInfo for 'ursa-labs-taxi-data/2009': type=FileType.Directory>,
 <FileInfo for 'ursa-labs-taxi-data/2010': type=FileType.Directory>,
 <FileInfo for 'ursa-labs-taxi-data/2011': type=FileType.Directory>,
 <FileInfo for 'ursa-labs-taxi-data/2012': type=FileType.Directory>,
 <FileInfo for 'ursa-labs-taxi-data/2013': type=FileType.Directory>,
 <FileInfo for 'ursa-labs-taxi-data/2014': type=FileType.Directory>,
 <FileInfo for 'ursa-labs-taxi-data/2015': type=FileType.Directory>,
 <FileInfo for 'ursa-labs-taxi-data/2016': type=FileType.Directory>,
 <FileInfo for 'ursa-labs-taxi-data/2017': type=FileType.Directory>,
 <FileInfo for 'ursa-labs-taxi-data/2018': type=FileType.Directory>,
 <FileInfo for 'ursa-labs-taxi-data/2019': type=FileType.Directory>]

[asfimport]

Alessandro Molina / @amol-:
Should we make aws-global the default region? That seems to improve the user experience

[asfimport]

Antoine Pitrou / @pitrou:
I don't know. One problem is that it will probably always query the us-east-1 region before redirecting, so 1) there is an added latency especially if on the wrong continent 2) perhaps this costs more API calls (I have no idea about the latter).

[asfimport]

Antoine Pitrou / @pitrou:
Note that there is automatic region resolution when you use a URI:

>>> from pyarrow.fs import _resolve_filesystem_and_path
>>> fs, path = _resolve_filesystem_and_path("s3://ursa-labs-taxi-data")
>>> fs
<pyarrow._s3fs.S3FileSystem at 0x7f348cd0b7b0>
>>> fs.region
'us-east-2'
>>> path
'ursa-labs-taxi-data'