[Python] ArrowInvalid error on reading partitioned parquet files from S3 (arrow-2.0.0)

[asfimport]

Hello

It looks like pyarrow-2.0.0 could not read partitioned datasets from S3 buckets: 

import s3fs
import pyarrow as pa
import pyarrow.parquet as pq

filesystem = s3fs.S3FileSystem()

d = pd.date_range('1990-01-01', freq='D', periods=10000)
vals = np.random.randn(len(d), 4)
x = pd.DataFrame(vals, index=d, columns=['A', 'B', 'C', 'D'])
x['Year'] = x.index.year

table = pa.Table.from_pandas(x, preserve_index=True)
pq.write_to_dataset(table, root_path='s3://bucket/test_pyarrow.parquet', partition_cols=['Year'], filesystem=filesystem)

 

 Now, reading it via pq.read_table:

pq.read_table('s3://bucket/test_pyarrow.parquet', filesystem=filesystem, use_pandas_metadata=True)

Raises exception: 

ArrowInvalid: GetFileInfo() yielded path 'bucket/test_pyarrow.parquet/Year=2017/ffcc136787cf46a18e8cc8f72958453f.parquet', which is outside base dir 's3://bucket/test_pyarrow.parquet'

 

Direct read in pandas:

pd.read_parquet('s3://bucket/test_pyarrow.parquet')

returns empty DataFrame.

 

The issue does not exist in pyarrow-1.0.1

Reporter: Vladimir

Related issues:

• [C++] Filesystems: detect if URI is passed where a file path is required and raise informative error (relates to)

_{Note: This issue was originally created as ARROW-10937. Please see the migration documentation for further details.}

[asfimport]

Antoine Pitrou / @pitrou:
cc @jorisvandenbossche

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Not necessarily related, but ARROW-10264 has a similar error message.

[~Filimonov] Thanks for the report!
Could you try something like this:

filesystem = s3fs.S3FileSystem()

from pyarrow.fs import FileSelector, PyFileSystem, FSSpecHandler
fs = PyFileSystem(FSSpecHandler(filesystem))
selector = FileSelector("s3://bucket/test_pyarrow.parquet", recursive=True)
fs.get_file_info(selector)

and tell us what it returns?
(note I didn't try that code, so there might be a mistake in it, but think it should work)

[asfimport]

Antoine Pitrou / @pitrou:
@jorisvandenbossche didn't you fix this issue at some point?

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Actually, @pitrou, I suppose the problem is that the get_file_info returns a string like "bucket/test_pyarrow.parquet/...", while the selector's base_dir is "s3://bucket/test_pyarrow.parquet". So the RemoveAncestor utility doesn't work, and we raise this error here:

arrow/cpp/src/arrow/dataset/discovery.cc

Lines 188 to 191 in 06ac750

	auto relative = fs::internal::RemoveAncestor(selector.base_dir, info.path());
	if (!relative.has_value()) {
	st = Status::Invalid("GetFileInfo() yielded path '", info.path(),
	"', which is outside base dir '", selector.base_dir, "'");

What does the filesystem layer expect here us to do? Should the base_dir have been stripped of the "s3://" before creating a Selector of it?

[asfimport]

Antoine Pitrou / @pitrou:

Should the base_dir have been stripped of the "s3://" before creating a Selector of it?

Yes, it should, and I believe this was already fixed.

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
[~Filimonov] can you also try pq.read_table('bucket/test_pyarrow.parquet', filesystem=filesystem, use_pandas_metadata=True) (so without the "s3://" part in the file path) ?
Since you are passing a filesystem object, you don't need to pass a URI for the file path. This might actually be the reason of the failure, although I think we should solve it on the arrow side as well (or at least detect the case and give a nice error message, as otherwise many users will run into this without a good clue what they are doing wrong).

[asfimport]

Vladimir:
Hello @jorisvandenbossche, @pitrou

On your questions:

1. Calling "fs.get_file_info" returns the full list of files in both pyarrow-1.0.1 and 2.0.0:

[<FileInfo for 'bucket/test_pyarrow.parquet/Year=1990/6583990469864a579b4a7a579b81bec4.parquet': type=FileType.File, size=20402>,
 <FileInfo for 'bucket/test_pyarrow.parquet/Year=1991/03ab50a5f5c9449bb10c8440358c7e35.parquet': type=FileType.File, size=20404>,
...

2. Calling pq.read_table without prefix "s3://" prefix ("pq.read_table('bucket/test_pyarrow.parquet', filesystem=filesystem, use_pandas_metadata=True)" returns empty pyarrow.Table - also both in 1.0.1 and 2.0.0

3. Correction to my original description: reading partitioned dataset via pandas as 

pd.read_parquet('s3://bucket/test_pyarrow.parquet')

returns empty dataframe in both 1.0.1 and 2.0.0. If I supply "filesystem" argument to pd.read_parquet then 1.0.1 reads dataset properly, and 2.0.0 raises the same ArrowInvalid exception.

 

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Thanks [~Filimonov]

So your original error shown at the top post, is indeed because of passing a file path with starting with "s3://..." while also passing a filesystem object. IMO we should improve the handling of that case (or at least improve the error message).

Now, that you get an empty table when removing the prefix is strange. Certainly given that listing the files seems to work, and shows that there are actually files in that folder.

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
I opened ARROW-10998 for the issue that we should give a clearer error message when passing a URI and specifying a filesystem object.

[asfimport]

Antoine Pitrou / @pitrou:
Does this need to be kept open? It seems the ARROW-10998 covers the "better error message" issue.