[Python] add option for taking all columns from all files in pa.dataset

[asfimport]

In PyArrow's dataset class, if I give it multiple parquet files in a list and these parquet files have potentially different columns, it will always take the schema from the first parquet file in the list, thus ignoring columns that the first file doesn't have. Getting all columns within the files into the same dataset implies passing a manual schema or constructing one by iterating over the files and checking for their columns.

 

Would be nicer if PyArrow's dataset class could have an option to automatically take all columns within the files from which it is constructed.

import numpy as np, pandas as pd
df1 = pd.DataFrame({
    "col1" : np.arange(10),
    "col2" : np.random.choice(["a", "b"], size=10)
})
df2 = pd.DataFrame({
    "col1" : np.arange(10, 20),
    "col3" : np.random.random(size=10)
})
df1.to_parquet("df1.parquet")
df2.to_parquet("df2.parquet")

import pyarrow.dataset as pds
ff = ["df1.parquet", "df2.parquet"]

### Code below will generate a DF with col1 and col2, but no col3

pds.dataset(ff, format="parquet").to_table().to_pandas()

 

 

Reporter: David Cortes

Related issues:

• [Python][Dataset] Expose schema inference / validation options in the factory (is duplicated by)

_{Note: This issue was originally created as ARROW-9455. Please see the migration documentation for further details.}

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
[~david-cortes] thanks for the report!

Yes, currently the schema is inferred from the 'first' file, and you can only override this by providing the full schema manually.

There are however options to specify the number of files/fragments to infer from, or to infer from all files. This functionality already exists in C++, but is not yet exposed in Python. See ARROW-8221 for that.

Closing as a duplicate of ARROW-8221, but feedback on that issue on a good API for this is very welcome!