[Python] Error with errno 22 when loading 3.6 GB Parquet file

[asfimport]

I saved a file using pandas to_parquet method, but can't read it back in. Here's the full stack trace:

 

Traceback (most recent call last):
File "src/data/CLXP_pull.py", line 214, in <module>
 main()
 File "/Users/mm51929/projects/2018/03-advisor-recruiting/pyenv/lib/python3.6/site-packages/click/core.py", line 722, in _call_
 return self.main(*args, **kwargs)
 File "/Users/mm51929/projects/2018/03-advisor-recruiting/pyenv/lib/python3.6/site-packages/click/core.py", line 697, in main
 rv = self.invoke(ctx)
 File "/Users/mm51929/projects/2018/03-advisor-recruiting/pyenv/lib/python3.6/site-packages/click/core.py", line 895, in invoke
 return ctx.invoke(self.callback, **ctx.params)
 File "/Users/mm51929/projects/2018/03-advisor-recruiting/pyenv/lib/python3.6/site-packages/click/core.py", line 535, in invoke
 return callback(*args, **kwargs)
 File "src/data/CLXP_pull.py", line 188, in main
 results[fullname] = pd.read_parquet(os.path.join(project_dir, "data", "raw", fullname+".parquet"), engine="pyarrow")
 File "/Users/mm51929/projects/2018/03-advisor-recruiting/pyenv/lib/python3.6/site-packages/pandas/io/parquet.py", line 257, in read_parquet
 return impl.read(path, columns=columns, **kwargs)
 File "/Users/mm51929/projects/2018/03-advisor-recruiting/pyenv/lib/python3.6/site-packages/pandas/io/parquet.py", line 130, in read
 **kwargs).to_pandas()
 File "/Users/mm51929/projects/2018/03-advisor-recruiting/pyenv/lib/python3.6/site-packages/pyarrow/parquet.py", line 939, in read_table
 pf = ParquetFile(source, metadata=metadata)
 File "/Users/mm51929/projects/2018/03-advisor-recruiting/pyenv/lib/python3.6/site-packages/pyarrow/parquet.py", line 64, in _init_
 self.reader.open(source, metadata=metadata)
 File "_parquet.pyx", line 651, in pyarrow._parquet.ParquetReader.open
 File "error.pxi", line 79, in pyarrow.lib.check_status
 pyarrow.lib.ArrowIOError: Arrow error: IOError: [Errno 22] Invalid argument

Any ideas what could cause this? The file itself is 3.6GB.

I'm running pandas==0.22.0.

Reporter: Andy Reagan
Assignee: Wes McKinney / @wesm

Related issues:

• [C++] Parquet arrow::Table reads error when overflowing capacity of BinaryArray (duplicates)

_{Note: This issue was originally created as ARROW-2654. Please see the migration documentation for further details.}

[asfimport]

Andy Reagan:
I upgraded to the latest version of Pandas (0.23.0), which now has pyarrow==0.9.0.post1, and I get basically the same

Traceback (most recent call last): File "src/data/CLXP_pull.py", line 214, in <module> main() File "/Users/mm51929/projects/2018/03-advisor-recruiting/pyenv/lib/python3.6/site-packages/click/core.py", line 722, in __call__ return self.main(*args, **kwargs) File "/Users/mm51929/projects/2018/03-advisor-recruiting/pyenv/lib/python3.6/site-packages/click/core.py", line 697, in main rv = self.invoke(ctx) File "/Users/mm51929/projects/2018/03-advisor-recruiting/pyenv/lib/python3.6/site-packages/click/core.py", line 895, in invoke return ctx.invoke(self.callback, **ctx.params) File "/Users/mm51929/projects/2018/03-advisor-recruiting/pyenv/lib/python3.6/site-packages/click/core.py", line 535, in invoke return callback(*args, **kwargs) File "src/data/CLXP_pull.py", line 188, in main results[fullname] = pd.read_parquet(os.path.join(project_dir, "data", "raw", fullname+".parquet"), engine="pyarrow") File "/Users/mm51929/projects/2018/03-advisor-recruiting/pyenv/lib/python3.6/site-packages/pandas/io/parquet.py", line 288, in read_parquet return impl.read(path, columns=columns, **kwargs) File "/Users/mm51929/projects/2018/03-advisor-recruiting/pyenv/lib/python3.6/site-packages/pandas/io/parquet.py", line 131, in read **kwargs).to_pandas() File "/Users/mm51929/projects/2018/03-advisor-recruiting/pyenv/lib/python3.6/site-packages/pyarrow/parquet.py", line 939, in read_table pf = ParquetFile(source, metadata=metadata) File "/Users/mm51929/projects/2018/03-advisor-recruiting/pyenv/lib/python3.6/site-packages/pyarrow/parquet.py", line 64, in __init__ self.reader.open(source, metadata=metadata) File "_parquet.pyx", line 651, in pyarrow._parquet.ParquetReader.open File "error.pxi", line 79, in pyarrow.lib.check_status pyarrow.lib.ArrowIOError: Arrow error: IOError: [Errno 22] Invalid argument

[asfimport]

Wes McKinney / @wesm:
Any chance you can provide a minimally reproducible example? 

[asfimport]

Andy Reagan:
Yeah, I realize there's not as much to be done without it.

The data is confidential, so I can't just provide a copy (which is why I didn't provide right away), but I'll see what I can do (at least, maybe I can narrow it down to a particular row and scramble that data).

[asfimport]

Antoine Pitrou / @pitrou:
Does it work if you make the file smaller than 2GB? This might be a 64-bitness issue (which may actually be fixed in git master).

[asfimport]

Wes McKinney / @wesm:
[~andyreagan] where is the data stored? The error suggests that the mmap call failed, but without more detail it's hard for me to tell. Can you please test using the appropriate wheel for your platform from

https://github.com/kszucs/crossbow/releases/tag/build-163

? I'm moving this issue off the 0.10.0 for now

I noticed that it's not possible to disable memory mapping when reading Parquet files: opened ARROW-2807

[asfimport]

Antoine Pitrou / @pitrou:
[~andyreagan] Can you upgrade again to Arrow 0.10.0 and see whether the issues is fixed?

[asfimport]

Andy Reagan:
I get a different error now on 0.10.0. Here is the full stacktrace:

~/projects/2018/03-advisor-recruiting/pyenv/lib/python3.6/site-packages/pandas/io/parquet.py in read_parquet(path, engine, columns, **kwargs)
286
287 impl = get_engine(engine)
--> 288 return impl.read(path, columns=columns, **kwargs)

~/projects/2018/03-advisor-recruiting/pyenv/lib/python3.6/site-packages/pandas/io/parquet.py in read(self, path, columns, **kwargs)
129 kwargs['use_pandas_metadata'] = True
130 result = self.api.parquet.read_table(path, columns=columns,
--> 131 **kwargs).to_pandas()
132 if should_close:
133 try:

~/projects/2018/03-advisor-recruiting/pyenv/lib/python3.6/site-packages/pyarrow/parquet.py in read_table(source, columns, nthreads, metadata, use_pandas_metadata)
1044 fs = _get_fs_from_path(source)
1045 return fs.read_parquet(source, columns=columns, metadata=metadata,
-> 1046 use_pandas_metadata=use_pandas_metadata)
1047
1048 pf = ParquetFile(source, metadata=metadata)

~/projects/2018/03-advisor-recruiting/pyenv/lib/python3.6/site-packages/pyarrow/filesystem.py in read_parquet(self, path, columns, metadata, schema, nthreads, use_pandas_metadata)
175 filesystem=self)
176 return dataset.read(columns=columns, nthreads=nthreads,
--> 177 use_pandas_metadata=use_pandas_metadata)
178
179 def open(self, path, mode='rb'):

~/projects/2018/03-advisor-recruiting/pyenv/lib/python3.6/site-packages/pyarrow/parquet.py in read(self, columns, nthreads, use_pandas_metadata)
896 partitions=self.partitions,
897 open_file_func=open_file,
--> 898 use_pandas_metadata=use_pandas_metadata)
899 tables.append(table)
900

~/projects/2018/03-advisor-recruiting/pyenv/lib/python3.6/site-packages/pyarrow/parquet.py in read(self, columns, nthreads, partitions, open_file_func, file, use_pandas_metadata)
459 table = reader.read_row_group(self.row_group, **options)
460 else:
--> 461 table = reader.read(**options)
462
463 if len(self.partition_keys) > 0:

~/projects/2018/03-advisor-recruiting/pyenv/lib/python3.6/site-packages/pyarrow/parquet.py in read(self, columns, nthreads, use_pandas_metadata)
150 columns, use_pandas_metadata=use_pandas_metadata)
151 return self.reader.read_all(column_indices=column_indices,
--> 152 nthreads=nthreads)
153
154 def scan_contents(self, columns=None, batch_size=65536):

~/projects/2018/03-advisor-recruiting/pyenv/lib/python3.6/site-packages/pyarrow/_parquet.pyx in pyarrow._parquet.ParquetReader.read_all()

~/projects/2018/03-advisor-recruiting/pyenv/lib/python3.6/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

ArrowIOError: Arrow error: Capacity error: BinaryArray cannot contain more than 2147483646 bytes, have 2147484002

 

[asfimport]

Wes McKinney / @wesm:
I think this issue has been reported elsewhere. I'm not going to have a chance to investigate in the next couple of weeks but if anyone wants to look in the meantime, producing a chunked array in this case should be similar to the way we handle such cases when converting to Arrow from pandas

[asfimport]

Wes McKinney / @wesm:
This will have to be tackled in the 0.12 release cycle if no one gets to it in time for 0.11

[asfimport]

Wes McKinney / @wesm:
Closing as duplicate of ARROW-3762. That issue contains a repro so we'll use that one for fixing the issue

[asfimport]

Andy Reagan:
👌

Sent from my Samsung Galaxy smartphone.

-------- Original message --------
From: "Wes McKinney (JIRA)" jira@apache.org
Date: 11/21/18 12:46 PM (GMT-05:00)
To: "Reagan, Andrew" AReagan@MassMutual.com
Subject: [EXTERNAL][jira] [Closed] (ARROW-2654) [Python] Error with errno 22 when loading 3.6 GB Parquet file

 [ https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_ARROW-2D2654-3Fpage-3Dcom.atlassian.jira.plugin.system.issuetabpanels-3Aall-2Dtabpanel&d=DwIFaQ&c=BX7Y4KpGhcDnIsrgHKqkbfoiiDvjhxwuYUpcrPD7xrE&r=a3UJQ6cVzThaj2bmSG-EKQHCpiQSdhnF6mf5fZPUK8Y&m=w_LcK_-uCZKvZ2zgobYJW5GfFcksTlVY0r8YUPcZnDM&s=0yFlBigppaVMfLgXwrIzmnIsCEK2cnpyQ139gtCKTZs&e= ]

Wes McKinney closed ARROW-2654.

Resolution: Duplicate
  Assignee: Wes McKinney

Closing as duplicate of ARROW-3762. That issue contains a repro so we'll use that one for fixing the issue

–
This message was sent by Atlassian JIRA
(v7.6.3#76005)

This e-mail transmission may contain information that is proprietary, privileged and/or confidential and is intended exclusively for the person(s) to whom it is addressed. Any use, copying, retention or disclosure by any person other than the intended recipient or the intended recipient's designees is strictly prohibited. If you are not the intended recipient or their designee, please notify the sender immediately by return e-mail and delete all copies