Distributed 2021.3.1 `distributed.protocol.serialize.pickle_loads` fails with `IndexError: tuple index out of range`

[gabicca]

What happened:

The following exception occred with the latest version of distributed, in a test that has previously passed:

header = {'compression': (None, None), 'num-sub-frames': 2, 'serializer': 'pickle', 'split-num-sub-frames': (1, 1), ...}
frames = [<memory at 0x1209deae0>, <memory at 0x1209dea10>]
    def pickle_loads(header, frames):
        x, buffers = frames[0], frames[1:]
        writeable = header["writeable"]
        for i in range(len(buffers)):
            mv = memoryview(buffers[i])
>           if writeable[i] == mv.readonly:
E           IndexError: tuple index out of range

"writeable" is an empty tuple in the above header.

What you expected to happen:

After digging a bit and comparing runs of the same test between 2021.3.0 and 2021.3.1, I found the following:

In version 2021.3.0
the input frames always has one element, hence buffers is always an empty list --> so the for loop, which contains writeable[i] never runs; writable is always an empty tuple

In version 2021.3.1
the third time it gets to this function, frames has 2 elements, hence buffers is not empty, and the for loop is executed; writable is still an empty tuple, hence code fails.

I saw that there were substantial changes to distributed.protocol.core.loads, where frames is passed down in its "truncated" from (sub_frames) to the function which eventually breaks. I don't know if this is a bug introduced, or our code needs changing. I'm not familiar with the underlying mechanisms, so I'd appreciate if someone could take a look.

Environment:

• Dask version: 2021.3.1
• Python version: 3.7.10
• Operating System: MacOS Mojave (but also fails on linux-based gitlab runners)
• Install method (conda, pip, source): pip

[jrbourbeau]

Thanks for raising an issue @gabicca. cc @jakirkham @madsbk

Are you able to provide a code snippet which reproduces the IndexError?

[gabicca]

Thanks for raising an issue @gabicca. cc @jakirkham @madsbk

Are you able to provide a code snippet which reproduces the IndexError?

I don't really understand the code to be honest, to get a working quick example. But I will spend a bit more time on it tomorrow and try to come up with one. But please don't wait for me with this. I'll let you know how it goes.

[jakirkham]

Are you able to run with pdb @gabicca? It would be interesting to know what writeable is here

[gabicca]

Are you able to run with pdb @gabicca? It would be interesting to know what writeable is here

Hi @jakirkham , writeabel is always an empty tuple, as I say in the description, hence the index error. The difference between the two executions, from what I can tell, was that while in the old version frames is always a single-element list, in the new version sometimes it has multiple elements. So in the first virsion the code never enterred the for loop, because buffers was an empty list. While in the new version, it enterred the loop and failed with writeable being empty.

[gabicca]

I can't get a simple example together. I'm happy to hop on a call with someone if we can share connection details privately somehow.

[alejandrofiel]

Hi, the same error here with this code. rollbacking to 2021.3.0 solves the problem

import dask.dataframe as dd
from dask.distributed import Client, LocalCluster

cluster = LocalCluster(n_workers=39, threads_per_worker=1)
client = Client(cluster)

ddf = dd.read_csv(f's3://xxxxxx/*.csv',
                      parse_dates=['test123'],
                      assume_missing=True,
                      storage_options={'config_kwargs': {'max_pool_connections': 20}}

df = ddf.compute()

Thanks!

edit:
I'll add this is similar to this: #4625
And the closer diff I saw was here: #4595

[Cedric-Magnan]

Hello, the same error happens when dask distributed reads multiple parquet files with the same schema (but without any parquet metadata, I haven't tried with parquet metadata).

The following code works well in dask-2021.3.1 and dask-2021.3.0 :

import dask.dataframe as dd

ddf = dd.read_parquet(f's3://xxxxxx/*.parquet.gzip')
df = ddf.compute()

The following code works in dask-2021.2.0 but not in dask-2021.3.1 and dask-2021.3.0 (because both versions install distributed-2021.3.1) :

from dask.distributed import Client
import dask.dataframe as dd

client = Client()
ddf = dd.read_parquet(f's3://xxxxxx/*.parquet.gzip')
df = ddf.compute()

[mrocklin]

Hi Everyone,

Thank you for reporting this. We'll get a fix in soon and issue a bugfix release.

However, it would be really helpful to develop a test here to ensure that this problem doesn't recur in the future. To the extent that people are able to reduce their examples to make them more reproducible by others that would be welcome. None of the test cases in the current test suite run into this problem, and so we're a bit blind at the moment.

For example, all of the stated examples talk about reading from S3. Does this problem occur if you're not reading data from S3? Does it require gzip? Does it require all of the keyword parameters that you're passing in? Does it go away if you remove a specific one of these? @Cedric-Magnan @alejandrofiel @gabicca you all are currently the best placed people to help us identify the challenge here. If you are able to reduce your problem to something that someone else can run that would be very helpful.

Given the information that you've provided so far I've tried to reproduce this issue by reading a CSV data from a public S3 dataset. This is the code that I use for that.

from dask.distributed import Client
client = Client()

df = dd.read_csv(
    "s3://nyc-tlc/trip data/yellow_tripdata_2019-*.csv",
    storage_options={"anon": True},
)
df.head()

Does this fail for you by any chance?

• If so, great, can you share more about your environment to help us reproduce your setup. Maybe a pip freeze or conda list output. If you have the time to craft a minimal environment then even better
• If not, then are you able to try to figure out what is different between what you're doing in your failing example and what this example is doing? Is it the dataset? Some keyword argument?

Thanks all

[mrocklin]

@williamBlazing I see that you've also reported something similar. If you or your team are able to help provide a reproducer that would be welcome.

[wmalpica]

@williamBlazing I see that you've also reported something similar. If you or your team are able to help provide a reproducer that would be welcome.

@mrocklin our reproducer is here dask/dask#7490