Skip to content

blosc_extension.error: Error -1 while decompressing data #212

@nikhilshinday

Description

@nikhilshinday

Some memory corruption is happening in my use case for Blosc. Blosc is being used to store numpy ndarrays in shared memory using plasma. It is unclear whether the issue is in compression/decompression or in writing/reading to shared memory. Here are my write/read methods:

def write_blosc(client: plasma.PlasmaClient, x: np.ndarray):
    object_id = plasma.ObjectID(np.random.bytes(20))
    buf = blosc.pack_array(x)
    client.create_and_seal(object_id, buf)
    return object_id


def read_blosc(client: plasma.PlasmaClient, object_id: plasma.ObjectID):
    [buf] = client.get_buffers([object_id], timeout_ms=100)
    res = blosc.unpack_array(buf.to_pybytes())
    return res

Numpy arrays are able to be successfully read and written most of the time, but occasionally (maybe 1 call out of every 10,000 to the read function) the error in the title shows up.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions