ParticleSpecies: Read to dask.dataframe

[ax3l]

Add a method that reads a particle species into a dask.dataframe.

Feel the power 🔥

Cheers to @dmitry-ganyushin for helping with this!

[ax3l]

Some issue if I pass this argument alone the lines "Series has no attribute chunks" deep inside Dask... Not sure if it refers to our series, though - I think not 😅

Also, let's ask the RAPIDS team tomorrow if we can also generate 2D and ND histograms. That would be tremendously helpful, but I cannot spot such a function in the docs.
Update: opened as dask/dask#7307

[ax3l]

It's a dask.dataframe.core.Series that does not have the chunks...

[ax3l]

Adding .to_dask_array() fixes this

[ax3l]

@franzpoeschel I tried to query available_chunks from a constant BaseRecordComponent and realized this throws a backend error.

Probably the cleanest way for us to handle this would be to check for constant() in the frontend and return the full extend as a single chunk in that case, what do you think?

[franzpoeschel]

Ah good catch, yeah that's probably the best solution.

[ax3l]

Implemented in #942 🎉

[franzpoeschel]

Ah good catch, yeah that's probably the best solution.

[franzpoeschel]

This assumes that chunks are equal across components. What happens if they're not? Will things just be less efficient or will things not work? In the latter case, we should probably guard for this case and throw an error.

[ax3l]

Yep, will just be less efficient.
(Also very unlikely.)

[franzpoeschel]

Similarly, this assumes that we are dealing with particle data (and hence 1D). Is this checked? We do have a Python class <openPMD.ParticleSpecies>, so this could theoretically be guarded against.
(Or are those lines enough checking?):

ParticleSpecies.to_df = particles_to_dataframe  # noqa
ParticleSpecies.to_dask = particles_to_daskdataframe  # noqa

[ax3l]

Good idea to check the chunk to be 1D, yep

Since this is implemented as species and ParticleSpecies, there should be little chance to accidentally pass in a field. Particle arrays are always 1D.

[jakirkham]

Would suggest moving this to a module level function instead of a closure. While using closures should work, there's extra overhead pickling closures vs. module level functions

[ax3l]

Thank you for the review & continued guidance!
Fixed in #951

[lgtm-com]

This pull request introduces 1 alert when merging 8c5c367 into 6e3b8b2 - view on LGTM.com

new alerts:

• 1 for Unused import

[ax3l]

2D histograms in dask for a phase space example:

• Adding support for multidimensional histograms with dask.array.histogramdd dask/dask#7387
• https://github.com/xgcm/xhistogram/

[lgtm-com]

This pull request introduces 1 alert when merging 62a5df4 into 24058e0 - view on LGTM.com

new alerts:

• 1 for Unused import

[lgtm-com]

This pull request introduces 1 alert when merging 0f5090a into 24058e0 - view on LGTM.com

new alerts:

• 1 for Unused import

[ax3l]

I think I found what we need for meshes: https://docs.dask.org/en/latest/array-api.html?highlight=from_array#other-functions

dask.array.from_array(x, chunks='auto', name=None, lock=False, asarray=None, fancy=True, getitem=None, meta=None, inline_array=False)

Create dask array from something that looks like an array.

Input must have a .shape, .ndim, .dtype and support numpy-style slicing.

Parameters: |
       x : array_like
  chunks : int, tuple
    How to chunk the array. Must be one of the following forms:
    * A blocksize like 1000.
    * A blockshape like (1000, 1000).
    * Explicit sizes of all blocks along all dimensions like ((1000, 1000, 500), (400, 400)).
    * A size in bytes, like “100 MiB” which will choose a uniform block-like shape
    * The word “auto” which acts like the above, but uses a configuration value array.chunk-size for the chunk size

  -1 or None as a blocksize indicate the size of the corresponding dimension.
...
 asarray : bool, optional
    If True then call np.asarray on chunks to convert them to numpy arrays. If False then chunks are passed through unchanged. If None (default) then we use True if the __array_function__ method is undefined.
...
   fancy : bool, optional
     If x doesn’t support fancy indexing (e.g. indexing with lists or arrays) then set to False.
     Default is True.
...

Not entirely sure yet how to tell it that it needs to call our .flush() once it really reads those blocks, assuming that the read happens delayed... I think we need to write a Record_Component wrapper for Dask that calls .flush() on .asarray()...