Inject dataset reading via Hdf5Reader.

[1uc]

The idea is that one injects call-backs for reading selections from datasets. These callbacks implement:

template<class T>
std::vector<T> readSelection(const HighFive::DataSet& dset, Selection& selection);
HighFive::File open_file(const std::string& filename);

The default calls _readSelection. Separate readers can implement variants suitable for their purpose, e.g. MPI-IO. The advantage is that the reader has control over both the required collective semantics; and setting HDF5 properties. This allows some readers to have non-collective behaviour, such as short-circuiting for empty selections. While others implement strictly collective reading of datasets.

Additionally, we can create a suite of readers for different usecases, e.g. MPI-IO for neurodamus, aggregating reader for serial workloads with GPFS (and a separate one for LUSTER if need be), the default implementation because it minimizes number of bytes read.

[ferdonline]

When SONATA_HAS_MPI is False I would rather support we dont even expose collective stuff, as it might be misleading to ask for a collective reader and get a normal one.

[1uc]

Summary: I think it's important that MPI is entirely optional in libsonata, even when writing code that wants to use collective MPI-IO. At most, I'd add a flag that prevents the returning a suitable default.

The lines aren't precise because likely we'll move anything MPI to its own repo. The reason is that mpi4py is a build dependency; and one can't have optional build dependencies (nicely).

Nevertheless, I think code that's written for collective IO should work even if the user fails to install mpi4py. HDF5 doesn't fail either if it can't do collective IO, instead it returns the correct values and sets an internal flag users can check. Functionally collective IO is optional. It's only important for performance reasons under certain conditions.

The risk is that installing mpi4py or libsonata with MPI support will fail to build under certain conditions. I suspect many of our users will be unable or unwilling to debug these failures. To me it's a big difference if what I'm running is a large production run that requires MPI-IO to run reasonably fast, or if it's something small (debugging) that runs fast simply because it's small. Large runs are always done on a cluster and we can expect someone adept at these issues to install the required toolchain (including MPI-enabled libsonata), even if not, there's a necessity and therefore it makes sense to spend time to debug why something failed to build. Small runs are carried out anywhere, including laptops where MPI might not be installed or it might not work correctly at that moment, etc. In my opinion I don't want to force users to have to figure these issues out just to run something that takes 2.3s with MPI-IO and 2.2s without MPI-IO. Similarly, I don't want every downstream project to remember to implement a graceful fallback.

To detect if the important applications are using MPI-IO we can:

• check GPFS waiters,
• run Darshan,
• use ROMIO_PRINT_HINTS=1 and check the output, e.g., during module testing on the BB5.

Therefore, I think, on BB5, we're in a sufficiently strong position to notice if it's silently not using collective IO when it really should be.

[ferdonline]

Interesting design. As I understand you are implementing a collective reader already, so in Neurodamus we would need basically to call make_collective_reader and use it as argument in calls to the config which return storage. I think this is fine.

[matz-e]

Nice! If I get this correct, we can just pull out the collective read interface into it's own library?

[1uc]

Nice! If I get this correct, we can just pull out the collective read interface into it's own library?

Yes, almost certainly, because mpi4py is a build dependency; and one can't have optional build dependencies (nicely).

[1uc]

This PR has been heavily reworked since the previous round of discussions. Eventually it will only include the Hdf5Reader API. The two commits that refactor libsonata have been split into their own PRs. The implementation of an MPI-IO reader has been moved to a separate PR (until it can be moved to its own repository).

[1uc]

The questions for review are:

1. Are the rules regarding collectiveness of libsonata acceptable?
2. Do we need the ability to pass options to the reader on getAttribute and similar? Currently, I can only think of collective which would be used to specify if the call is collective. Things like block size, I fell would go into the reader.
3. Naming of Hdf5Reader.

The next review would include question like:

1. Can we have an optional dependency libsonata[mpi]? The advantage is that libsonata can control the version of libsonata_mpi it's compatible with, which is useful if the API of Hdf5Reader needs to change.
2. Can we have wrappers for libsonata_mpi.make_collective_reader in libsonata? The advantage is that one could then always do libsonata.make_collective_reader which would always succeed. If libsonata_mpi is available it will return a collective reader, otherwise it'll return the default reader.

With the above, code written to use collective IO, should automatically run even if the user can't install mpi4py or libsonata_mpi. Therefore, users can avoid the pain of installing the difficult dependencies; and devs don't need to worry about making it easy for their users.

[mgeplf]

LGTM, let's get #319 merged, and then hopefully this

[mgeplf]

Last chance for comments @matz-e / @ferdonline; otherwise I'll approve this at the end of the day.

[1uc]

@mgeplf we're in a reasonably good place to hold off on merging this. With all the preliminary work out of the way, we'll not suffer too much from merge conflicts and we can work on multiple other things (like cleaning up a bit more after #319).

We only need this in once we have it fully working and tested in neurodamus. Until then retaining the freedom to change details in might be nice.