h5py string compatible IO test fails with HDF5 1.12.1

[planetmarshall]

The following test scenario fails

Scenario: testing h5py compatible string IO
Given: a UTF8 string dataset
And given: a vector buffer
Then: we can read the data back without a special data type

Steps to reproduce

GCC 7.5
CMake 3.10.2
HDF5 1.12.1

These are the default tools packaged with Ubuntu 18.04 LTS.

For a Docker image to reproduce this issue see

docker pull algodynamic/ubuntu-dev:18.04

See also https://github.com/planetmarshall/h5cpp/runs/4204396185?check_suite_focus=true

mkdir build
cd build
cmake .. -D WITH_BOOST=False
cmake --build . 
ctest

Expected results

All tests run and pass

Actual Results

The above test fails with

HDF5-DIAG: Error detected in HDF5 (1.12.1) thread 0:
  #000: /tmp/hdf5-hdf5-1_12_1/src/H5Dio.c line 179 in H5Dread(): can't read data
    major: Dataset
    minor: Read failed
  #001: /tmp/hdf5-hdf5-1_12_1/src/H5VLcallback.c line 2011 in H5VL_dataset_read(): dataset read failed
    major: Virtual Object Layer
    minor: Read failed
  #002: /tmp/hdf5-hdf5-1_12_1/src/H5VLcallback.c line 1978 in H5VL__dataset_read(): dataset read failed
    major: Virtual Object Layer
    minor: Read failed
  #003: /tmp/hdf5-hdf5-1_12_1/src/H5VLnative_dataset.c line 166 in H5VL__native_dataset_read(): can't read data
    major: Dataset
    minor: Read failed
  #004: /tmp/hdf5-hdf5-1_12_1/src/H5Dio.c line 403 in H5D__read(): unable to set up type info
    major: Dataset
    minor: Unable to initialize object
  #005: /tmp/hdf5-hdf5-1_12_1/src/H5Dio.c line 923 in H5D__typeinfo_init(): unable to convert between src and dest datatype
    major: Dataset
    minor: Feature is unsupported
  #006: /tmp/hdf5-hdf5-1_12_1/src/H5T.c line 4973 in H5T_path_find(): can't find datatype conversion path
    major: Datatype
    minor: Can't get value
  #007: /tmp/hdf5-hdf5-1_12_1/src/H5T.c line 5195 in H5T__path_find_real(): no appropriate function for conversion path
    major: Datatype
    minor: Unable to initialize object

/tmp/h5cpp/test/node/dataset_h5py_string_compat_test.cpp:75: Failure:
due to unexpected exception with message:
  Failure to read variable length string data from dataset [/utf8_string]!

[jkotan]

Hi @planetmarshall,
Indeed. It looks like the issue comes withhdf5 1.12, i.e. it looks like you cannot read UFT8 strings to one character string type anymore. Probably we need a new Type trait with i.e. UTFString.
I propose to limit your workaround #501 (comment) only to strings

diff --git a/src/h5cpp/node/dataset.hpp b/src/h5cpp/node/dataset.hpp
index 7270d4bf..0565cb2a 100644
--- a/src/h5cpp/node/dataset.hpp
+++ b/src/h5cpp/node/dataset.hpp
@@ -1178,9 +1178,14 @@ void Dataset::write_reshape(const T &data,
 template<typename T>
 void Dataset::read(T &data,const property::DatasetTransferList &dtpl)
 {
-  hdf5::datatype::DatatypeHolder mem_type_holder;
-  hdf5::dataspace::DataspaceHolder mem_space_holder(space_pool);
-  read_reshape(data, mem_type_holder.get(data), mem_space_holder.get(data), dtpl);
+  hdf5::dataspace::DataspaceHolder mem_space_holder(space_pool);
+  if(file_type_class == datatype::Class::String){
+    read_reshape(data, file_type, mem_space_holder.get(data), dtpl);
+  }
+  else {
+    hdf5::datatype::DatatypeHolder mem_type_holder;
+    read_reshape(data, mem_type_holder.get(data), mem_space_holder.get(data), dtpl);
+  }
 }
 
 template<typename T>

What do you think?

BTW, it would be good to add a test instance on the ESS Jenkins node with hdf5 1.12.

[jkotan]

I've rerun tests with hdf5 1.12.0 on ESS Jenkins (with the Eugen's fix for #506) and I don't get the above error
https://jenkins.esss.dk/dm/blue/organizations/jenkins/ess-dmsc%2Fh5cpp/detail/hdf5_1_12/3/pipeline/32
(ubuntu 18.04 tests do not work because of conan problems).
So it does not look like v1.12 issue (maybe 1.12.1 or something else like ubuntu 18.04 issue). It needs to be checked.

[jkotan]

I've reproduced the error in your docker container. The problem appears then the encoding of file_type (UTF8) and created memory type (default ASCII) are different. Changing encoding of the memory type in this case from ASCII to UTF8 works.
So the above workaround works as well.

[jkotan]

@planetmarshall would you like to create PR or shall I do it?

[planetmarshall]

This appears to be a change from 1.12.0 -> 1.12.1

@planetmarshall would you like to create PR or shall I do it?

Sure, go ahead. Thanks.

[jkotan]

It looks like it is related to HDFGroup/hdf5#544