HDF5: datasets that get their data from NDAttributes are initially created with the size of numArrays.

[MarkRivers]

datasets that get their data from NDAttributes are initially created with the size of numArrays. This is not correct, they should not be statically allocated they should grow in an unlimited dimension as new NDArrays arrive, just like the detector data does. The user should be able to set numArrays to a very large value and then stop after the desired time without causing the dataset to be unnecessarily large. This is how the netCDF plugin works. This may also why it crashes when setting numArrays to 0.

[MarkRivers]

Here is an example using test2_attributes.xml and test2_layout.xml. I set NumCapture=10000, FileWriteMode="Single" and set WriteFile=1 (Save button). Here is the output from h5dump:
DATASET "data" {
DATATYPE H5T_STD_U8LE
DATASPACE SIMPLE { ( 10, 10 ) / ( 10, 10 ) }

     DATASET "integral" {
        DATATYPE  H5T_IEEE_F64LE
        DATASPACE  SIMPLE { ( 10000 ) / ( 10000 ) }
        DATA {
        (0): 0.001, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
        (19): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,

...
DATASET "preset" {
DATATYPE H5T_IEEE_F64LE
DATASPACE SIMPLE { ( 10000 ) / ( 10000 ) }
DATA {
(0): 0.001, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
(19): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
...
DATASET "ColorMode" {
DATATYPE H5T_STD_I32LE
DATASPACE SIMPLE { ( 10000 ) / ( 10000 ) }
DATA {
(0): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
(19): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,

Notice that even though I was in Single mode all of these datasets are of size 10000. Even in Stream mode these datasets should not be preallocated to NumCapture, they should just have the dimension of the number of NDArrays actually in the file.

[MarkRivers]

Note that the title of this issue has a mistake, numArrays should be NumCapture.

This is an example when NumCapture=10000, FileWriteMode=Stream, and file writing was stopped after 6 NDArrays had been saved:
DATASET "data" {
DATATYPE H5T_STD_U8LE
DATASPACE SIMPLE { ( 6, 10, 10 ) / ( H5S_UNLIMITED, 10, 10 ) }
DATA {
(0,0,0): 130, 131, 132, 133, 134, 135, 136, 137, 138, 139,
(0,1,0): 131, 132, 133, 134, 135, 136, 137, 138, 139, 140,
(0,2,0): 132, 133, 134, 135, 136, 137, 138, 139, 140, 141,
(0,3,0): 133, 134, 135, 136, 137, 138, 139, 140, 141, 142,
(0,4,0): 134, 135, 136, 137, 138, 139, 140, 141, 142, 143,

        DATASET "NDArrayUniqueId" {
           DATATYPE  H5T_STD_I32LE
           DATASPACE  SIMPLE { ( 10000 ) / ( 10000 ) }
           DATA {
           (0): 22915, 22916, 22917, 22918, 22919, 22920, 0, 0, 0, 0, 0,
           (11): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           (30): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           (49): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           (68): 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,

Note that the detector dataset has dimension H5D_UNLIMITED and an actual size of 6. UniqueId on the other hand has a dimension of 10000, of which only the first 6 elements have valid data. UniqueId should also be declared with H5S_UNLIMITED and grow with the number of arrays. This is how the netCDF file plugin works.

[ulrikpedersen]

Fix in hdf5-issues branch PR #85