Trouble reading files when only the first block has 1 sample

[pscheidler]

First_Block_Examples.zip
In the attached examples, GPS data (channelID 88) in DAQ12497_000050.IDE has a first CDB with only 1 measurement. Subsequent blocks have 112 to 512 samples. Looks like IDELib can't open this file, and the GPS channels won't display in enDAQ Lab.
DAQ12497_000050_rem2.ide has been rearranged so the single sample CDB is moved to the second position (timestamps are also adjusted), and the first block has many samples. That file opens fine.
Notes:

• IDELib can open recordings where all CDBs for a channel are single measurement. This happens for the MPL3115 measurements.
• All CDBs in DAQ12497_000050_rem2.ide are NOT the same size.
• May be an issue with using the first CDB to estimate the data rate?

[StokesMIDE]

All recordings to date contained data blocks that were more-or-less uniformly sized, and if a channel's first block contained 1 sample, all its blocks would also have one sample. Various optimizations were made based on this, and it is going to be non-trivial to modify those optimizations without losing the gains.

[XanthanGum]

Loading an IDE file with endaq.ide.get_doc, where a passed start time generates an EventArray where the first block of the GNSS Location channel (88) has a single sample, causes the following error:

File "C:\...\eda.py", line 3, in <module>
    doc = endaq.ide.get_doc("inputs/DAQ17372_000019.IDE", start="14:00:01", end="14:00:05")
...
File "C:\...\idelib\dataset.py", line 1463, in append
    block.min, block.mean, block.max = mmmArr
ValueError: too many values to unpack (expected 3)

Observed Behaviour

• The first block has numSamples = 1, so _singleSample is set to True.
• Subsequent blocks have numSamples > 1, causing mmmArr to have more than 3 values, which triggers the ValueError.

Example debug output:

block.numsamples: 1
parent: <Channel 102 'GNSS Time': Unix Epoch Reference (s)>
block.numsamples: 1
parent: <Channel 102 'GNSS Time': Unix Epoch Reference (s)>
block.numsamples: 1
parent: <Channel 88 'GNSS Location': Location (Degrees), Location (Degrees), Unix Epoch (s), GNSS Speed (m/s)>
block.numsamples: 22
parent: <Channel 88 'GNSS Location': Location (Degrees), Location (Degrees), Unix Epoch (s), GNSS Speed (m/s)>

The problematic logic in EventArray.append():

# _singleSample hint not explicitly set; set it based on this block.
# There will be problems if the first block has only one sample, but
# future ones don't. This shouldn't happen, though.
if self._singleSample is None:
    self._singleSample = block.numSamples == 1
    if self._parentList is not None:
        self._parentList._singleSample = self._singleSample
    if self.parent.singleSample is None:
        self.parent.singleSample = self._singleSample
    if self.parent.parent is not None:
        self.parent.parent.singleSample = self._singleSample

# HACK: Single-sample-per-block channels get min/mean/max
if self._singleSample is True:
    block.minMeanMax = np.tile(block.payload, 3)
    mmmArr = np_recfunctions.structured_to_unstructured(
            block._minMeanMax.view(self._npType))
    print('block.numsamples:', block.numSamples)
    print('parent:', self.parent)
    block.min, block.mean, block.max = mmmArr
    self.hasMinMeanMax = False
elif block.minMeanMax is not None:
    mmmArr = np_recfunctions.structured_to_unstructured(
        np.frombuffer(block.minMeanMax, self._npType))
    block.min, block.mean, block.max = mmmArr
    self.hasMinMeanMax = True
else:
    vals = np_recfunctions.structured_to_unstructured(block.payload.view(self._npType))
    block.min = vals.min(axis=0)
    block.mean = vals.mean(axis=0)
    block.max = vals.max(axis=0)
    self.hasMinMeanMax = True

Root Cause

• _singleSample is determined from the first block only.
• If the first block has a single sample but subsequent blocks have multiple samples, the code attempts to unpack a larger array (mmmArr) into 3 variables, causing the error.

[StokesMIDE]

@XanthanGum Thanks for your contribution! I'm sorry I haven't yet had time to look at #165 in detail, but I want to be able to give it the attention it deserves. This library is used in dozens of critical scripts we use internally, so I need to do some careful testing against them.