RevD Poll Issue with empty slots

[ksmith0]

From Nathan Brewer:

Hello,

On RevD pixie there had been an issue with a particular situation.

We have a reason to attach signals in modules 0,1,2 and 4, skipping 3.

Poll2 in pacman mode
this lead to corrupted data for module 4.
and failed on the slotRead != slotExpected check.

If, however, at the beginning of Poll::ReadFIFO in the "Loop over each module's FIFO" I put a check for the explicit case that nWords[mod]==0 and continue without writing an empty buffer ie. simply:

if (nWords[mod]==0)
continue;

Then it seems to run without error.
In your opinion is this sensible?

(btw. there was a hack in previous versions of poll for this kind of error: ie slotRead = slotExpected before checking, which I think is undesirable. )

If this is sensible I or you can update the git.

[ksmith0]

Interesting, this should be handled by the section of the code that reads:

//if the module has no words in the FIFO we continue to the next module
if (nWords[mod] < MIN_FIFO_READ) {
// write an empty buffer if there is no data
fifoData[dataWords++] = 2;
fifoData[dataWords++] = mod;    
continue;
}

I have a suspicion that slotRead is reporting module number and not slot number. This is something I've observed at the NSCL as well. Was there a firmware change ?

If your fix is working for you and data is reliable I don't see why you shouldn't run that way temporarily. Unfortunately, it may be a while until we can investigate this due to an upcoming experiment. Is there a particular critical deadline your are working towards or will the current fix if reliable be sufficient for the time being?

[ksmith0]

From Nathan Brewer:

I'll open a git issue then, with more details.

'I have a suspicion that slotRead is reporting module number and not slot number"

Nope. Details to come.

New stuff needs to be tested for RevD is all. There isn't a pressing deadline, because we old poll until new poll is tested. I just haven't gone through it all yet. But this summer is a good time for these tests. We'll see what we can do.

Further issues I will just send directly to git.

Thanks!
N

[ntbrewer]

Problem Definition:
REVD. slotRead not slotExpected.

If a given module is empty and is ordered in between non-empty modules.
(this could be done if a slot gets burnt or as in our case,
the module has a configuration to be used again
and it was desired not to be changed and we connected an additional module in the next available slot. )
There is for some reason information from the empty slot and it is read.
The information is of course corrupted and/or otherwise useless.
This results in a 'slot read not as expected' error.
It seems that there should be a way to read a few modules skip one or more
and then read again. Reading FIFO data goes sequentially
module by module. However, it seems that it is expected that slots are also in order
, but I haven't been able to tell.

Two options seem to be available.

1. It could be fixed that slot_def.set accepts a map where slots are not in sequential order.
   So far I couldn't run with this option. It would boot but acq would hang and not begin (pacman mode).
2. The data from this empty module could be checked and dumped and reading progresses to the next module, throwing an error in case there is a real issue.

The first seems preferable to me, but I was only able to implement the second by reordering some logic in ReadFIFO.
The second runs the risk of missing data that was corrupted for some reason, but otherwise should have been counted and then correlations (if necessary) would have an efficiency hit coming from DAQ issues. This seems risky. But the reordering is included below FYI.

"I have a suspicion that slotRead is reporting module number and not slot number"

This is ruled out because, slotRead would return numbers beyond the capacity of the crate and also where there were not modules numbered. Additionally, It would return several different values for the error if it was allowed to continue. This and looking at data spills indicated corrupted of data.

Suggested Fix #2:

Replace in bool Poll::ReadFIFO():---------------------------------------------------------------

if (parseWords > dataWords + nWords[mod]) {
                word_t missingWords = parseWords - dataWords - nWords[mod];
                word_t partialSize = eventSize - missingWords;
                if (debug_mode) std::cout << "Partial event " << partialSize << "/" << eventSize << " words!\n";

                //We could get the words now from the FIFO, but me may have to wait. Instead we store the partial event for the next FIFO read.
                for(unsigned short i=0;i< partialSize;i++) 
                    partialEvent[mod].push_back(fifoData[parseWords - eventSize + i]);

                //Update the number of words to indicate removal or partial event.
                nWords[mod] -= partialSize;

            }
            //If parseWords is small then the parse failed for some reason
            else if (parseWords < dataWords + nWords[mod]) {
                std::cout << Display::ErrorStr() << " Parsing indicated corrupted data at " << parseWords - dataWords << " words into FIFO.\n";

                if (!is_quiet) {
                    //Print the first 100 words
                    std::cout << std::hex;
                    for(int i=0;i< 100;i++) {
                        if (i%10 == 0) std::cout << std::endl << "\t";
                        std::cout << fifoData[dataWords + i] << " ";
                    }
                    std::cout << std::dec << std::endl;
                }

                do_stop_acq = true;
                had_error = true;
                return false;
            }

            //Assign the first injected word of spill to final spill length
            fifoData[dataWords - 2] = nWords[mod] + 2;
            //The data should be good so we iterate the position in the storage array.
            dataWords += nWords[mod];

---

With a reordered version:

            //If parseWords is small then the parse failed for some reason
            if (parseWords < dataWords + nWords[mod]) {
                std::cout << Display::ErrorStr() << " Parsing indicated corrupted data at " << parseWords - dataWords << " words into FIFO.\n";

                if (!is_quiet) {
                    //Print the first 100 words
                    std::cout << std::hex;
                    for(int i=0;i< 100;i++) {
                        if (i%10 == 0) std::cout << std::endl << "\t";
                        std::cout << fifoData[dataWords + i] << " ";
                    }
                    std::cout << std::dec << std::endl;
                }

                //do_stop_acq = true;
                had_error = true;
                continue;
            } else {

                if (parseWords > dataWords + nWords[mod]) {
                    word_t missingWords = parseWords - dataWords - nWords[mod];
                    word_t partialSize = eventSize - missingWords;
                    if (debug_mode) std::cout << "Partial event " << partialSize << "/" << eventSize << " words!\n";

                    //We could get the words now from the FIFO, but me may have to wait. Instead we store the partial event for the next FIFO read.
                    for(unsigned short i=0;i< partialSize;i++) 
                        partialEvent[mod].push_back(fifoData[parseWords - eventSize + i]);

                    //Update the number of words to indicate removal or partial event.
                    nWords[mod] -= partialSize;
                }

                //Assign the first injected word of spill to final spill length
                fifoData[dataWords - 2] = nWords[mod] + 2;
                //The data should be good so we iterate the position in the storage array.
                dataWords += nWords[mod];
            }      
        } //End loop over modules for reading FIFO

---