Initial import of a filter module to check data integrity #590
Conversation
…ing the stage 0 processing chain. Note that this initial import is aiming to detect "incomplete events" and is not a thorough integrity check just yet...
wesketchum
left a comment
wesketchum
left a comment
There was a problem hiding this comment.
I think this looks fine, but we also know (generally) how many fragments should be expected for good physics runs, no? Could we add a fcl parameter that is the expected fragment count, and then compare that as well? Would then work well for the first event in a file too.
PetrilloAtWork
left a comment
PetrilloAtWork
left a comment
There was a problem hiding this comment.
I need a clarification: the filter is configured to interrupt a path, but it does not seem to be bound to the output module(s). If I am not wrong, it means that the events that are filtered out will be still saved (with no data products except maybe the trigger data fragment if we are now keeping it as it was proposed). In that case, the following stages may find the event, try to process it, look for a raw waveform data product that is not there and throw ProductNotFound (or worse, silently ignore the problem — which happens to be what we need but because of a bug). I think the solution is to add selectEvents: [ pathname ] to the output module(s).
For the rest, I have dropped some minor suggestions.
|
|
||
| if (missingFragments.size() > 0 || emptyFragments > 0) | ||
| { | ||
| mf::LogInfo("FilterDataIntegrity") << "Bad fragments: " << missingFragments.size() << ", " << emptyFragments; |
There was a problem hiding this comment.
More info maybe?
| mf::LogInfo("FilterDataIntegrity") << "Bad fragments: " << missingFragments.size() << ", " << emptyFragments; | |
| mf::LogInfo log("FilterDataIntegrity"); | |
| log << "Bad fragments (" << missingFragments.size() << "):"; | |
| for (int fragmentID: missingFragments) log << " " << fragmentID; | |
| if (emptyFragments) log << "; empty: " << emptyFragments; |
|
I remind also in this venue that DAQ "promised" the addition of a list of configured components and their fragments, with in conjunction with code like this one or like feature/gp_missingFragments would allow a consistent and robust check. |
…done on events passing this check.
PetrilloAtWork
left a comment
PetrilloAtWork
left a comment
There was a problem hiding this comment.
All looks good to me now.
|
trigger build |
|
✔️ CI build for LArSoft Succeeded on slf7 for e20:prof -- details available through the CI dashboard |
|
✔️ CI build for LArSoft Succeeded on slf7 for c7:prof -- details available through the CI dashboard |
|
❌ CI build for ICARUS Failed at phase build ICARUS on slf7 for c7:prof -- details available through the CI dashboard 🚨 For more details about the failed phase, check the build ICARUS phase logs parent CI build details are available through the CI dashboard |
|
❌ CI build for ICARUS Failed at phase build ICARUS on slf7 for e20:prof -- details available through the CI dashboard 🚨 For more details about the failed phase, check the build ICARUS phase logs parent CI build details are available through the CI dashboard |
|
trigger build |
|
✔️ CI build for LArSoft Succeeded on slf7 for e20:prof -- details available through the CI dashboard |
|
✔️ CI build for LArSoft Succeeded on slf7 for c7:prof -- details available through the CI dashboard |
|
❌ CI build for ICARUS Failed at phase unit_test ICARUS on slf7 for e20:prof - ignored warnings for build -- details available through the CI dashboard 🚨 For more details about the failed phase, check the unit_test ICARUS phase logs parent CI build details are available through the CI dashboard |
|
❌ CI build for ICARUS Failed at phase unit_test ICARUS on slf7 for c7:prof - ignored warnings for build -- details available through the CI dashboard 🚨 For more details about the failed phase, check the unit_test ICARUS phase logs parent CI build details are available through the CI dashboard |
…mehow made it into a release... sigh...
|
trigger build |
|
✔️ CI build for LArSoft Succeeded on slf7 for e20:prof -- details available through the CI dashboard |
|
✔️ CI build for LArSoft Succeeded on slf7 for c7:prof -- details available through the CI dashboard |
|
trigger build |
|
✔️ CI build for LArSoft Succeeded on slf7 for e20:prof -- details available through the CI dashboard |
|
✔️ CI build for LArSoft Succeeded on slf7 for c7:prof -- details available through the CI dashboard |
|
❌ CI build for ICARUS Failed at phase unit_test ICARUS on slf7 for e20:prof - ignored warnings for build -- details available through the CI dashboard 🚨 For more details about the failed phase, check the unit_test ICARUS phase logs parent CI build details are available through the CI dashboard |
|
❌ CI build for ICARUS Failed at phase unit_test ICARUS on slf7 for c7:prof - ignored warnings for build -- details available through the CI dashboard 🚨 For more details about the failed phase, check the unit_test ICARUS phase logs parent CI build details are available through the CI dashboard |
|
❌ CI build for ICARUS Failed at phase unit_test ICARUS on slf7 for e20:prof - ignored warnings for build -- details available through the CI dashboard 🚨 For more details about the failed phase, check the unit_test ICARUS phase logs parent CI build details are available through the CI dashboard |
|
❌ CI build for ICARUS Failed at phase unit_test ICARUS on slf7 for c7:prof - ignored warnings for build -- details available through the CI dashboard 🚨 For more details about the failed phase, check the unit_test ICARUS phase logs parent CI build details are available through the CI dashboard |
|
trigger build |
|
✔️ CI build for LArSoft Succeeded on slf7 for e20:prof -- details available through the CI dashboard |
|
✔️ CI build for LArSoft Succeeded on slf7 for c7:prof -- details available through the CI dashboard |
|
❌ CI build for ICARUS Failed at phase unit_test ICARUS on slf7 for e20:prof - ignored warnings for build -- details available through the CI dashboard 🚨 For more details about the failed phase, check the unit_test ICARUS phase logs parent CI build details are available through the CI dashboard |
|
❌ CI build for ICARUS Failed at phase unit_test ICARUS on slf7 for c7:prof - ignored warnings for build -- details available through the CI dashboard 🚨 For more details about the failed phase, check the unit_test ICARUS phase logs parent CI build details are available through the CI dashboard |
|
trigger build |
|
✔️ CI build for LArSoft Succeeded on slf7 for c7:prof -- details available through the CI dashboard |
|
✔️ CI build for LArSoft Succeeded on slf7 for e20:prof -- details available through the CI dashboard |
|
🚨 For more details about the warning phase, check the ci_tests ICARUS phase logs parent CI build details are available through the CI dashboard |
|
🚨 For more details about the warning phase, check the ci_tests ICARUS phase logs parent CI build details are available through the CI dashboard |
…h now ported to the production branch
…roduction Copy of PR #590 and PR 591 for production branch
The idea is to check the integrity of the incoming data before running the processing chain with the hope to prevent job failures due to data issues. The main issue we have at the moment are incomplete events which can cause the decoder modules to throw exceptions and the processing to stop, so this initial import really aims to address that. The caveat is that it uses code similar to that used by the daq/dqm folks and it uses the first event, which is assumed to be fine, to initialize... so if the incomplete event is the first event then this module will miss it. Work in progress... but better than nothing.