Create "Stage 2" FCL files for ICARUS workflows

[wesketchum]

per discussion at 11 Aug release meeting, lots of new high-level reco/matching/analysis additions would be good candidates for a separate 'stage 2' fcl file.

Work on this can proceed with the 2021B release, and complete prior to 2021C release.

(@SFBayLaser and others should comment, make new issues, etc. as necessary)

[gputnam]

Hey! So I missed this meeting but I believe I have implemented something like this planned for MCP2021B. This is currently setup as the "caf" configurations on #223: https://github.com/SBNSoftware/icaruscode/tree/feature/MCP2021Reco/fcl/caf. I have put a few high-level-reco-modules in this "stage": space charge corrections, momentum calculations, PID, etc. Are these configurations a good candidate for what "stage2" would be?

[SFBayLaser]

Opinion: all things that are not directly "reconstruction" should move to a stage2 fhicl file - for example any summary type tuple that would be output from the job. Stage 1 could be written to tape, the summary jobs can run on existing stage 1 output as they continue to evolve. Also, all MC truth could be done there too. Feel free to comment.

[SFBayLaser]

I have created a feature branch (see https://github.com/SBNSoftware/icaruscode/tree/feature/usher_data_MC_fhicl_updates) with some updates based on challenges of the past couple of days. So there is protection against missing TPC data so things won't crash, there is a fix to allow the noise_factor to be set again by fhicl, etc. I've also done more work on trying to get the stage0 MC updated to better match the output from the "old" version of reconstruction, so it includes the "MC" optical hit reconstruction and the CRT stuff to first order.
Issue #1 (first seen by Gray): the "new" noise model might overdo the noise compared to data and is resulting in a very large number of hits being found, enough to overwhelm pandora making it run too slow to be used. A way to deal with this is to scale the level of noise, this is done with fhicl parameters in SignalShapingServices by setting the value of NoiseFactVec where a value of 1. will make no change to the noise model scaling, a factor less than one scales by that much, etc. This is done in multitpc_detsim_icarus.fcl where I've left it at 0.6 for now but this needs to be "tuned" to make sure we are matching the data. Note that this appears as a vector of vectors, the top level vector is allowing for different shaping times in the electronics (there is only one now), the low level vectors are for planes 0-2. I'd just change them all to what value appears to work best.
Issue #2: I'm not sure how to translate the "old" MC reconstruction of optical flashes with the "new" scheme with the simple flash finder. If I understand correctly, this has been switched, not added as an extra. In the next 30 minutes I'm not going to figure it out, I would suggest contacting @drinkingkazu to make sure this is done correctly.
Issue #3: I think the CRT hit finding for data will be held hostage to getting the CRT channel mapping into the sqlite database. It seems it is in postgres but we already know we can't run large productions from postgres so before adding the CRT hit finding I think we need to resolve the sqlite database situation. I did copy the basic CRT stuff for the MC into the stage0 stuff... but my test data set does not seem to have the base CRT information so I was not able to perform a real test.

So currently the stage0_multiTPC_icarus_MC.fcl includes a file stage0_icarus_mc_defs.fcl which contains (or will eventually) MC specific stuff for stage0 processing. After that it includes the standard data definitions and set up. After that there is a redefinition of the basic physcis producers to get the MC defintiions, then the list of modules to call for the job. There are also some necessary overrides for processing.

Note that all of this is work in progress. There are still some updates needed to make sure the PMT and CRT stuff is working correctly and we probably need to clean up some things. As well I've been able to run the current stage 1 MC file and get output (once I scaled down the noise in the detector simulation) but I've not had time yet to really start working on it.

Generally, I think we could get away with a vanilla stage 1 file and start creating a stage 2 file which can handle all the output tuple making (CAF, etc.), plus MC specific tasks.

Note that I have not reconciled this feature branch with other branches that have tried to make updates to fhicl files. If those updates are also to stage0/stage1 I suggest trying to figure out how to merge into this branch.

I will vanish here shortly... be in email contact through Sunday late afternoon but not able to work on this anymore. I have flight home from Boston early Monday morning and should be able to make a release with current stuff (plus any augmentations others might want to include).