mosart1.1.09: Put instantaneous and non-inst. fields on separate history files

[slevis-lmwg]

This is the MOSART equivalent of the work that I did in ESCOMP/CTSM#2445 for the CLM. For MOSART this work started in #69 originally.

Resolves issue #52.

• Remember to add Added code to make st-archive handle DART files #118 to the same mosart tag

[slevis-lmwg]

Testing guidance in this comment:
ESCOMP/CTSM#2445 (comment)

Latest attempt on izumi ./run_sys_tests -s mosart --skip-compare --skip-generate

FAIL ERP_D.f10_f10_mg37.I1850Clm60Sp.izumi_nag.mosart-qgrwlOpts RUN

Made it past earlier errors of no file name in case2run and later no rh files copied to case2run. For the latter, eventually I just needed to submit the test anew. The new error to fix is here:
/scratch/cluster/slevis/ERP_D.f10_f10_mg37.I1850Clm60Sp.izumi_nag.mosart-qgrwlOpts.20250624_175353_ni1nta/run/case2run/cesm.log...

Runtime Error: /fs/cgd/data0/slevis/git/sep_inst_flds_iss1059/cime/../components/mosart/src/riverroute/mosart_histfile.F90, line 1552: Subscript 2 of NCID_HIST (value 3) is out of range (1:2)
/fs/cgd/data0/slevis/git/sep_inst_flds_iss1059/cime/../components/mosart/src/riverroute/mosart_histfile.F90, line 1552: Error occurred in MOSART_HISTFILE:MOSART_HIST_RESTART
/fs/cgd/data0/slevis/git/sep_inst_flds_iss1059/cime/../components/mosart/src/riverroute/mosart_restfile.F90, l

See new attempt with the same test.

[slevis-lmwg]

./run_sys_tests -s mosart --skip-compare --skip-generate

• on izumi and derecho noting
  FAIL SMS_D_Ld5.f10_f10_mg37.I1850Clm60Sp.derecho_nvhpc.mosart-default SETUP
  that I've cleared before with Jim Edwards.
• work on Erik's code review in #2445 as it applies to #2445 and here
• run tests as recommended here: ctsm5.3.062: Put instantaneous and non-inst. fields on separate history files CTSM#2445 (comment)
• confirmed mosart history output has not changed bitwise by trying this command
  diff /glade/derecho/scratch/slevis/tests_0627-180421de/SMS_Ld5.f10_f10_mg37.I1850Clm60Sp.derecho_intel.mosart-mosartCold.GC.0627-180421de_int/run/SMS_Ld5.f10_f10_mg37.I1850Clm60Sp.derecho_intel.mosart-mosartCold.GC.0627-180421de_int.mosart.h0a.0001-01-06-00000.asc /glade/work/slevis/git/mksurfdata_toolchain/tests_0627-170402de/SMS_Ld5.f10_f10_mg37.I1850Clm60Sp.derecho_intel.mosart-mosartCold.GC.0627-170402de_int/run/SMS_Ld5.f10_f10_mg37.I1850Clm60Sp.derecho_intel.mosart-mosartCold.GC.0627-170402de_int.mosart.h0.0001-01-06-00000.asc
  and
  diff /fs/cgd/data0/slevis/git/sep_inst_flds_iss1059/tests_0627-180718iz/SMS_D_Ld5.f10_f10_mg37.I1850Clm60Sp.izumi_nag.mosart-default.GC.0627-180718iz_nag/run/SMS_D_Ld5.f10_f10_mg37.I1850Clm60Sp.izumi_nag.mosart-default.GC.0627-180718iz_nag.mosart.h0a.0001-01-06-00000.asc /fs/cgd/data0/slevis/git/b4b-dev/tests_0627-171420iz/SMS_D_Ld5.f10_f10_mg37.I1850Clm60Sp.izumi_nag.mosart-default.GC.0627-171420iz_nag/run/SMS_D_Ld5.f10_f10_mg37.I1850Clm60Sp.izumi_nag.mosart-default.GC.0627-171420iz_nag.mosart.h0.0001-01-06-00000.asc

[slevis-lmwg]

@ekluzek this is ready for review/approval. The changes to mosart will look very familiar now that you have reviewed ESCOMP/CTSM#2445.

[slevis-lmwg]

For Sprint accounting purposes I consider this PR done, and I am moving the tracking to ESCOMP/RTM#32

[ekluzek]

This all looks similar to everything in the CTSM and RTM PR's. I ask at least one question. And some of what I had in CTSM and/or RTM might apply here as well.

But, approving so you don't have to wait on me. Thanks for all the work to bring this in. There's just a lot of changes needed that have to be done right to get it all working.

[slevis-lmwg]

• finalize and push ChangeLog