New COMPASS EC60to30, NAEC60to30cr8 and QU60 test cases don't spin up

[xylar]

As part of #229, I tried to run all global_ocean (but not global_ocean_V1) test cases. Most test cases ran to completion or could be fixed with small tweaks to the configuration that I will include in #229. The cases that are broken beyond seemingly easy repair are:
The following is a list of tests that failed (but for reasons I believe are unrelated to this PR):

• -o ocean -c global_ocean -r EC60to30 -t spin_up during simulation because of CFL violation:

CFLNumberGlobal = 0.0917271835580922, 0.122637831833784, 0.150533786272102, 
    0.17555059144412, 0.197662099329266, 0.217073518294376, 
    0.234323872726046, 0.250136386921142, 0.264790486294835, 
    0.27825253635325, 0.290323188086416, 0.301051958089178, 
    0.310209832190098, 0.3176288821503, 0.32330034866235, 0.328730874921286, 
    0.340932315339754, 0.39929006348568, 0.818147632488227, 1.93089241771095, 
    3.78167130632891, 9.49790843196592, 0.358590340863955

• -o ocean -c global_ocean -r NAEC60to30cr8 -t spin_up: the final (test_final_settings) phase seems to be misconfigured in various ways. Links to restart and Restart_timestamp are missing. timeSeriesStatsMonthly is enabled but there isn't a corresponding stream. It attempts to change to monthly restarts but the last restart is at 0001-01-07, which isn't an allowable combination in MPAS.
• -o ocean -c global_ocean -r QU60 -t spin_up during simulation for unknown reasons during the first time step

I don't believe any of these issues are related to #229 but haven't had time to run the tests with ocean/develop to be completely sure.

[xylar]

@mark-petersen, I think you are likely to be the best person to handle these but @sbrus89 was also involved in creating some of these cases so I've assigned you both.

[mark-petersen]

@kristin-hoch and I spun up the jigsaw EC60to30 a month ago, we will make sure the spin-up sequence matches what we did for a successful run. Thanks.

[kristin-hoch]

@xylar I ran the EC60to30 and NAEC60to30cr8 a couple of months ago on another branch. At that point, test_final_settings wasn't configured. I was copying files and manually changing the timing in the restart files. I also ended up using the NAEC60to30 spin up for my JIGSAW EC60to30 run. I don't remember what the issue was with the EC60to30 setup, but it was enough of a problem that I decided not to use it for my simulations. Unless something has changed since I ran those simulations, I imagine the issues are still there.

[xylar]

@kristin-hoch, do you have a way to look up what settings you used for your spin-up? It's a pretty big problem in my view if the test cases in the repo don't actually work. It makes it particularly hard to test when things get updated like #229. So it would be really great if we could fix these test cases.

[kristin-hoch]

@xylar
I used a NAEC60to30cr8 spin up setup. I don't believe I changed anything there other than linking to the EC60to30 initialization files.

I don't remember why I didn't originally use the EC60to30 spin up. @mark-petersen ran the spin up initially.

[xylar]

Thanks @kristin-hoch. While I could imagine that working, I think it's probably way overkill since the 8 km resolution for the NAEC60to30cr8 setup requires really small time steps and a lot more stages of spin-up.

I'm working on a 3-stage spin-up of the EC60to30 setup right now. I'll work on the QU60 once that's done. I'll see if the NAEC60to30cr8 spin-up is still broken after recent merges and see if I can fix it (maybe with help from @sbrus89) if so.

[xylar]

@kristin-hoch, I tried your idea (copying the same spin-up procedure as in NAEC60to30cr8 and using it for EC60to30). Perhaps it has changed but I get a blow-up during spin_up6

 CFLNumberGlobal = 0.077117660983108, 0.083101029671738, 0.0887325960489197, 
    0.0939077084412331, 0.0982270351340179, 0.10111611438146, 
    0.102084064917903, 0.100714450924211, 0.102851363767319, 
    0.108888092656042, 0.115104317546689, 0.124287100245929, 
    0.133364167739886, 0.141517968876622, 0.149063282681481, 
    0.154423651493676, 0.157945939738342, 0.161422797060784, 
    0.161030062876986, 0.165995200433268, 0.286422116757691, 
    0.327758978525572, 0.467824754996828, 0.678616050803111, 
    0.656826257523782, 0.546276148557583, 0.415468662405752, 0.920192873123066

I also tried running my own tests where I used dt = 10 min and dt_br = 20 seconds. I tried ramping down from Rayleigh damping of 1e-4 to zero without increasing the time step but also got a blow-up when I went to 5e-6.

I am very much open to suggestions from anyone who has experience spinning up Jigsaw runs because I'm stuck.

[xylar]

I'm giving up on the EC60to30 for now. I tried dt = 10 min and dt_br = 20:

1. config_Rayleigh_damping_coeff = 1.0e-4 for 20 days
2. config_Rayleigh_damping_coeff = 5.0e-5 for 20 days
3. config_Rayleigh_damping_coeff = 2.0e-5 for 10 days
4. config_Rayleigh_damping_coeff = 5.0e-6 for 10 days
5. ...
   During stage 4, the CFL keeps spiking to dangerous levels, though the simulation never quite blows up. I could try config_Rayleigh_damping_coeff = 1.0e-5 but I don't think that would then help when I halve the coefficient later on. I just think there's something wrong...

Here's a graph of the CFL:

[kristin-hoch]

Xylar, That's weird. I'll look over the setting that I used to see if anything has changed in the newest version.

…

-Kristin

________________________________ From: Xylar Asay-Davis <notifications@github.com> Sent: Monday, May 13, 2019 12:53:57 PM To: MPAS-Dev/MPAS-Model Cc: Hoch, Kristin; Mention Subject: Re: [MPAS-Dev/MPAS-Model] New COMPASS EC60to30, NAEC60to30cr8 and QU60 test cases don't spin up (#235) I'm giving up on the EC60to30 for now. I tried dt = 10 min and dt_br = 20: 1. config_Rayleigh_damping_coeff = 1.0e-4 for 20 days 2. config_Rayleigh_damping_coeff = 5.0e-5 for 20 days 3. config_Rayleigh_damping_coeff = 2.0e-5 for 10 days 4. config_Rayleigh_damping_coeff = 5.0e-6 for 10 days 5. ... During stage 4, the CFL keeps spiking to dangerous levels, though the simulation never quite blows up. I could try config_Rayleigh_damping_coeff = 1.0e-5 but I don't think that would then help when I halve the coefficient later on. I just think there's something wrong... Here's a graph of the CFL: [cfl]<https://user-images.githubusercontent.com/4179064/57646588-359b0300-75c1-11e9-986a-02ab2bda4e5d.png> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#235?email_source=notifications&email_token=AKE6JCPLHJU2WX725SGABT3PVG2MLA5CNFSM4HLJG242YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVJHJNI#issuecomment-491943093>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AKE6JCM3JEKG37ORNUNFL4TPVG2MLANCNFSM4HLJG24Q>.