New Mesh: SOwISC12to60E2r4

[xylar]

Long name: SOwISC12to60kmL60E3SMv2r4

This version of the Southern Ocean Regionally Refined Mesh (SORRM) includes:

• 12 km resolution around Antarctica, about 1000 km farther north than in revision 3 of the mesh
• 45 km resolution at southern mid-latitudes
• 30 km resolution at the equator and the north Atlantic
• 60 km resolution in the north Pacific
• 35 km resolution in the Arctic

It matches the EC30to60 mesh except in the Southern Ocean and north Atlantic.

The merge includes some reorganization of the SOwISC test cases within COMPASS:

• the "resolution" has been renamed from SO10to60wISC to SOwISC12to60
• a new test case files_for_e3sm has been added to create the initial condition, graph files, analysis region masks and mapping files, etc.

The script that produces files for E3SM has been modified to use the short name of the mesh in output file names, rather than the long name.

Initial condition, spin-up and files for E3SM are on Anvil at:

lcrc/group/e3sm/ac.xylar/mpas/spinup_expand_hr_SOwISC12to60E2r4/ocean/global_ocean/SOwISC12to60/

[xylar]

Base mesh resolution

[xylar]

Culled Mesh

	nCells = 428742 ;
	nEdges = 1299360 ;
	nVertices = 870229 ;

[xylar]

Initial State

Path on Anvil:

/lcrc/group/e3sm/ac.xylar/mpas/spinup_SOwISC12to60E2r4/ocean/global_ocean/SOwISC12to60/init/initial_state

Histograms:

[xylar]

Spin-up

I'm having trouble with spin-up. I'll try again in a bit but currently getting blow-up on day 15.

Update: The blow-up is happening at a specific ice-shelf, maybe one of these troublesome one-cell ice shelves that have been such trouble in the past:

[milenaveneziani]

@xylar, one thing we discussed in the past when making the SORRM meshes was to avoid placing the transition region 20-30km resolution within the ACC and the Brazil-Malvinas retroflection, so that to avoid transitioning from no-GM to GM in the eddy-rich and energetic regions. I think we did have one such mesh in the past, but this one is concerning to me in these regards (the dark blue-to-yellow region in the SH plot above goes right through the ACC/BM retroflection).

[milenaveneziani]

I would like to suggest to push the 20km resolution line to where the 30km line is now (in the Southern Ocean) and transition to 30km equatorward of that.
This probably means that, in the South Atlantic, it may not be worthwhile to transition to any lower resolution than 30km, if cell # is not impacted much.

[xylar]

@milenaveneziani, thanks for your thoughts! I'm definitely willing to run that version and see what the cell count looks like.

My feeling is that we had this conversation already and made a decision to go for lower resolution to keep the mesh cheaper even at the expense of SO dynamics with the understanding that the SO is not the main scientific focus of the study. But we certainly can see what the additional cost of moving the transition further north would be and decide if it's okay.

@vanroekel had recommended moving the GM transition so it is completely off at 30 km and fully on at 40 km (as AWI apparently does). Would that relate in any way to your concern?

[milenaveneziani]

I thought about the possibility of changing the limits of the GM ramp, but for this mesh, that would mean having a lot of the global domain over the no-GM-to-GM transition region (check dark-to-green area in global plot above).

with the understanding that the SO is not the main scientific focus of the study

Could you please clarify this? The shelf dynamics is linked to the off-shelf circulation: for example, any impact on the Weddell and Ross gyres must in turn impact the associated shelf and coastal regions. Same should be true for West Antarctica, where the ACC is so much closer to the coast than in other areas.

[xylar]

The shelf dynamics is linked to the off-shelf circulation: for example, any impact on the Weddell and Ross gyres must in turn impact the associated shelf and coastal regions. Same should be true for West Antarctica, where the ACC is so much closer to the coast than in other areas.

@milenaveneziani, that's fair. At the same time, that argument is somewhat endless and we could keep adding more and more high-res regions to places that matter. I wish we had the human and computer time to really study where the transition matters.

We had quite a bit of discussion here about the previous iteration of the mesh.
MPAS-Dev/MPAS-Model#518
To my surprise, there wasn't much about the motivation for pushing the southern ocean to be coarser and the transition to be further south. That conversation must have happened elsewhere but I distinctly recall a conversation very similar to the current one.

[milenaveneziani]

We definitely had this discussion in person (you on the phone :), when Kristin was working on the SORRM meshes.

As for the PR that you pointed to, I see that earlier versions of that mesh did have the transition 20-to-30km further north, exactly where I was thinking to have it for this mesh (see this for example: https://user-images.githubusercontent.com/4179064/79228805-59666900-7e62-11ea-96bf-4b29fe90943d.png). But then the very last version (Another variant: 12 km finest resolution, 1600 km transition) is more similar to this current mesh.

[milenaveneziani]

I agree with you that we need to find a compromise between what is ideal and what is feasible. But we also want to minimize the number of iterations, and I think keeping GM off for most of the SO is a good idea, given that it impacts our stratification and subsurface heat content so much south of 55S.

I think others should chime in as well in this discussion.

[xylar]

As for the PR that you pointed to, I see that earlier versions of that mesh did have the transition 20-to-30km further north, exactly where I was thinking to have it for this mesh (see this for example: https://user-images.githubusercontent.com/4179064/79228805-59666900-7e62-11ea-96bf-4b29fe90943d.png). But then the very last version (Another variant: 12 km finest resolution, 1600 km transition) is more similar to this current mesh.

Yes, revision 1 was with high resolution further north and 10 km highest resolution. Revisions 2 and 3 were the same except for the number of vertical levels. They had exactly the same SO as I have here. The only difference in this version, aside from critical passages added to geometric features since April, is that the northern hemisphere has less high resolution.

[xylar]

Resolution further North

About 33% more cells than before, so ~1.3x the cost:

	nCells = 569915 ;
        nEdges = 1723935 ;
	nVertices = 1153614 ;

Since the number of cells above is almost identical to revision 3 of the mesh, this version would go from about 5 simulated years per day to a little under 4.

path on Anvil:

/lcrc/group/e3sm/ac.xylar/mpas/spinup_expand_hr_SOwISC12to60E2r4/ocean/global_ocean/SOwISC12to60/

[xylar]

@milenaveneziani, this looks pretty good to me and worth a try. @darincomeau implied there was some leeway to add more resolution and still get decent throughput. What do you think about moving the high res further north?

[xylar]

Spin-up also seems to be behaving much better and the histogram of layer thickness certainly looks less weird. So presumably the higher resolution is making for better ice-shelf cavities.

[milenaveneziani]

yes, I agree, this looks like a good candidate. Thanks Xylar.

Could you clarify what you mean by 'What do you think about moving the high res further north?'? Where, in the SO or in other regions?

[xylar]

Could you clarify what you mean by 'What do you think about moving the high res further north?'? Where, in the SO or in other regions?

That was a question for @darincomeau, sorry for not being clearer.

[darincomeau]

In terms of cell count, I think this should still be practical.

[proteanplanet]

I'm running QC this afternoon to complete the review by this evening.

[xylar]

@proteanplanet, thank so much! I assume your analysis is for the second version of the mesh (in the spinup_expand_hr_SOwISC12to60E2r4 folder)?

@milenaveneziani, thank you as well for your thoughts so far. Keep them coming.

@mark-petersen, @alicebarthel and @wlin7, I have put you down on the corresponding confluence page as reviewers of this stage of the mesh. If you are willing, please take a look at these plots, make your own if you like or request that I make what you need, and let me know by a week from today, January 14th (sooner if possible) if the mesh looks good to you.

If more iteration is required, we will generate new versions of the mesh and extend the deadline if necessary. Assuming all approve, I'll merge this and we can move on to testing in E3SM (@darincomeau and @jonbob), followed by another round of analysis (@alicebarthel, @milenaveneziani, @maltrud, @proteanplanet, @darincomeau and @wlin7).

If anyone would prefer not to be involved that I have nominated, just comment on the confluence page to nominate someone else.

If someone is not listed but would like to be, we can also add you formally to the list of reviewers. Anyone is welcome to comment at any stage.

[proteanplanet]

@xylar I'm running checks on /lcrc/group/e3sm/ac.xylar/mpas/spinup_expand_hr_SOwISC12to60E2r4/ocean/global_ocean/SOwISC12to60/init/initial_state/initial_state.nc

Let me know if I've got that wrong.

[xylar]

@proteanplanet, perfect, just wanted to make sure!

[milenaveneziani]

sounds great. Thanks.

[xylar]

Spin-up

Spin-up was successful and results are here:
/lcrc/group/e3sm/ac.xylar/mpas/spinup_expand_hr_SOwISC12to60E2r4/ocean/global_ocean/SOwISC12to60/spin_up

Files for E3SM

Note: These files have been generated to ensure that everything is working as expected. They will be re-generated from the master branch once the PR is approved and merged, and before any E3SM simulations take place.

The E3SM ocean and sea-ice initial conditions, graph partition files, scrip files, and MOC masks are here:
/lcrc/group/e3sm/ac.xylar/mpas/spinup_expand_hr_SOwISC12to60E2r4/ocean/global_ocean/SOwISC12to60/files_for_e3sm/files_for_e3sm/assembled_files_for_upload/inputdata/

Mapping files and transects needed for analysis are here:
/lcrc/group/e3sm/ac.xylar/mpas/spinup_expand_hr_SOwISC12to60E2r4/ocean/global_ocean/SOwISC12to60/files_for_e3sm/files_for_e3sm/assembled_files_for_upload/diagnostics

I had the impression, looking at how existing E3SM v2 meshes are being handled (https://web.lcrc.anl.gov/public/e3sm/inputdata/ocn/mpas-o/), that the desire on the E3SM side is to use the short name and to have a single-digit revision number. This was not what we decided in MPAS-Dev/MPAS-Model#507, instead agreeing that the long name would be used in almost every context except where the names of multiple components appeared.

Nevertheless, it seems that the E3SM v2 standard has been to use the short name so far, perhaps due to constraints within CIME. So the E3SM output files I created follow this convention and use a single-digit revision number, e.g.:

ocean.SOwISC12to60E2r4.scrip.210107.nc
SOwISC12to60E2r4_moc_masks_and_transects.nc
SOwISC12to60E2r4_no_xtime.nc

@vanroekel and @jonbob, could you comment on the use of the short name vs. the long name?

[proteanplanet]

This mesh looks terrific from my angle. Here are panned views and enlargements that are most relevant for the application in this mesh to complement information already included in this PR:

Panned views:

Selected enlargements corresponding to annotated areas in the above maps:

[proteanplanet]

Based on my review of the mesh for basic coastal and bathymetric acuity around the globe, this looks good.

[milenaveneziani]

I am satisfied too. Thanks @xylar.

[alicebarthel]

Thanks @xylar for bringing me on board, and @proteanplanet for those amazing plots!
I am happy with the resolution on the narrow continental shelves (e.g. in dronning maud land) and even the George VI ice shelf looks good. Obviously, I have limited experience with how E3SM might behave, but I saw nothing in topography or resolution that raises red flags.

(I am also pretty excited to have a refined SO in general!)

[mark-petersen]

I ran on LANL IC with Grizzly. Successfully completed init, ssh_adjustment, and spin-up. Looked through the vtk file, a quick check looked good. See
/lustre/scratch3/turquoise/mpeterse/runs/210112_pr37_xylar_add_SOwISC12to60E2r4/ocean/global_ocean/SOwISC12to60

[mark-petersen]

Thanks @xylar!

[xylar]

Thanks so much @mark-petersen!

[wlin7]

@xylar, thanks for the hard work refining this mesh. The discussions and complementary information above are enlightening to me. Our experts have offered their approval. I have no comment on the mesh itself and see no concern on coupling with the atmosphere. Looking forward to E3SM simulations with this RRM; I can then contribute to the climate analysis to complete the mesh approval process.

[xylar]

Thanks @wlin7 and thanks again to everyone!

[mark-petersen]

Sticking here for future use, the image with min and max on colorbar, and compared to EC on same colorbar: