verification exp.: simplify secondary tests usage of "prepare_run"

[jm-c]

What changes does this PR introduce?

Implement changes following issue #964

What is the current behaviour?

In secondary tests "${sfx}", both prepare_run from current "../input. ${sfx}" directory and from primary test input directory "../input" are run, each after links from respective input dir are made.

What is the new behaviour

In secondary tests, only run one "prepare_run" (the one linked into the current run-dir) after links from both "../input. ${sfx}" and "../input" are made.

Does this PR introduce a breaking change?

Other information:

Suggested addition to tag-index

o verification:

• for secondary tests, change testreport to only run a single "prepare_run"
  from run-dir, after linking file from both ../input.${sfx}/ and ../input/.
• adjust several verification exp. secondary tests accordingly and update
  instructions in README files + in the docs.

[jm-c]

This PR is not finished, still need to update the README and the docs, but the changes to verification experiments are done and could be reviewed.

I had to add a copy of few parameter files (i.e., "data.*") and even two links for pickup file, mostly in experiment offline_exf_seaice and global_ocean.cs32x15/input.seaice respectively, because of the ordering changes: in master branch, prepare_run from "input.${sfx}" is run just after linking files from "../input.${sfx}" and before linking files from "../input". But with this modified testreport, the same prepare_run is run after linking files from both input dir, "../input.${sfx}" and "../input".

[mjlosch]

I ran this for fwd and TAF-AD testreport and there are no problems.

I find it a little counterintuitive that you need the extra files and links. The fact that these files (e.g. the namelist files in offline_exf_seaice) are identical to other copies in the experiments, confused me at first. I see now why this is necessary if we first link input.$sfx, then input and then run ./prepare_run. My first solution was using ./prepare_run to unlink certain files again and replace by the correct links, but that will make things even more confusing, I guess. How about calling linkdata twice for secondary tests, like this:

	    run_clean $dir/$pfxdir.$ex
	    linkdata $dir/$pfxdir.$ex $inputdir.$ex
	    linkdata $dir/$pfxdir.$ex $inputdir
	    runmodel $dir/$pfxdir.$ex && run=Y \
	    && results=`testoutput_run $dir $pfxdir.$ex $refExOut`

It would mean that many operations are done twice (the second time without any effect), so that may be confusing again. In summary, I don't have a good solution, so we should go with what is here.

[jm-c]

I updated the instructions related to prepare_run in the docs and in the README files. So this PR is now ready for review.

And regarding the extra set of parameter files that got added (mainly for offline_exf_seaice exp. secondary tests), it might prevent some confusion that could have occurs before, not knowing which version would be used until running testreport.

Regarding the two pickup files (symbolic links added in global_ocean.cs32x15/input.seaice),
I will try to change nIter0 and rename the pickup files (so that different files have different names), but will do that in a separated PR ; so, for now, leave it like this.

[mjlosch]

I followed the instructions in a set of sampled README's and was successful,

LGTM.

[jm-c]

Update: I tried to shift nIter0 in few secondary test from exp. global_ocean.cs32x15 to see if I can remove the two additional pickup files (symbolic links added in global_ocean.cs32x15/input.seaice), and this works nicely, no need to update any reference output. Will open a new PR once this one is merged.

Also, changing testreport to run only one prepare_run does improve clarity in the few experiments from verification_other that have secondary tests, see MITgcm/verification_other#79 .

[jm-c]

I think this PR is ready to be merged, and could do this tomorrow, unless someone wants to add something.