Organization of server data

[PaulWessel]

Currently, we only have earth_relief_xxy files in the gmt/data directory (everything else is under gmt/data/cache). However, we are about to add both blue and black marbles, the global crustal ages, and it is likely there will be more data sets in the future that should not be considered for cache (since they will have multiple resolutions etc). To peak ahead, it is likely we will split large global items into tiles, similar to SRTM. Whether we do that or not right now, it seems we should think about organization. How about this:

gmt/data/cache: Odds and ends used for tests and examples tutorial etc.
gmt/data/server: Data served by us.  In here there will be subdirectories:
    earth_relief
    earth_ages
    earth_marble [maybe one each for black and blue, or some clever scheme]
    ...

Inside these directories are the actual files: earth_relief_xym plus srtm1, strtm3 will be in the earth_relief folder, etc.

Perhaps the gmtserver needs to produce or maintain a listing of what is in server so that gmt can discover that we have added more data. We would at least need to know if a dataset is tiled or not to know what to do. I think the decisions that happen in gmt_remote.c depending on earth_relief resolution (get file or get tiles) need to be abstracted away and be based on a setup file we refresh, just like we refresh the hashes.

[PaulWessel]

One issue up front: If we make any structural changes to the gmtserver directories, we break access for everybody else. I think curl will follow symbolic links? If so then we could add links such as earth_relief_01m.grd that points to, say, earth_relief/earth_relief_01mg.grd (Notice the g for "Gridline-registered").
I guess we can do an experiment on that.

[PaulWessel]

Experiment worked. A symbolic link in the right place can point to another file and be followed. So that is how we could introduce earth_relief/earth_relief_01mp|g.grd files with links to the gridline version from the directory above.

[PaulWessel]

To get ready for 6.1.0 release, we should do this:

1. Upload the new sets of earth_relief_xxy_g|p.grd files supporting both pixel and gridline resolutions and place them in a subdirectory called earth_relief.
2. Replace the current earth_relief_xxy.grd files with symbolic links to the corresponding files in the earth_relief directory (grid-line versions, except for 15s which is the original pixel).
3. Add the marbles images (they now report -Rg instead of thousands so they are returned as geo). These should go into subdirectories too. What do you think is the best approach?:
   3a. Directory earth_image, rename all files to earth_day_xxy and earth_night_xxy
   3b. Directories Blue_Marble and Black_Marble, with the Black|Blue_Marble_xxy names
   3c. Directories earth_day and earth_night, with corresponding xxy names
   3d. ????
4. Add new directory earth_age and add the earth_age_xxy_g|p.grd files as soon as EarthByte wraps up their 1x1 min grid. We have no examples of using these so it is OK if these files take longer to be added and then we update docs in 6.1.x.

Please give feedback on this now, @joa-quim and @seisman. I am trying to avoid changes down the road, which will see (hopefully)

1. Earth gravity data [Sandwell, others]
2. Other planets (Mars, Moon, ...)
3. Mechanism for plotting modules to not specify _xxy and have the module auto-compute needed resolution for the given map dimensions and GMT_IMAGE_RESOLUTION = 300 (or whatever).

The @ algorithm will need to learn what is available via gmt_hash_server.txt. BTW, my hash_server file is 0 byes on May 20, so something failed - what does yours say.

[PaulWessel]

I should add one more thing: The SRTM15+v2.1 is a floating point grid with more precision than the integers. This time I wrong out this format =ns+sa which auto-scales the data to fit the full -32767/+32767 range, and hence the precision in the values is about 0.25-0.3 meters instead of 1 meter. This makes the files a bit larger since more different bits. As an example, the full 15s file grows from 2.6 Gb to 3.1 Gb, and the 1 minute goes from 215 Mb to 257 Mb, both about 20% increase. Do you wnat to dumb down to nearest meter and retain the smaller file sizes? I.,e trading 3-4 times the precision for 1.2 times longer download time. I would certainly prefer the higher quality.

[PaulWessel]

How do we handle this scenario:

1. User runs with old name @earth_relief_30s. We download earth_relief_30s_g.grd because of the link.
2. Users runs it again. We dont want to download again. So gmt_remove needs to be aware of the deprecated names and look for both that name and the new name.

[joa-quim]

2. Symbolic links. So user will say @earth_relief_xxy.grd and the link will give him @earth_relief_xxyp.grd, right? But what will happen next time he issues the same command? Will it not download the same file again because earth_relief_xxy.grd doesn't exist in his system?

3. 3b because those are known names whilst earth_day|night_ are unknowns

4. For the same reason as 3 the files should have their known names *age.xxx.nc``

Ok, I see that you are addressing my first point too

[PaulWessel]

Are you saying drop earth_ from the ages files? FYI, there are actually two files

age.2020.1.GTS2012.1m.nc
age.2020.1.GK2007.1m.nc

for two different time-scales. The people who care about these grids are the same peolple who care about the different time-scales... So we may have to do

age_GTS2012_xxy_p|g and age_GK2007_xxy_p|g but we could accept age_xxy_g|p ti sekect GK2007 (I think they prefer that and that is what they only used in the past).

[joa-quim]

I would certainly prefer the higher quality.

Me too

[joa-quim]

Yes, drop the earth_ from name. And if they are both 1m only we don't need the _xxy

[PaulWessel]

You are forgetting 30m, 15m, all the way down to 2m. So xxy is there to stay.

[PaulWessel]

I wonder if this is a better solution:

1. Leave the current data directory on the server as is. Ubuntu users will still try to access taht in 2023.
2. Create a new dir with another name than data, e.g., server (to match what we create in the user's directory)
3. Place all the new subdirs and data we discussed in the new server directory.

If we don't, then we will need to carry much complexity in the gmt_remote.c file to handle these cases:

1. Remember, any file not listed in the gmt_hash_table is assumed to have been removed and will be DELETED on the user side. Since earth_relief_xxy.grd won't be listed, it will be deleted, then re-downloaded, etc. We don't want that.
2. When users finally upgrade to 6.1 they move to the new system, and they will get the new data files.

This way the old system will work fine - they wont get the new files until they upgrade (a good argument to do so), and we don't have to deal with legacy file names and complicated checks for old and new file names.

Also remember: We will need to maintain two separate hash tables, one for pre 6.1 and one new ones. GMT 5-6.0 will download the old one with the old files, 6.1 will download the new one with the new files and directory structure.

[PaulWessel]

Yes, drop the earth_ from name.

I will ask EarthByte what they prefer - it is their files.

[joa-quim]

6.1 will download the new one with the new files and directory structure.

But can they still call them @earth_relief_xxy.grd and get the old names, right?