Data finder cannot find data spread over different versions

[schlunma]

On DKRZ, it can happen that data for a specific dataset is spread over different versions. For example, the first 150 years of CESM2's tas are located in a different subdirectory than the rest of the data:

.../CMIP/NCAR/CESM2/abrupt-4xCO2/r1i1p1f1/Amon/tas/gn
├── v20190425
│   └── tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_000101-015012.nc
└── v20190829
    ├── tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_015101-019912.nc
    ├── tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_020001-024912.nc
    ├── tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_025001-029912.nc
    ├── tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_030001-034912.nc
    ├── tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_035001-039912.nc
    ├── tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_040001-044912.nc
    ├── tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_045001-049912.nc
    ├── tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_050001-054912.nc
    ├── tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_055001-059912.nc
    ├── tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_060001-064912.nc
    ├── tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_065001-069912.nc
    ├── tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_070001-074912.nc
    ├── tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_075001-079912.nc
    ├── tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_080001-084912.nc
    ├── tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_085001-089912.nc
    ├── tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_090001-094912.nc
    └── tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_095001-099912.nc

In this case, the tool does not find the data for the years 1-150 anymore, because it just looks in the latest version directory.

[valeriupredoi]

Interesting case, I'll have a stab at it on my return from the city of very expensive beer on Friday 🍺 Dr Valeriu Predoi. Computational scientist NCAS-CMS University of Reading Department of Meteorology Reading RG6 6BB United Kingdom

…

On Wed, 2 Oct 2019, 11:52 Manuel Schlund, ***@***.***> wrote: Assigned #286 <#286> to @valeriupredoi <https://github.com/valeriupredoi>. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#286?email_source=notifications&email_token=AG5EFI7JDVCPN24K4U5EDK3QMRVNTA5CNFSM4I4UGHDKYY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOT66VR7A#event-2679986428>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AG5EFIYVJ6COW24OPITJWF3QMRVNTANCNFSM4I4UGHDA> .

[schlunma]

Started working on this now

[valeriupredoi]

I can take it from there if you point me to your branch Dr Valeriu Predoi. Computational scientist NCAS-CMS University of Reading Department of Meteorology Reading RG6 6BB United Kingdom

…

On Fri, 4 Oct 2019, 09:20 Manuel Schlund, ***@***.***> wrote: Started working on this now — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#286?email_source=notifications&email_token=AG5EFI66FUEUI3GOBGXBSGTQM34F7A5CNFSM4I4UGHDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAK4OQY#issuecomment-538298179>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AG5EFI7ZHNUCYINXCP33DXTQM34F7ANCNFSM4I4UGHDA> .

[schlunma]

I though a little bit about this issue and I think we can tackle this together with #54 (handling of duplicate files).

Would it make sense to completely remove the latest version selection in _data_finder.py and write a "smarter" concatenate function using the creation_date attribute (or something similar)?

What do you think @valeriupredoi @bouweandela?

[valeriupredoi]

#54 should be tackled by #280 so I reckon we can use that PR as a launch pad

[schlunma]

#54 should be tackled by #280 so I reckon we can use that PR as a launch pad

Perfect, didn't know about that one.

[valeriupredoi]

On CEDA Jasmin/BADC all's fine and dandy 😁

[valeriu@jasmin-sci1 ~]$ ls -la /badc/cmip6/data/CMIP6/CMIP/NCAR/CESM2/abrupt-4xCO2/r1i1p1f1/Amon/tas/gn/latest/
total 9
drwxr-x--- 1 badc open     0 Oct 14 23:59 .
drwxr-x--- 1 badc open     0 Oct 14 23:59 ..
lrwxrwxrwx 1 badc badcint 75 Oct 14 23:59 tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_000101-015012.nc -> ../files/d20190927/tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_000101-015012.nc
lrwxrwxrwx 1 badc badcint 75 Oct 14 23:59 tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_015101-019912.nc -> ../files/d20190927/tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_015101-019912.nc
lrwxrwxrwx 1 badc badcint 75 Oct 14 23:59 tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_020001-024912.nc -> ../files/d20190927/tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_020001-024912.nc
lrwxrwxrwx 1 badc badcint 75 Oct 14 23:59 tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_025001-029912.nc -> ../files/d20190927/tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_025001-029912.nc
lrwxrwxrwx 1 badc badcint 75 Oct 14 23:59 tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_030001-034912.nc -> ../files/d20190927/tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_030001-034912.nc
lrwxrwxrwx 1 badc badcint 75 Oct 14 23:59 tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_035001-039912.nc -> ../files/d20190927/tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_035001-039912.nc
lrwxrwxrwx 1 badc badcint 75 Oct 14 23:59 tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_040001-044912.nc -> ../files/d20190927/tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_040001-044912.nc
lrwxrwxrwx 1 badc badcint 75 Oct 14 23:59 tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_045001-049912.nc -> ../files/d20190927/tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_045001-049912.nc
lrwxrwxrwx 1 badc badcint 75 Oct 14 23:59 tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_050001-054912.nc -> ../files/d20190927/tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_050001-054912.nc
lrwxrwxrwx 1 badc badcint 75 Oct 14 23:59 tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_055001-059912.nc -> ../files/d20190927/tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_055001-059912.nc
lrwxrwxrwx 1 badc badcint 75 Oct 14 23:59 tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_060001-064912.nc -> ../files/d20190927/tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_060001-064912.nc
lrwxrwxrwx 1 badc badcint 75 Oct 14 23:59 tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_065001-069912.nc -> ../files/d20190927/tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_065001-069912.nc
lrwxrwxrwx 1 badc badcint 75 Oct 14 23:59 tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_070001-074912.nc -> ../files/d20190927/tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_070001-074912.nc
lrwxrwxrwx 1 badc badcint 75 Oct 14 23:59 tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_075001-079912.nc -> ../files/d20190927/tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_075001-079912.nc
lrwxrwxrwx 1 badc badcint 75 Oct 14 23:59 tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_080001-084912.nc -> ../files/d20190927/tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_080001-084912.nc
lrwxrwxrwx 1 badc badcint 75 Oct 14 23:59 tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_085001-089912.nc -> ../files/d20190927/tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_085001-089912.nc
lrwxrwxrwx 1 badc badcint 75 Oct 14 23:59 tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_090001-094912.nc -> ../files/d20190927/tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_090001-094912.nc
lrwxrwxrwx 1 badc badcint 75 Oct 14 23:59 tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_095001-099912.nc -> ../files/d20190927/tas_Amon_CESM2_abrupt-4xCO2_r1i1p1f1_gn_095001-099912.nc

[valeriupredoi]

This is a true bitch, I don't see it solved before the 2.0 release, man 😭

[bouweandela]

@schlunma Is the data still stored in the way you reported in this issue, or was it a mistake that has now been corrected?

[schlunma]

I recently found another example where the data was structured like this (can't remember which data is was, though), so I guess this is not a mistake.