organize: option to specify keys to be used instead of current "hard coded"

[yarikoptic]

We mandate subject, session, and then optional list of others,... @satra has mentioned a use case by @bendichter where people prefer to avoid using session altogether. I am yet to argue one way (against) or another (support), but here is at least an issue to document this desire/use case ;)

[satra]

this is the relevant dictionary to re-order / update: https://github.com/dandi/dandi-cli/blob/master/dandi/organize.py#L23

[satra]

also if a disambiguating key is the same at a given level, it could be automatically removed. so for example if session was the same for all files within a subject, it would not be used in the filename.

perhaps this already happens, but after ben fixes the id's, we should check.

[bendichter]

So for the case of the Tolias lab, here are the IDs as I received them:

...	Mouse	Slice	Cell
1310	mouse_ZNDVW	20191108_slice_4	20191108_sample_4
1311	mouse_ZNDVW	20191108_slice_5	20191108_sample_5
1312	mouse_YAVWS	20191114_slice_1	20191114_sample_1
1313	mouse_YAVWS	20191114_slice_2	20191114_sample_2
1314	mouse_YAVWS	20191114_slice_2	20191114_sample_3
1315	mouse_YAVWS	20191114_slice_3	20191114_sample_4
1316	mouse_YAVWS	20191114_slice_4	20191114_sample_5
1317	mouse_YAVWS	20191114_slice_6	20191114_sample_7
1318	mouse_YAVWS	20191114_slice_7	20191114_sample_8
1319	mouse_YAVWS	20191114_slice_8	20191114_sample_9

So how should we restructure these? Keep in mind, this is part of a cross-modality metadata table, and changing any of these will make it hard to match up with other sources. I think we may be creating more problems than it is worth by changing any of these metadata values. If our scope structure is dandiset > subject > slice > cell, then this could transform to:

...	subject_id	slice_id	cell_id
1310	ZNDVW	4	1
1311	ZNDVW	5	1
1312	YAVWS	1	1
1313	YAVWS	2	1
1314	YAVWS	2	2
1315	YAVWS	3	1
1316	YAVWS	4	1
1317	YAVWS	6	1
1318	YAVWS	7	1
1319	YAVWS	8	1

If I understand @satra correctly, I think this is what you want. We could start counting at 0 too if we want to get really crazy. The way they give it to us, it is entirely denormalized and all of the ids are unique (scoped to the dandiset). This new way, not only have we changed all of the id's, but also they will need to do a 3-way concatenation to recover unique ids (subject_id+slice_id+cell_id). I am of course capable of making this change, but I would not expect the Tolias Lab to be enthusiastic about this. We'd also have to convince BICCN and the other repos to use this convention. Thoughts? Do we want to move forward with the change as described above, or am I misunderstanding something?

[satra]

i would keep the cell id as whatever the lab wants to call it for now. especially because that is what would link presently to the other archives. for this dataset, iirc they didn't have subject or slice ids, only cell ids to begin with. the original filename looked like 20191108_sample_4.nwb

so a file could look like (if i were interpreting these values)
sub-ZNDVW_ses-20191108_slice-4_cell-20191108_sample_4.nwb

but if ses and slice are not helping with disambiguation within a subject it should simply look like:
sub-ZNDVW_cell-20191108_sample_4.nwb

so i don't want to enforce the values of those keys to the labs. i want a simple principle that says:

1. sub is mandatory
2. the rest are provided in order to disambiguate the filename as necessary and if my suggestion in this issue is implemented, then users will get to use what the disambiguating key order is.
3. try not to encode too much provenance into the value. keep the value simple.
4. the values of these keys should be in the nwb file.

yes, in the ideal world folks would label their metadata in a consistent way - unique id (not necessarily uuid) for subject, slice, and cell - and not include additional metadata into every key (e.g., datestamp or species)

any uuid should be up to the data format and the archive not the lab.

[bendichter]

OK so I'm glad I checked because I was misunderstanding. It sounds like we will not be changing IDs at all and this is all a matter of dandi organize settings. @yarikoptic looks like the ball is in your court here.

[bendichter]

Though to be clear, given the current way the metadata is, the filename would be
sub-mouse_ZNDVW_cell-20191108_sample_4.nwb. Not perfect but much better than it is now.

[yarikoptic]

so a file could look like (if i were interpreting these values)
sub-ZNDVW_ses-20191108_slice-4_cell-20191108_sample_4.nwb

(un)fortunately not -- _ is a separator between _key-value pairs. In https://github.com/dandi/dandi-cli/blob/master/dandi/organize.py#L221 we sanitize values by replacing _ with -, so should become something (I might have failed to parse now the boundaries above, since slice seems to not have simple index 4) like:

sub-ZNDVW_ses-20191108_slice-4_cell-20191108-sample-4.nwb.

It seems to such cases what could work is using the "lowest" level, here would be cell if it is sufficient to disambiguate between all the files, and thus (if we relax further to not make ses required):

`sub-ZNDVW_cell-20191108-sample-4.nwb`.

With above sanitization though - we should

• make it clear that metadata in file names might not 100% correspond to the metadata in the files. Actual values should be picked up from the metadata within files.
• recommend not using _ (and other tricky symbols and spaces ?!) in the metadata values.

[satra]

@yarikoptic - so should sub be mandatory in the filename? i don't have strong feelings about this, but would be willing to leave this as optional to the user if we have a sub folder.

second, do we deploy ordering option to user and use the simplest disambiguation within a folder? i.e. if ses doesn't matter across files in a folder it gets removed.

also how would we handle incremental updates? so today ses doesn't matter, but tomorrow it does, etc.,.

[yarikoptic]

@yarikoptic - so should sub be mandatory in the filename? i don't have strong feelings about this, but would be willing to leave this as optional to the user if we have a sub folder.

well, that is what I was proposing also in the early days of BIDS - to avoid duplication etc. I now have love/hate feeling to having sub and ses in both directories structure and filename. On one hand -- redundant. On another, I could just copy/share some (selected) files and not worry about renaming or placing them into subfolders to disambiguate. With reasonable subject ids, sub usually does not add much to the filename length. I would prefer to avoid it to be "optional" but rather prefer for us to decide one way or another.

And given large typical sizes of .nwb files, I do see people download/copying just a handful of files. If they start mixing up and overwriting one with another -- data loss, pain and suffering would come.

second, do we deploy ordering option to user and use the simplest disambiguation within a folder? i.e. if ses doesn't matter across files in a folder it gets removed.

sorry -- not entirely following on what "ordering option"? I think the order of the fields is pretty much from larger to smaller (whole subject -> session -> tissue -> cell ) and I do not think so far we saw need for an alternative order.

also how would we handle incremental updates? so today ses doesn't matter, but tomorrow it does, etc.,.

that is exactly the concern I had against "minimized" version or allowing any other optional flexibility. ATM, besides disambiguation via .nwb object id (before might not needed, then becomes needed or vise versa if files acquire new metadata) everything is consistent. You can incrementally organize into an existing dandiset an additional file, and it would take its place in the right location, as if when you reorganized full collection of files. In case of symlink or hardlink file mode, it is quite feasible to just rm sub-*; dandi organize -d . /path/to/full-collection quickly, thus to do incremental via full "redo". But then users might not see explicitly what actually has changed/got renamed. I have now added a comments to an existing issue on that topic.

[yarikoptic]

As #69 (comment) comment from @bendichter hints, we still might like to come back to dandi/dandi-schema#141 and also agree on specifying scopes of any id in the dandiset.yaml, which some defaults we assume stated somewhere.
That information could also assist in automatically deciding what keys are useful for deciding keys to use in the file name. E.g. if cell_id scope is tissue -- both must be present from the first file since their pair constitutes a unique ID (scope of which is the scope cell_id defined in). If cell_id's scope is anything higher than a session - could be used alone. If any of the id is higher than a subject -- sub- theoretically be omitted.

[satra]

i think we came to the conclusion when discussing with @bendichter that IDs are specific to the user/lab/consortium and we should not be overriding them. so they have no scope, just ids - whatever they decide to embed in the nwb file.

[yarikoptic]

I was thinking to file a fresh distilled version of this issue asking @jwodder to add --mandatory-fields option to organize, but then I decided against it. We need to figure out a more "holistic" approach:

• either such configuration should be a property of a dandiset to prevent users' running organize with different options thus again bringing dichotomy. cons: not really pertinent to dandiset.yaml I guess, but we do not have another location ATM which would be "propagating" across copies of a dandiset.
• or we analyze all assets and perform global dandi organize -f move in the end (see related fresh discussion Session has disappeared from filename construction #614 (comment)). cons: would not work for partial local ones.
• or we just instruct users with INFO message that if it is an incremental/partial organize (we might not know really if it is so reliably since it might be incremental addition of a subject into otherwise empty locally dandiset) - they needing dandi organize -f move on the entire dandiset.
• another wild supplemental idea: get a listing of assets on the server (if any) and use it in "organize" decision. But that would require knowing which server (ok, we can default), but also could bring problems whenever some local assets potentially changed versions of those which are already on the server.

So altogether, for the robust operation, it smells like we do need some kind of "dandiset-level" description on how any new addition should be organized to guarantee consistency.