CLI: graph extensions

[sbesson]

Following recent discussion with @manics @ximenesuk @joshmoore @mtbc

[ximenesuk]

obj was originally designed to handle commands of the form:

obj cmd Object:id field1=value1 field2=value2 ...

it has already been abused slightly to deal with other forms so if some of the graphspec syntax suggested in the design does make sense then it may be worth fully reviewing obj - or perhaps creating an obj2 and eventually replacing obj.

I'm all for both the use of Object:3-6 and Object:*, though obviously dealing with intermediate ids that are outwith the current permissions in a clean way.

[joshmoore]

EOB comments:

• My general feeling is that obj should remain an object level action, rather than dealing with graphs & sets, but that may be an outdated viewpoint (cF. IDelete.deleteObject)
• The other graph operations have well-known counterparts (chgrp, chmod), and so where something is graph-y and has a counterpart, trying to re-use it fits with other naming schemes.
• However, our concept of "duplicate" might be too distant from "cp" for this to hold up. If so, I'd favor having a single plugin which provides access to left-panel actions (similar to the way obj provides access to right-panel actions)
• This would certainly map with the listing commands (children, parents, ...) {Would a directory listing approach then be better? ls dunno}
• I'm also not sure we can approach a full language at this point. i.e. initially, explicit "orphan" or "contents" command/flag is likely more usable than //Image:*: e.g. bin/omero tree cp --link Image:1 ...
• At this point, I realize I don't know what this syntax should look like. 😄 But if we assume the listing sytnax is bin/omero tree X then I could imagine bin/omero tree X [(--move | --delete | --copy) target] being a good starting point.

[sbesson]

Makes sense, happy to incorporate this into the text file by differentiating proposal 1 (extend obj) and proposal 2 (tree/graph new command).

[manics]

One other thing that'd be useful is to filter on field=value e.g. annotation namespaces:

omero tree-obj ls Screen:1/*/FileAnnotation[ns='bulk-annotation']

(or replace namespaces with Annotation-Folders?)

[manics]

The previous comment also helps with some of the IDR scripts where the workflow is

1. omero import /path/to/ScreenNameA /path/to/ScreenNameB
2. omero hql get list of screen-ids and screen-names, parse them
3. omero metadata populate ... Screen:Id ScreenNameA-metadata-file

Step 2 could be replaced with something like omero tree-obj get Screen[name=ScreenNameA] id

[ximenesuk]

I'd second some of @joshmoore's comments and go fro a tree command to avoid overloading obj. It could though use a similar subcommand style for to obj:

bin/omero tree (move | delete | copy | ...) source [target]

A separate command to obj would certainly make it easier to develop a graph spec argument style along the lines of someof @manics's suggestions.

[sbesson]

Tried to capture most of the comments above into the last commits.

[joshmoore]

So, it looks like we're approaching:

• tree read-op graph-specs
• tree write-op graph-specs obj-spec

as the 2 basic command structures. (Or are there more?). Does that mean the primary discussion is the one or more formats of graph-spec?

• Image:id (i.e. obj-space)
• Image:orphan
• Image --orphan
• //Image/*
• Image[name=x]
• Image:id/Roi:*
• Screen:id/*/Roi:*
• Screen:1 --filter FileAnnotation[ns='bulk-annotation']
• Screen:1/*/FileAnnotation[ns='bulk-annotation']

are all mentioned. It would be good to have votes concerning: difficulty of implementing, current need, and aesthetic appeal for each of these.

My brief opinions to the open questions:

• "(orphan string vs option vs graph representation)" -- of these I think I'd vote for Image:orphan if we can stay away from xpath for the moment.
• "output of the listing commands?" -- Simon's output format was certainly useful. Perhaps arguments to ls could provide just IDs.
• "link vs copy" / "delete vs unlink" -- here I'd avoid delete (since that's already a graph operation) and stick closer to what the right-hand panel provides.
• "subcommand vs options, i.e. tree copy vs tree --copy" -- if it's happening at the front (i.e. if ls is explicit) then I'd say drop the -- from the command. e.g. either tree graph-spec --copy object-spec or tree copy graph-spec object-spec. The benefit of the former is probably only that ls can be omitted.)

[manics]

From the experimental metadata plugin:

$ omero metadata allanns Image:123

MapAnnotation:419

$ omero metadata allanns --parents Image:123

MapAnnotation:419
WARNING: Failed to get annotations for WellSample:121
FileAnnotation:360

$ omero metadata allanns --report --parents Image:123

Image:123
  MapAnnotation:419
    ns: openmicroscopy.org/omero/bulk_annotations
    description:
    date: 2016-04-04T14:55:04
    value:
      Gene Identifier=YLL019C
      Gene Identifier URL=http://www.ensembl.org/id/YLL019C
      Gene Symbol=KNS1
WARNING: Failed to get annotations for WellSample:121
  WellSample:121
    Well:128
      Plate:2
        Screen:2
          FileAnnotation:360
            ns: openmicroscopy.org/omero/bulk_annotations
            description: bulk_annotations
            date: 2016-04-04T14:55:03
            file: OriginalFile:242363
              name: bulk_annotations
              size: 275762

It's almost YAML, so we could go all the way and make it actual yaml/json

[joshmoore]

See https://trello.com/c/aRBg5h6O/577-omero-cli-yaml-input-output

[sbesson]

Pushed a last commit which captures the content of the last few comments. At this point, I suggest we review everything discussed above has been included in the first draft of the document, get this merged and handle new suggestions/improvements in new PRs during the OMERO 5.3.x scoping round.

[ximenesuk]

Here the xpath style reads much better than the older style. While there can be only images in a Dataset this command suggests the linking of the two Datasets rather than the contents of one. In this case I prefer the explicitness of the xpath format. Though from an example of tree copy below I assume it could also be expressed as:

omero tree link Dataset:1/* Dataset:2

[manics]

Image-Image links have been on the cards for a while, one day we might have Dataset-Dataset links.

[ximenesuk]

Yes, having a future-proof (as much as it is possible) format makes sense, the xpath format seems to offer more possibility for that.

Also here I would read:

omero tree link Dataset:1/Image:* Dataset:2
omero tree link Dataset:1/* Dataset:2

as equivalent while the only thing we can have in Datasets are Images but potentially different if we include things attached to Datasets. Is there a proposed way of distinguishing, say, Image from Annotations?

[manics]

How about something along the lines of:

• omero tree link Dataset:1/Image:* Dataset:2
• omero tree link Dataset:1/Annotation:* Dataset:2
• omero tree link Dataset:1/* Dataset:2 Everything including any future child types

To link some images (only those with a particular annotation):

• omero tree link Dataset:1/Image[annotation=123]: Dataset:2
• omero tree link Dataset:1/Image[annotation.value=abc]: Dataset:2

[sbesson]

I definitely see the readability of the xpath style over the other one. Would be interesting to hear the drawbacks (implementation time?). As suggested in #31 (comment), do you want to formalize your xpath proposal in a follow-up PR?

[manics]

Merge this then?