CLI Metadata Plugin by manics · Pull Request #4037 · ome/openmicroscopy

manics · 2015-08-11T17:31:03Z

For inspecting the availability of "metadata" (items shown in the right-hand panel of the GUIs) from the CLI, a bin/omero metadata plugin is being added along with tests to show how to add and query such data. The plugin also wraps populate_metadata.py and populate_roi.py mostly to simplify login.

Also:

populate_metadata and populate_roi are now integrated into this CLI plugin (though calling the original scripts directly should still work)
fixes for populate_roi
commands for listing annotations
commands for listing and deleting ROIs
Initial implementation of https://trello.com/c/vcpdnS92/38-work-on-bulk-annotations-vs-map-annotations

Testing:

omero metadata --help
omero metadata populate --file input.csv Screen:123
omero metadata populate --file input.csv Plate:123
omero metadata populate --context bulkmap (uses bulk-annotation OMERO.table)
omero metadata populate --context bulkmap --file fileid
omero metadata populateroi Plate:123
omero metadata {summary,original,measures,bulkanns,mapanns,allannas,rois} ...
omero metadata rois Object:Id
and for some sub-commands try the --report flag for additional output

@pwalczysko The fix for populate_roi should be in here thanks to @joshmoore

For a server who's policy restricts the download of plates, populate_roi.py can no longer get access to the results file. For the moment, we're detecting files based on their file ending and permitting the download.

This is an initial example to show how a bulk-annotation table can be converted into MapAnnotations

This is for internal testing only, and will be changed when this is moved to the metadata CLI plugin.

BlitzGateway didn't work with self.client

Initial attempt at integrating the populate_metadata.py script into the OMERO CLI

Also fix ignored name parameter in make_file_annotation

This is consistent with the fs plugin

ximenesuk · 2015-08-18T12:12:42Z

I have tried the workflow at https://www.openmicroscopy.org/private/ome-internal/testing_scenarios/BulkAnnotations.html but using the cli without much luck.

ls30778:openmicroscopy colin$ omero metadata populate --file ~/Desktop/Bulk\ annotations/test_screen.csv Screen:1
Using session 54a20a55-d317-4db6-89ef-dc012c19b8d1 (user-0@localhost:4064). Idle timeout: 10 min. Current group: pg-0
WARNING:omero.util.populate_metadata:Screen is missing plate: 365700055641
WARNING:omero.util.populate_metadata:Screen is missing plate: 365700055641
WARNING:omero.util.populate_metadata:Screen is missing plate: 365700055641
WARNING:omero.util.populate_metadata:Screen is missing plate: 365700055641
WARNING:omero.util.populate_metadata:Screen is missing plate: 365700055641
WARNING:omero.util.populate_metadata:Screen is missing plate: 365700055641
WARNING:omero.util.populate_metadata:Screen is missing plate: 365700055641
WARNING:omero.util.populate_metadata:Screen is missing plate: 365700055641
WARNING:omero.util.populate_metadata:Screen is missing plate: 365700055641
WARNING:omero.util.populate_metadata:Screen is missing plate: 365700055641
WARNING:omero.util.populate_metadata:Screen is missing plate: 365700055641
WARNING:omero.util.populate_metadata:Screen is missing plate: 365700055641
WARNING:omero.util.populate_metadata:Screen is missing plate: 365700055641
WARNING:omero.util.populate_metadata:Screen is missing plate: 365700055641
WARNING:omero.util.populate_metadata:Screen is missing plate: 365700055641
WARNING:omero.util.populate_metadata:Screen is missing plate: 365700055641
WARNING:omero.util.populate_metadata:PlateColumn is unimplemented
INFO:omero.util.populate_metadata:Created new table OriginalFile:767
INFO:omero.util.populate_metadata:Table initialized with 11 columns.
INFO:omero.util.populate_metadata:Added data column data.
ls30778:openmicroscopy colin$ omero metadata summary Screen:1
Using session 54a20a55-d317-4db6-89ef-dc012c19b8d1 (user-0@localhost:4064). Idle timeout: 10 min. Current group: pg-0
Screen:1
--------
Name: Screen1
Bulk annotations: 1
FileAnnotations: 1

There is an annotation on the Screen but not on individual wells. And with,

ls30778:openmicroscopy colin$ omero metadata populate --file ~/Desktop/Bulk\ annotations/test_plate.csv Plate:3
Using session 54a20a55-d317-4db6-89ef-dc012c19b8d1 (user-0@localhost:4064). Idle timeout: 10 min. Current group: pg-0
INFO:omero.util.populate_metadata:Missing image name column, skipping.
INFO:omero.util.populate_metadata:Missing plate name column, skipping.
INFO:omero.util.populate_metadata:Missing image name column, skipping.
INFO:omero.util.populate_metadata:Missing plate name column, skipping.
INFO:omero.util.populate_metadata:Missing image name column, skipping.
INFO:omero.util.populate_metadata:Missing plate name column, skipping.
INFO:omero.util.populate_metadata:Missing image name column, skipping.
INFO:omero.util.populate_metadata:Missing plate name column, skipping.
INFO:omero.util.populate_metadata:Missing image name column, skipping.
INFO:omero.util.populate_metadata:Missing plate name column, skipping.
INFO:omero.util.populate_metadata:Missing image name column, skipping.
INFO:omero.util.populate_metadata:Missing plate name column, skipping.
INFO:omero.util.populate_metadata:Missing image name column, skipping.
INFO:omero.util.populate_metadata:Missing plate name column, skipping.
INFO:omero.util.populate_metadata:Missing image name column, skipping.
INFO:omero.util.populate_metadata:Missing plate name column, skipping.
INFO:omero.util.populate_metadata:Missing image name column, skipping.
INFO:omero.util.populate_metadata:Missing plate name column, skipping.
INFO:omero.util.populate_metadata:Missing image name column, skipping.
INFO:omero.util.populate_metadata:Missing plate name column, skipping.
INFO:omero.util.populate_metadata:Missing image name column, skipping.
INFO:omero.util.populate_metadata:Missing plate name column, skipping.
INFO:omero.util.populate_metadata:Missing image name column, skipping.
INFO:omero.util.populate_metadata:Missing plate name column, skipping.
INFO:omero.util.populate_metadata:Missing image name column, skipping.
INFO:omero.util.populate_metadata:Missing plate name column, skipping.
INFO:omero.util.populate_metadata:Missing image name column, skipping.
INFO:omero.util.populate_metadata:Missing plate name column, skipping.
INFO:omero.util.populate_metadata:Missing image name column, skipping.
INFO:omero.util.populate_metadata:Missing plate name column, skipping.
INFO:omero.util.populate_metadata:Missing image name column, skipping.
INFO:omero.util.populate_metadata:Missing plate name column, skipping.
Traceback (most recent call last):
  File "/Users/colin/Work/repos/openmicroscopy/dist/bin/omero", line 125, in <module>
    rv = omero.cli.argv()
  File "/Users/colin/Work/repos/openmicroscopy/dist/lib/python/omero/cli.py", line 1437, in argv
    cli.invoke(args[1:])
  File "/Users/colin/Work/repos/openmicroscopy/dist/lib/python/omero/cli.py", line 951, in invoke
    stop = self.onecmd(line, previous_args)
  File "/Users/colin/Work/repos/openmicroscopy/dist/lib/python/omero/cli.py", line 1028, in onecmd
    self.execute(line, previous_args)
  File "/Users/colin/Work/repos/openmicroscopy/dist/lib/python/omero/cli.py", line 1110, in execute
    args.func(args)
  File "/Users/colin/Work/repos/openmicroscopy/dist/lib/python/omero/plugins/metadata.py", line 324, in populate
    ctx.write_to_omero()
  File "/Users/colin/Work/repos/openmicroscopy/dist/lib/python/omero/util/populate_metadata.py", line 572, in write_to_omero
    "Unable to create table: %s" % name)
omero.util.populate_metadata.MetadataError: Unable to create table: bulk_annotations

Is there anything else I need on my local branch? And @pwalczysko is that scenario and are the files used valid?

pwalczysko · 2015-08-18T12:27:02Z

@ximenesuk : The scenario was last tested during the Paris meeting this year. It worked up to the points which are documented in the tickets. One of the points in the tickets is that when you run the script against a Screen (as you did above in step 1) then the annotation on single wells is missing. When run against a plate this shoiuld work as of June this year. The scenario was not updated with respect to this PR (scenario is unaware of this PR). I do not think you should be missing anything on your branch. If your setup is wrong, in my experience you would not get accross to the output like

NFO:omero.util.populate_metadata:Missing image name column, skipping.

pwalczysko · 2015-08-18T12:29:57Z

@ximenesuk : The "populate_roi.py does not work" ticket is https://trac.openmicroscopy.org/ome/ticket/12235

ximenesuk · 2015-08-18T12:32:12Z

@pwalczysko Note from the description that this PR fixes populate_roi though I have not yet tested that.

@manics do you have any sources of test data that should test the cli? And any guidance on --context ?

pwalczysko · 2015-08-18T12:38:10Z

@ximenesuk : This is the other ticket considering the populate_metadata.py - the two tickets pasted by me here fully describe all my knowledge about the status of this problematics (as indicated, my last test on this was done this year on the users' meeting in Paris) https://trac.openmicroscopy.org/ome/ticket/12211#comment:12

manics · 2015-08-18T12:42:27Z

--context is an exploratory option for some of the IDR work, so it may be removed/changed before this ends up in develop (sorry for not describing it, I thought josh was around this week and would be reviewing it).

Essentially,
OMERO-TEST/bin/omero metadata populate --context bulkmap Object:Id should convert a bulk-annotations table (previously added using populate) into a MapAnnotation on each image or well (depending on what was referenced in the BulkAnnotations table). The MapAnnotations are in the bulk-annotations namespace. See the Trello card if you're interested in the rationale behind this.

The corresponding UI work has not been done- i.e. the table annotation is not removed, and annotations on wells aren't displayed, only those on images. #4038 is a temporary hack to force the display of Image and Well annotations.

OMERO-TEST/bin/omero metadata populate --context deletemap Object:Id should remove all MapAnnotations created as a result of the bulkmap context.

I'd expect the behaviour of populate_metadata to be unchanged in terms of what does/doesn't work. populate_roi should be fixed, at least for the test scenario.

manics · 2015-08-18T12:49:14Z

omero metadata populate --report Plate:XX --file bulk_annotations/test_plate.csv works for me, do you have any server error logs?

manics · 2015-08-18T12:53:22Z

And sources of data...just what's in the scenario. Assuming populate_rois works Well/Image A1 should have 173 ROIs, and you can use it to test most of the sub-commands e.g. bulkanns/measures sub-commands with {--parents,--report} options:
omero metadata bulkanns Image:ID --parents --report

ximenesuk · 2015-08-18T13:10:37Z

Okay, so after a restart of the server the bulk annotation of a Plate worked. However, the webclient on my local server is taking 15 seconds to respond to every request so I think the earlier problem is likely a local issue unless there is anything else in the metadat branch that might complicate things. I'll continue later after a laptop restart.

manics · 2015-08-18T13:16:06Z

You're welcome to test against dogfish (which is where the openmicroscopy:metadata merge-branch is deployed)

ximenesuk · 2015-08-19T11:48:33Z

I think I have gone as far as I can with this one without spending a bit more time catching up on the IDR side of things. I've tested the basic listing and populating using --file but not any of the --context stuff. No problems other than the flakiness of my server. It may be that @joshmoore can review the areas I haven't more easily?

manics · 2015-08-19T12:03:38Z

I've got another PR building on top of this, further extending the plugin, and it's staying on the metadata branch for now. I think the important thing as far as testing is concerned is it doesn't break the existing populate_* functionality (or anything else).

manics · 2015-08-25T17:01:51Z

@joshmoore Over to you for further comments (note I'm already building on this in #4071 so if you prefer I could close this one and have the whole plugin reviewed at once)

joshmoore · 2015-09-04T08:18:42Z

Various issues while testing: (I'll update this comment)

o metadata original Screen:2 fails
o metadata summary Screen:1 fails if it doesn't exist
REF: o metadata populate --attach to attach the CSV for download

manics · 2015-09-16T18:12:09Z

First two points in above comment fixed in #4062 08faf6f

manics · 2015-09-16T18:13:12Z

What do you prefer, close this PR and review #4062 as one big PR when it's done? Or merge this (without fixes) since it doesn't break anything?

FYI Your fs changes are in commit manics@8347d8a which only changes plugins/fs.py.

manics · 2015-09-16T18:41:43Z

Regarding the RFE metadata populate --attach, if you can attach the CSV this implies you should also be able to create the bulk-annotations from a previously uploaded OriginalFile, which is fine.

However, with the increasing number of changes to existing methods in populate_metadata.py should __main__ be removed so omero metadata populate becomes the only way to run it?

manics · 2015-09-18T10:14:52Z

@pwalczysko Regarding this PR, would you have any objections to me removing populate_{metadata,roi}.py in a future PR and only supporting omero metadata populate ... in 5.2, obviously with updated docs?

pwalczysko · 2015-09-21T13:58:35Z

@manics : assuming that omero metadata populate... does at least the same scope or even more as did populate_(metadata, roi).py, I think this is fine. The only question would be whether we know about places which are totally dependent on the old scripts functioning ? @joshmoore @sbesson

joshmoore · 2015-10-02T15:07:45Z

@pwalczysko : I assume will finish that conversation when this comes back to develop, perhaps for 5.2?

Merging for SiB.

cc: @jburel @ximenesuk

CLI Metadata Plugin

joshmoore and others added 30 commits August 3, 2015 11:16

Initial metadata.py plugin

5bf235e

test_metadata.py: first 'original' test

fd856ba

Rename test_populate_metadata.py for adding rois

b1e62ed

library.py: add importPlates method

841a665

test_populate.py Add ROI test

e8ae26d

BinaryAccessPolicy: workaround for .txt

a46aa27

For a server who's policy restricts the download of plates, populate_roi.py can no longer get access to the results file. For the moment, we're detecting files based on their file ending and permitting the download.

fs.py: add ls command for listing files

8347d8a

metadata.py: refactor and clean

cdc69b7

metadata.py: add populate method

b19d819

populate_roi.py: add support for OMERO 5 (Fix #12235)

a5b270e

Fix flake8

5b0fda0

metadata.py: error in missing measurements

84fb66c

library.py: fix importMIF series logic

f5c3a54

Add wrapper method for handling lists and singles

2a6fac3

Output bulk annotations, optionally search parents

415085e

Flesh out summary, add allanns command

d7b4531

Add method for formatting annotations. Add mapanns command

f871a3e

Formatting improvements. summary shows annotation counts

23ddef6

Use try-except blocks around each attribute in summary

79f9275

Add BulkToMapAnnotationContext to create MapAnns from Bulk Anns

0e042f3

This is an initial example to show how a bulk-annotation table can be converted into MapAnnotations

Add basic usage information for BulkToMapAnnotationContext

750791d

This is for internal testing only, and will be changed when this is moved to the metadata CLI plugin.

Only annotate Images and Well, ignore Screens and Plates

41fdef7

Fix typo in BulkToMapAnnotationContext file arg

ef30f79

Add context to delete MapAnnotations in the BulkAnns namespace

6ec3f33

Fix delete in DeleteMapAnnotationContext

b68d98e

BlitzGateway didn't work with self.client

omero metadata populate runs populate_metadata.py

be35258

Initial attempt at integrating the populate_metadata.py script into the OMERO CLI

Add namespace param to library.ITest annotations

beef5e8

Also fix ignored name parameter in make_file_annotation

Add tests for CLI Metadata wrapper

29b39e6

Use --report for verbose output

7af80fa

This is consistent with the fs plugin

Split common methods into MetadataTestBase, add roi test

76f9e53

Add --dry-run option to metadata rois delete

80adddc

manics mentioned this pull request Aug 18, 2015

Configuration file for controlling display of bulk-annotations #4062

Closed

manics mentioned this pull request Aug 20, 2015

Add get subcommand to CLI obj command #4071

Merged

joshmoore added a commit that referenced this pull request Oct 2, 2015

Merge pull request #4037 from manics/cli-metadata

f645cdb

CLI Metadata Plugin

joshmoore merged commit f645cdb into ome:metadata Oct 2, 2015

manics deleted the cli-metadata branch November 25, 2015 17:38

joshmoore mentioned this pull request Jan 21, 2016

Extenstions to populate metadata 52 #4426

Merged

sbesson removed the ON_HOLD label Feb 3, 2016

manics mentioned this pull request Feb 3, 2016

Populate metadata example #4455

Merged

joshmoore mentioned this pull request Feb 4, 2016

Temporary web UI workarounds (rebased onto metadata52) #4457

Merged

Conversation

manics commented Aug 11, 2015

Uh oh!

ximenesuk commented Aug 18, 2015

Uh oh!

pwalczysko commented Aug 18, 2015

Uh oh!

pwalczysko commented Aug 18, 2015

Uh oh!

ximenesuk commented Aug 18, 2015

Uh oh!

pwalczysko commented Aug 18, 2015

Uh oh!

manics commented Aug 18, 2015

Uh oh!

manics commented Aug 18, 2015

Uh oh!

manics commented Aug 18, 2015

Uh oh!

ximenesuk commented Aug 18, 2015

Uh oh!

manics commented Aug 18, 2015

Uh oh!

ximenesuk commented Aug 19, 2015

Uh oh!

manics commented Aug 19, 2015

Uh oh!

manics commented Aug 25, 2015

Uh oh!

joshmoore commented Sep 4, 2015

Uh oh!

manics commented Sep 16, 2015

Uh oh!

manics commented Sep 16, 2015

Uh oh!

manics commented Sep 16, 2015

Uh oh!

manics commented Sep 18, 2015

Uh oh!

pwalczysko commented Sep 21, 2015

Uh oh!

joshmoore commented Oct 2, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants