render.py: new CLI plugin for dealing with rendering settings (manual rebase onto metadata52)

[manics]

This was the same as gh-4385 gh-4430 but rebased onto metadata52. Now with additional commits.

---

See the bin/omero render -h output and the test for expected behavior. Primary driver for the creation of render.py was scripting the rendering settings along with the imports so that everything could be re-done if necessary:

PLATE=$(bin/omero import ... | grep Plate:)
bin/omero render copy Image:1 $PLATE

See IDR/idr-metadata#75 for information on the intended workflow for omero render edit.

Perhaps the main discussion point is the use of yaml/json for these settings, and the implementation of pydict_text_readers.py. load is designed to transparently handle:

• json or yaml (former is stricter on types and is better for generated definitions, the latter is more user friendly if you're crafting it by hand)
• /local/file/paths or remote OriginalFile:Ids
• If you're happy with this design I'll re-use it in the metadata plugin.
• For the future: could have json/yaml write methods which could tie in with the new obj/tree plugin CLI: graph extensions design#31 in which case pydict_text_readers.py should be renamed.

---

Additional commits implement omero render edit to allow rendering settings to be set on an image.

• Editing of channel names isn't implemented yet, and tests are pending.
• When testing this plugin I ran into several Too many open files errors on the server, and I'm not sure if it's an error in this code, a system misconfiguration, or an OMERO.server bug

[manics]

Update with a generic yaml/json file util

[manics]

Tests added. One is xfail/broken, see manics@3bf3749
(https://trello.com/c/lyyGuRow/657-incorrect-logical-channels-in-clitest-importplates)

Error in the Python test library fixed.

[will-moore]

Re yaml vv json: I can't help feeling that at some point it will be useful to support json in some format - simply because it is so ubiquitous and is the format used in the web API.
Perhaps it is not too much work to support both, just use the right parser depending on file extension?

I could also imagine grabbing rendering settings from an image I'm looking at in web and doing something like:

bin/omero render edit Screen:1 c=1|1:3797$0000FF,-2|45:549$00FF00&m=c&p=normal&ia=0&q=0.9&t=1&z=1&zm=93&x=0&y=0

[ximenesuk]

colin$ omero render info Dataset:1
TBD: DatasetI

TBD?

[ximenesuk]

colin$ omero render --help
   ...
    # List available rendering settings
    bin/omero render list Image:456
   ...
    # Render as JPEG
    bin/omero render jpeg Image:5 > test.jpg
    ...

I assume list and jpeg are work in progress?

[ximenesuk]

colin$ omero render --help
    ...
    # Show details of a rendering setting
    bin/omero render info RenderingDef:1
    ...

colin$ omero render info RenderingDef:1
TBD: RenderingDefI

and this is presumably work in progress?

[ximenesuk]

colin$ omero render info Image:1
rdefv1: model=color, z=14, t=0
tiles: False
ch0: active=True,color=0000FF,label=CFP-JP4,min=103.0,start=103.0,end=197.0,max=197.0
ch1: active=True,color=FF0000,label=RD-TR-PE,min=96.0,start=96.0,end=124.0,max=124.0

Would it be useful to get the Object:id forms back for things like RenderDef and Channel? Either in this command or this command with an option? (It is possible obj would be able to do this but it might be useful here.)

[joshmoore]

c=1|1:3797$0000FF,-2|45:549$00FF00&m=c&p=normal&ia=0&q=0.9&t=1&z=1&zm=93&x=0&y=0

@will-moore : I had thought about this, but 2 issues:

• there's a card for dropping the pipe, right?
• these are all symbols which aren't CLI-friendly, so something to be aware of.

[joshmoore]

@ximenesuk : the TBDs can mostly be assumed mine. Probably best to focus on if any of the implemented functionality is incorrect or any of the design needs further discussion. (@will-moore, I like the idea of auto-supporting multiple formats based on the file ending. I think https://github.com/openmicroscopy/openmicroscopy/pull/4542/files#diff-7a4e03af84e54b56d8c2247958cb1998 should make that seamless, but we'd have to check with @manics)

[ximenesuk]

@joshmoore understood, though documenting some of the TODOs may help in their not being forgotten.

[ximenesuk]

@manics do you have an example of a minimal YAML file that should work? This minimal JSON file does what I expect it to do:

  "channels" : {
    "1" : {
      "color" : "00FF00"
    }
  }
}

This similar YAML file doesn't.

---
  channels:
    1:
      color:"FF00FF"

I get this error

colin$ omero render edit Image:1 edittest1.yaml
ERROR: 'str' object has no attribute 'get'
Invalid channel description: color:"FF00FF"

I may be misunderstanding the YAML syntax, though a few variations give the same error.

Other than this problem it looks like it does happily deal with both file formats by extension /cc @will-moore @joshmoore

One further suggestion would be having render info spit out YAML/JSON as well as the default short form.

[ximenesuk]

Other than the YAML glitch above (which may be my misunderstanding) the plugin looks like it would be a useful addition. It certainly works as expected where implemented.

[joshmoore]

All items mentinoed captured in https://trello.com/c/8F8QMGr7/276-render-py-followup

I'd encourage any follow-up work on this PR to assume that it will be rebased on top of this PR before hitting the mainline.

[joshmoore]

For mitocheck, support for "greyscale" (see line 192) is needed in the yaml file (unless I'm overlooking it)

[manics]

Re yaml vv json: I can't help feeling that at some point it will be useful to support json in some format - simply because it is so ubiquitous and is the format used in the web API.

I think https://github.com/openmicroscopy/openmicroscopy/pull/4542/files#diff-7a4e03af84e54b56d8c2247958cb1998 should make that seamless

@will-moore @joshmoore Correct, it chooses between yaml or json depending on mimetype if available (application/x-yaml, application/json), followed by extension (yml, yaml, js, json). Suggestions on what file extensions to handle?

components/tools/OmeroPy/test/integration/metadata/test_pydict_text.py contains tests for the autodetection. The yaml/json examples are minimal (TestPydictTextRead.getTest{Json,Yaml}), but I think they're enough to distinguish between the two formats.

I could also imagine grabbing rendering settings from an image I'm looking at in web and doing something like:

One further suggestion would be having render info spit out YAML/JSON as well as the default short form.

@will-moore @ximenesuk I thought about this to enable a round-trip, but wanted some feedback on the general design before devoting more time. If everyone's happy I can see pydict_text_readers.py being extended to output yaml/json, and used in other plugins such as obj.

This similar YAML file doesn't.
color:"FF00FF"

You need a space after the :. PyYAML treats color:"FF00FF" as valid YAML for a single string, instead of a key: value.

For mitocheck, support for "greyscale" (see line 192) is needed in the yaml file (unless I'm overlooking it)

@joshmoore Not implemented yet

[joshmoore]

@joshmoore Not implemented yet

Happy for that to be outwith this PR (in general, ok to merge this?) but it will likely be needed for next week.

[manics]

@joshmoore If this isn't blocking anything I'll do further development here, if not merge it and I'll open another PR.

[joshmoore]

Nope, merging isn't blocking anything.

[manics]

• Added optional greyscale: <boolean> option, e.g.

channels:
  1:
    label: Channel 1
    color: FF0080
    min: 400
    max: 16000
  2:
  3:
    color: 808080
    active: False
    max: 251
    min: 112

greyscale: True

• Renamed/extended omero/util/pydict_text_io.py with a dump (output) method
• render info now has a --style option (modelled on hql --style)
  • json and yaml output formats only include the fields that render edit supports to enable a round-trip (and to minimise debate over field names, especially given the web-api/omero-marshall work). They are also only implemented for single images.
• render info uses ctx.out instead of print

There's lot's of potential for refactoring in the CLI, including using pydict_text_io for input/output in other plugins.

[joshmoore]

This breaks https://github.com/openmicroscopy/infrastructure/pull/61/files#diff-bf4435d4dde93c8ad0a4566ee4315a52R14 -- to be corrected in a follow-up PR.

[joshmoore]

Tested the latest changes with today's build (67). greyscale and style both work well. Merging

[atarkowska]

--rebased-to #5215