CLI duplicate command by ximenesuk · Pull Request #4574 · ome/openmicroscopy

ximenesuk · 2016-04-13T16:07:46Z

This PR adds a basic duplicate command to the CLI, this should duplicate simple objects and hierarchies. It includes integration tests for simple objects.

Testing

See:

$ omero duplicate --help
...
Examples:

    # Duplicate a dataset
    omero duplicate Dataset:50
    # Do the same reporting all the new duplicate objects
    omero duplicate Dataset:50 --report

    # Do a dry run of a duplicate reporting the outcome
    # if the duplicate had been run
    omero duplicate Dataset:53 --dry-run
    # Do a dry run of a duplicate, reporting all the objects
    # that would have been duplicated
    omero duplicate Dataset:53 --dry-run --report

Multiple IDs and objects should also work:

$ omero duplicate Dataset:263,264
$ omero duplicate Dataset:263 Dataset:264

In addition hierarchies (P/D/I for example) and other complex objects should be duplicated. Create a simple P/D/I structure or import a Screen, then:

$ omero duplicate Project:123 --report
$ omero duplicate Screen:321 --report

the --report option is useful here to check the graph duplication.

To do

Once this PR is merged @pwalczysko plans to add further tests for multiple arguments, simple hierarchies (P/D/I) and complex objects (Screen).

The command itself needs improving to deal with the options in @mtbc's underlying command.

manics · 2016-04-13T16:09:55Z

typo (Datset)

At least that is in my code and not @pwalczysko's! I noticed it when copying it to the PR description but thought I could sneak it through/leave it in case there are other changes.

…d object types

joshmoore · 2016-04-14T10:57:24Z

Probably the only other question from my side is whether or not to include this in 5.2.3 and/or hide it via OMERO_DEV_PLUGINS awaiting ome/design#31 discussions.

Duplicate command

ximenesuk · 2016-04-14T11:05:41Z

Reopening with @pwalczysko being identified as a contributor.

ximenesuk · 2016-04-14T13:33:59Z

@joshmoore re #4574 (comment) I'd say it works as a stand-alone command even though it may be that another command eventually subsumes it. However, at the moment it is limited by being an all or nothing duplicate. I'd vote for allowing it in to 5.2.3 as a useful utility but, for now, hiding it from non-developers.

joshmoore · 2016-04-18T08:26:30Z

+                self.ctx.out("  %s:%s" % (k, objIds[k]))
+
+try:
+    register("duplicate", DuplicateControl, HELP)


This still needs the DEV marker: see affa152

I will add today - I hadn't realised you had 👍ed my comment as I don't seem to get notifications of reactions.

…he process

mtbc · 2016-04-19T09:47:06Z

omero duplicate --help now doesn't seem to work. Is there a special trick since the OMERO_DEV_PLUGINS change?

ximenesuk · 2016-04-19T09:54:06Z

Have you added OMERO_DEV_PLUGINS to your env?

mtbc · 2016-04-19T09:56:02Z

No. What value do I put in it? (Is this documented anywhere?)

ximenesuk · 2016-04-19T09:56:28Z

No value, just export OMERO_DEV_PLUGINS= will do.

mtbc · 2016-04-19T09:56:46Z

Goodness, so it does, thank you!

ximenesuk · 2016-04-19T10:00:56Z

I don't think there is any documentation yet other than its use in

affa152

/cc @joshmoore

manics · 2016-04-19T10:01:53Z

It's for dev/internal use, so we probably don't want to document it.

sbesson · 2016-04-19T20:13:25Z

Tested using the daily DEV merge build. All worked as expected both with and without --dry-run mode. Also tested in the permissions context while duplicating other's data. Integration tests are passing. This is ready to merge.

On the OMERO_DEV_PLUGINS topic, I would be inclined to list it under ENV_HELP with sufficient warnings about its usage but I can see @manics point about not publicizing it. Looking at the list of CLI plugins, others could be classified under this category as well (perf, testengine) if not even deprecated.

joshmoore · 2016-04-19T20:30:19Z

(perf, testengine) if not even deprecated.

This is beyond this PR but as described in https://trello.com/c/tVSjHnee/55-add-warnings-for-windows-end-of-support there are several plugins (perf, testengine, submit) which should be deprecated and removed. In comparison, childre, duplicate, render, are "experimental" and the API will be decided on soon for inclusion. (Another way of doing this would be to have them in a separate repo and versioned low, e.g. 0.1.0, and then install them via pip in a virtualenv.)

manics · 2016-04-19T21:21:56Z

On the OMERO_DEV_PLUGINS topic, I would be inclined to list it under ENV_HELP with sufficient warnings about its usage but I can see @manics point about not publicizing it.

The alternative is to not include these plugins in the mainline, so I think it's best to treat them as non-existent. Originally they were only in the metadata branch, but since they require changes to other files outside of plugins this would lead to conflicts as the two branches diverged even more. We could add just a subset of files but this makes things more complicated, and difficult to test.

…ication, clean up

joshmoore · 2016-04-22T19:29:55Z

On the OMERO_DEV_PLUGINS topic, I would be inclined to list it under ENV_HELP ...

👍 Minimally that should help the team remember what the secret invocation was. (Can be external to this PR)

Add tests for hierarchy and skiphead duplication

mtbc · 2016-04-28T08:33:22Z

+        params.addId(did)
+        query = ("select p from Project as p where exists"
+                 "(select l from ProjectDatasetLink as l where "
+                 "l.child.id=:id and l.parent=p.id)")


this might simply be something like,
SELECT DISTINCT parent FROM ProjectDatasetLink WHERE child.id = :id

Thanks for the suggestion @mtbc, @pwalczysko may have adapted his query from elsewhere in our codebase?

As @ximenesuk suggested, I indeed did adapt this code from the import test https://github.com/openmicroscopy/openmicroscopy/blob/develop/components/tools/OmeroPy/test/integration/clitest/test_import.py#L189.
@mtbc : Would it be okay to merge this as it is and return to the cleanup of the queries in the tests later ?

Sure, not blocking, just commenting. Be warned that there are some quite exciting examples of HQL around our codebase. 😃

jburel · 2016-04-28T16:00:36Z

Reading the comments, especially #4574 (comment)
it looks like we need to scope "pip install plugins", we will need the same for web. This could be done under Facilities/Institutions management. Card created https://trello.com/c/KUGBzOdm/22-pip-install-plugins

Better to hold off for 5.2.3

mtbc · 2016-05-15T19:42:12Z

What more needs doing here -- review tests?

joshmoore · 2016-05-16T07:33:20Z

My understanding is that this was just waiting for 5.2.3 to be released.

ximenesuk · 2016-05-16T07:40:08Z

Yes, that was my understanding too. If any further tests were to be needed @pwalczysko would open a PR directly against develop once this is merged,

pwalczysko · 2016-05-16T14:36:32Z

So should this be relisted ? Would give me a boost if this could go in really :)

joshmoore · 2016-05-16T19:03:35Z

Well in that case .... merging. I will organize a general discussion on the de-DEV-PLUGIN-ification in the summer as we try to get the IDR work back on the mainline. Thanks all!

Add a basic duplicate command

293bb92

manics reviewed Apr 13, 2016
View reviewed changes

pwalczysko added 5 commits April 14, 2016 11:53

Add basic test for duplicate

cf88a58

Rename the test to more suitable name

b4e9a71

Revise test with further assertions

6f9cb1d

Add test for --dry-run

33adf63

Add test for --dry-run & --report and remove iterations over model an…

0e1c189

…d object types

ximenesuk force-pushed the duplicate-command branch from beeea1e to 293bb92 Compare April 14, 2016 10:58

ximenesuk closed this Apr 14, 2016

Colin Blackburn added 2 commits April 14, 2016 12:02

Merge pull request #15 from pwalczysko/duplicate-command

202d0bc

Duplicate command

Add copyright and comment

9ed2456

ximenesuk reopened this Apr 14, 2016

Clarify duplication in help and fix typo

47b5245

sbesson added the develop label Apr 14, 2016

joshmoore reviewed Apr 18, 2016
View reviewed changes

Colin Blackburn and others added 4 commits April 18, 2016 15:56

Disable plugin for normal use

2fbf9f1

Clarify the clarification in help

fbc6c70

Add test for simple P/D/I hierarchy duplication

0f6772d

Add check that the P/D linked to the D/I are the ones duplicated in t…

6ae0446

…he process

mtbc mentioned this pull request Apr 20, 2016

fix #13197: allow Duplicate to be used with SkipHead #4593

Merged

pwalczysko added 6 commits April 22, 2016 14:27

Neaten up the hql query

ff1f0c9

Immediately check whether 2 Projects are present

b585ac0

Check immediately for the duplicated Dataset and its linkage

75d35d5

Add immediate check for images and round up the test for P and D dupl…

8de09cc

…ication, clean up

Add test for the --report output content, introduce pid variable

cbdf4fc

Do not check just for IDs, but for the Object:ID string

86df64d

pwalczysko and others added 3 commits April 25, 2016 17:59

Add test for Skiphead functionality of Duplicate command

aa49995

Neaten up the project query

d71b522

Merge pull request #16 from pwalczysko/duplicate-command-further-tests

0885367

Add tests for hierarchy and skiphead duplication

mtbc reviewed Apr 28, 2016
View reviewed changes

jburel added the ON_HOLD label Apr 28, 2016

joshmoore mentioned this pull request Apr 28, 2016

setup.py template based on python-package-template ome/omero-cli-duplicate#1

Merged

jburel removed the ON_HOLD label May 4, 2016

joshmoore merged commit 991c41b into ome:develop May 16, 2016

sbesson added this to the 5.3.0 milestone Sep 10, 2016

Conversation

ximenesuk commented Apr 13, 2016

Testing

To do

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joshmoore commented Apr 14, 2016

Uh oh!

ximenesuk commented Apr 14, 2016

Uh oh!

ximenesuk commented Apr 14, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mtbc commented Apr 19, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ximenesuk commented Apr 19, 2016

Uh oh!

mtbc commented Apr 19, 2016

Uh oh!

ximenesuk commented Apr 19, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mtbc commented Apr 19, 2016

Uh oh!

ximenesuk commented Apr 19, 2016

Uh oh!

manics commented Apr 19, 2016

Uh oh!

sbesson commented Apr 19, 2016

Uh oh!

joshmoore commented Apr 19, 2016

Uh oh!

manics commented Apr 19, 2016

Uh oh!

joshmoore commented Apr 22, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jburel commented Apr 28, 2016

Uh oh!

mtbc commented May 15, 2016

Uh oh!

joshmoore commented May 16, 2016

Uh oh!

ximenesuk commented May 16, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pwalczysko commented May 16, 2016

Uh oh!

joshmoore commented May 16, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

mtbc commented Apr 19, 2016 •

edited

Loading

ximenesuk commented Apr 19, 2016 •

edited

Loading

ximenesuk commented May 16, 2016 •

edited

Loading