tables: retry repo_uuid lookup by joshmoore · Pull Request #5593 · ome/openmicroscopy

joshmoore · 2017-12-06T07:55:08Z

This adds on to 1264642 (gh-5481)
but not just adding a wait to the .omero/repository lookup but as
well to the .omero/repository/*/repo_uuid lookup, so that the 20
REPEATS don't progress within 2-3 seconds but instead take long
enough for the integration server has time to configure itself

cF. https://ci.openmicroscopy.org/view/Failing/job/OMERO-DEV-merge-training/811/

Testing this PR

table tables (incl. merge-training) should be green for an extended period
setting omero.repo.wait to a longer period (in MT) should help alleviate any issues

This adds on to 1264642 (omegh-5481) but not just adding a wait to the .omero/repository lookup but as well to the .omero/repository/*/repo_uuid lookup, so that the 20 REPEATS don't progress within 2-3 seconds but instead take long enough for the integration server has time to configure itself

mtbc · 2017-12-06T09:23:08Z

I wonder why legacyRepositoryTakeoverRun takes so long in the first place, mainly Hibernate?

joshmoore · 2017-12-06T09:25:20Z

Would certainly be worth adding some debugging to figure it out. E.g. a "starting hibernate initialize" and "finishing hibernate initialization" in the log so that we know how long. (There are certainly DEBUG level statements that would show that)

joshmoore · 2017-12-06T21:36:29Z

The downstream jobs from https://ci.openmicroscopy.org/job/OMERO-DEV-merge-integration/863/ were nicely green:

dominikl · 2017-12-18T11:42:42Z

Looks good. And I've not seen the typical tables not enabled error in the integration jobs since the PR is opened, so good to merge from my point of view 👍

mtbc · 2018-01-03T09:11:59Z

Re. #5593 (comment) I'm happy to open a #5586-like PR if that could help, not sure which package/class(es) are best targeted.

joshmoore · 2018-01-05T08:45:36Z

@mtbc: likely we should take a step back and see if we can't re-architect how the repositories are acquiring their UUIDs in the first place. This strategy has proven generally brittle and prone to race-conditions.

joshmoore · 2018-01-12T08:53:19Z

Subjobs of https://ci.openmicroscopy.org/view/Failing/job/OMERO-DEV-merge-integration/894/ still failed since the tables server didn't start:

2018-01-12 00:46:53,996 INFO  [                       omero.util.Server] (MainThread) Waiting 10000 ms on startup
2018-01-12 00:47:03,996 INFO  [                       omero.util.Server] (MainThread) Starting
2018-01-12 00:47:04,016 INFO  [                       omero.util.Server] (MainThread) Loaded dependency numpy (1.4.1)
2018-01-12 00:47:04,017 INFO  [                       omero.util.Server] (MainThread) Loaded dependency tables (2.4.0, hdf=1.8.7)
2018-01-12 00:47:04,648 INFO  [                    omero.util.Resources] (Thread-2  ) Starting
2018-01-12 00:47:41,650 INFO  [                             omero.utils] (MainThread) Failed to get session on attempt 1
2018-01-12 00:47:51,864 INFO  [                    omero.tables.TablesI] (MainThread) Using storage factory: omero.hdfstorageV1.HdfList
2018-01-12 00:47:52,034 INFO  [                    omero.tables.TablesI] (MainThread) /repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid doesn't exist
2018-01-12 00:47:52,034 WARNI [                    omero.tables.TablesI] (MainThread) Failed to find repo_svc: [Errno 2] No such file or directory: path(u'/repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid')
2018-01-12 00:47:52,160 INFO  [                    omero.tables.TablesI] (MainThread) /repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid doesn't exist
2018-01-12 00:47:52,161 WARNI [                    omero.tables.TablesI] (MainThread) Failed to find repo_svc: [Errno 2] No such file or directory: path(u'/repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid')
2018-01-12 00:47:52,345 INFO  [                    omero.tables.TablesI] (MainThread) /repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid doesn't exist
2018-01-12 00:47:52,345 WARNI [                    omero.tables.TablesI] (MainThread) Failed to find repo_svc: [Errno 2] No such file or directory: path(u'/repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid')
2018-01-12 00:47:52,521 INFO  [                    omero.tables.TablesI] (MainThread) /repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid doesn't exist
2018-01-12 00:47:52,521 WARNI [                    omero.tables.TablesI] (MainThread) Failed to find repo_svc: [Errno 2] No such file or directory: path(u'/repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid')
2018-01-12 00:47:52,741 INFO  [                    omero.tables.TablesI] (MainThread) /repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid doesn't exist
2018-01-12 00:47:52,743 WARNI [                    omero.tables.TablesI] (MainThread) Failed to find repo_svc: [Errno 2] No such file or directory: path(u'/repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid')
2018-01-12 00:47:52,953 INFO  [                    omero.tables.TablesI] (MainThread) /repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid doesn't exist
2018-01-12 00:47:52,953 WARNI [                    omero.tables.TablesI] (MainThread) Failed to find repo_svc: [Errno 2] No such file or directory: path(u'/repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid')
2018-01-12 00:47:53,118 INFO  [                    omero.tables.TablesI] (MainThread) /repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid doesn't exist
2018-01-12 00:47:53,119 WARNI [                    omero.tables.TablesI] (MainThread) Failed to find repo_svc: [Errno 2] No such file or directory: path(u'/repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid')
2018-01-12 00:47:53,308 INFO  [                    omero.tables.TablesI] (MainThread) /repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid doesn't exist
2018-01-12 00:47:53,309 WARNI [                    omero.tables.TablesI] (MainThread) Failed to find repo_svc: [Errno 2] No such file or directory: path(u'/repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid')
2018-01-12 00:47:53,522 INFO  [                    omero.tables.TablesI] (MainThread) /repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid doesn't exist
2018-01-12 00:47:53,522 WARNI [                    omero.tables.TablesI] (MainThread) Failed to find repo_svc: [Errno 2] No such file or directory: path(u'/repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid')
2018-01-12 00:47:53,764 INFO  [                    omero.tables.TablesI] (MainThread) /repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid doesn't exist
2018-01-12 00:47:53,764 WARNI [                    omero.tables.TablesI] (MainThread) Failed to find repo_svc: [Errno 2] No such file or directory: path(u'/repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid')
2018-01-12 00:47:54,007 INFO  [                    omero.tables.TablesI] (MainThread) /repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid doesn't exist
2018-01-12 00:47:54,008 WARNI [                    omero.tables.TablesI] (MainThread) Failed to find repo_svc: [Errno 2] No such file or directory: path(u'/repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid')
2018-01-12 00:47:54,248 INFO  [                    omero.tables.TablesI] (MainThread) /repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid doesn't exist
2018-01-12 00:47:54,248 WARNI [                    omero.tables.TablesI] (MainThread) Failed to find repo_svc: [Errno 2] No such file or directory: path(u'/repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid')
2018-01-12 00:47:54,459 INFO  [                    omero.tables.TablesI] (MainThread) /repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid doesn't exist
2018-01-12 00:47:54,460 WARNI [                    omero.tables.TablesI] (MainThread) Failed to find repo_svc: [Errno 2] No such file or directory: path(u'/repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid')
2018-01-12 00:47:54,667 INFO  [                    omero.tables.TablesI] (MainThread) /repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid doesn't exist
2018-01-12 00:47:54,667 WARNI [                    omero.tables.TablesI] (MainThread) Failed to find repo_svc: [Errno 2] No such file or directory: path(u'/repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid')
2018-01-12 00:47:54,835 INFO  [                    omero.tables.TablesI] (MainThread) /repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid doesn't exist
2018-01-12 00:47:54,835 WARNI [                    omero.tables.TablesI] (MainThread) Failed to find repo_svc: [Errno 2] No such file or directory: path(u'/repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid')
2018-01-12 00:47:54,976 INFO  [                    omero.tables.TablesI] (MainThread) /repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid doesn't exist
2018-01-12 00:47:54,976 WARNI [                    omero.tables.TablesI] (MainThread) Failed to find repo_svc: [Errno 2] No such file or directory: path(u'/repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid')
2018-01-12 00:47:55,165 INFO  [                    omero.tables.TablesI] (MainThread) /repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid doesn't exist
2018-01-12 00:47:55,165 WARNI [                    omero.tables.TablesI] (MainThread) Failed to find repo_svc: [Errno 2] No such file or directory: path(u'/repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid')
2018-01-12 00:47:55,331 INFO  [                    omero.tables.TablesI] (MainThread) /repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid doesn't exist
2018-01-12 00:47:55,332 WARNI [                    omero.tables.TablesI] (MainThread) Failed to find repo_svc: [Errno 2] No such file or directory: path(u'/repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid')
2018-01-12 00:47:55,403 INFO  [                    omero.tables.TablesI] (MainThread) /repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid doesn't exist
2018-01-12 00:47:55,403 WARNI [                    omero.tables.TablesI] (MainThread) Failed to find repo_svc: [Errno 2] No such file or directory: path(u'/repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid')
2018-01-12 00:47:55,519 INFO  [                    omero.tables.TablesI] (MainThread) /repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid doesn't exist
2018-01-12 00:47:55,520 WARNI [                    omero.tables.TablesI] (MainThread) Failed to find repo_svc: [Errno 2] No such file or directory: path(u'/repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid')
2018-01-12 00:47:55,520 ERROR [                       omero.util.Server] (MainThread) Failed initialization
Traceback (most recent call last):
  File "/opt/hudson/workspace/OMERO-DEV-merge-integration/src/dist/lib/python/omero/util/__init__.py", line 494, in run
    self.impl = self.impl_class(ctx)
  File "/opt/hudson/workspace/OMERO-DEV-merge-integration/src/dist/lib/python/omero/tables.py", line 410, in __init__
    raise e
IOError: [Errno 2] No such file or directory: path(u'/repositories/OMERO-DEV-merge-integration/.omero/repository/9654590d-37a1-4de9-976a-1d77dc8742f3/repo_uuid')
2018-01-12 00:47:55,520 INFO  [                       omero.util.Server] (MainThread) Cleanup
2018-01-12 00:47:55,520 INFO  [                       omero.util.Server] (MainThread) Stopped
2018-01-12 00:47:55,520 INFO  [                       omero.util.Server] (MainThread) ********************************************************************************
2018-01-12 00:47:55,521 INFO  [                    omero.tables.TablesI] (MainThread) Cleaning up

joshmoore · 2018-01-16T08:20:22Z

Note: this fix wasn't sufficient. See gh-5615

joshmoore force-pushed the tables-retry branch from ef2906d to 91c94e5 Compare December 6, 2017 08:13

joshmoore merged commit b58a6fc into ome:develop Dec 18, 2017

joshmoore deleted the tables-retry branch December 18, 2017 12:10

joshmoore mentioned this pull request Jan 12, 2018

tables: use omero.repo.wait rather than SECONDS_SLEEP #5615

Merged

joshmoore added this to the 5.4.2 milestone Jan 16, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tables: retry repo_uuid lookup#5593

tables: retry repo_uuid lookup#5593
joshmoore merged 1 commit intoome:developfrom
joshmoore:tables-retry

joshmoore commented Dec 6, 2017

Uh oh!

mtbc commented Dec 6, 2017

Uh oh!

joshmoore commented Dec 6, 2017

Uh oh!

joshmoore commented Dec 6, 2017

Uh oh!

dominikl commented Dec 18, 2017

Uh oh!

mtbc commented Jan 3, 2018 •

edited

Loading

Uh oh!

joshmoore commented Jan 5, 2018

Uh oh!

joshmoore commented Jan 12, 2018

Uh oh!

joshmoore commented Jan 16, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

joshmoore commented Dec 6, 2017

Testing this PR

Uh oh!

mtbc commented Dec 6, 2017

Uh oh!

joshmoore commented Dec 6, 2017

Uh oh!

joshmoore commented Dec 6, 2017

Uh oh!

dominikl commented Dec 18, 2017

Uh oh!

mtbc commented Jan 3, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

joshmoore commented Jan 5, 2018

Uh oh!

joshmoore commented Jan 12, 2018

Uh oh!

joshmoore commented Jan 16, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mtbc commented Jan 3, 2018 •

edited

Loading