Defined new group for deploying the searchengine in the pilot IDRs

[sbesson]

Related to #359, the next step discussed at the weekly IDR meeting is to deploy the searchengine stack (Elasticsearch, searchengine app, searchengineclient) in a pilot environment.

So far all the deployment has been happening through the management VM which contains the monitoring stack. This was always understood to be a staging location allowing ourselves to assess the prototype. As we are looking into moving the searchengine API into production, a few requirements need to be considered immediately:

• the searchengine should be deployable into pilots i.e. as part of the validation workflow
• the indexing/caching step needs additional memory capabilities to run in a performant manner
• in a production environment, the searchengine endpoint will also need to be available internally at the level of each OMERO deployment (as opposed to only at the proxy level currently).

Deploying the search engine in a single-node pilot context will allow us to evaluate this production deployment.

This PR

• defines two new extra groups "{{ idr_parent_environment }}-searchengine-hosts" and searchengine-hosts in the metadata of the individual pilot OMERO VMs.
• adds the pilot OMERO VMs to the dockerworker group
• updates the idr-docker.yml playbook installing Docker to run on all docker-hosts i.e. both VMs created with idr_vm_dockerworker: True and idr_vm_dockermanager: True

Once deployed in a test pilot instance, the playbook in #359 can be adjusted to run on this group rather than "{{ idr_parent_environment }}-management-hosts" and also increase the number of rows that can be indexed. /cc @khaledk2

[sbesson]

With the inclusion of #359, I think this is now ready for review, together with the companion role PR. So far a pilot VM has been deployed with the correct group, allowing for the playbooks introduced in #359 to deploy the associated services.

The proposed next step would be to adjust https://github.com/IDR/deployment/blob/master/ansible/openstack-create-publicidr.yml with similar modifications to allow the upcoming prod107 deployment to have all the prerequisites (docker, group) allowing to deploy the searchengine stack

[joshmoore]

Other than not understanding the driver for the extra non-IDR-namespaced searchengine-hosts group, this makes sense to me. Also, 👍 for "dockerworker" to go along with the "manager".

[sbesson]

Thanks for raising this as I also stumbled upon this duality in group names - see https://github.com/IDR/ansible-role-openstack-idr-instance/blob/afe071917bb29b37950cfac1ef6831edce98bae1/defaults/main.yml#L77-L123. My current understanding is that:

• the prefixed groups e.g. "{{ idr_environment }}-omeroreadwrite-hosts" allow to limit the playbooks execution to a given deployment e.g. prod106
• the non-prefixed groups e.g. omeroreadwrite-hosts allow to manage group variables across deployments under https://github.com/IDR/deployment/tree/master/ansible/group_vars

Capturing another thought on the dockerworker fix, a secondary benefit of aligning the behavior with dockermanager is that the Docker installaton in OMERO servers might be useful when coming back to #326.

[joshmoore]

MSTM

[sbesson]

Thank you. I'll wait until IDR/ansible-role-openstack-idr-instance#8 is reviewed and approved. I'll tag a new patch release of the role and bump in this PR before merging.