Skip to content

Conversation

@vdel
Copy link
Contributor

@vdel vdel commented Apr 10, 2019

Docker links duplicate many notions and are not necessary. In order to unify how we handle the various concepts in a deployment pipeline, let's get rid of them.

Migration indications:

For each link, create a service. Use the following mapping to update the service definition:

Before After
docker_links::image_name services::config::docker_image
docker_links::link_name services::service_name
docker_links::probe_ports services::config::probe_ports
docker_links::env services::config::env_override
docker_links::volumes services::config::volumes
docker_links::need_gpu services::config::need_gpu
docker_links::testing_options services::config::docker_opts
docker_links::env_exports services::needed_services::env_exports

I'm not sure the last one will work, why is it injected when calling needed_services ? Anyway, I haven't changed this behavior here.

@vdel vdel requested a review from thomas-riccardi April 10, 2019 17:41
@vdel vdel force-pushed the dev-deprecated-links branch 3 times, most recently from 6c1efea to 11f538d Compare April 11, 2019 06:41
@vdel vdel changed the base branch from master to dev-allow-external-image April 11, 2019 06:42
@vdel vdel force-pushed the dev-allow-external-image branch from 63853b1 to 5d84812 Compare April 11, 2019 08:27
@vdel vdel force-pushed the dev-deprecated-links branch 2 times, most recently from 21013c3 to 16ad687 Compare April 15, 2019 13:44
@thomas-riccardi thomas-riccardi changed the base branch from dev-allow-external-image to master April 15, 2019 13:58
@thomas-riccardi
Copy link
Contributor

/rebuild

# We link the service using its name, unless the service name has ':' in which case it is a variant
# and it is better to let the user chose the link name if needed
if self.service_name.find(':') < 0:
self.__fields__['link_name'] = self.service_name
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we keep this, we should document it in the field description

# We link the service using its name, unless the service name has ':' in which case it is a variant
# and it is better to let the user chose the link name if needed
if self.service_name.find(':') < 0:
self.__fields__['link_name'] = self.service_name
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it will possibly lead to silent link name conflict in case of multiple needed services on same service but with different env customization.
=> I suggest to not do it then.
=> maybe also add an explicit check on link name duplication somewhere?

fi
if [ ! -z "${PROBE_PORTS}" ]; then
ROOT=$(dirname $0)
>&2 docker run --rm --link ${CONTAINER_ID}:link_to_wait -v ${ROOT}/dmake_wait_for_it:/usr/bin/dmake_wait_for_it -v ${ROOT}/dmake_wait_for_it_wrapper:/usr/bin/dmake_wait_for_it_wrapper -i ubuntu dmake_wait_for_it_wrapper "link_to_wait" "${PROBE_PORTS}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: logs could be more informative with the real link name instead of the hardcoded link_to_wait, even if it requires passing it as an extra argument

set -e

CONTAINER_ID=${1}
PROBE_PORTS=${2}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: the {} are not necessary, and are unusual for positional arguments

return 'bash -c "%s"' % cmd

class DeployConfigSerializer(YAML2PipelineSerializer):
class DeployConfigSerializer(YAML2PipelineSerializer, ProbePortMixin):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This enables the auto port probing for all Services: this could break things.

For example test-gpu root image declare some ports, which are now auto waited-for when test-gpu is run ('dmake run test-gpu, or if test-gpuis ever added as aneeded_services`).

Copy link
Contributor

@thomas-riccardi thomas-riccardi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Additional side effect of using real services instead of docker_links: they now inherit the dmake.yml root env, whereas with the docker_links the environments where completely isolated.

Maybe add an option to ignore the root env per service? (env_override is not enough: it just adds additional environment variables).
The current workaround would be to move all docker_links to a separate dmake.yml file (and put it a new clear directory; which hinders readability).

@vdel vdel added lifecycle/PAUSED Indicate a PR paused: temporarily closed, will be re-opened at a later date and removed lifecycle/READY labels Nov 21, 2019
@vdel vdel closed this Nov 21, 2019
@thomas-riccardi
Copy link
Contributor

thomas-riccardi commented Mar 31, 2020

One more issue: docker_links host volumes are sometimes ignored (host persistence for developers, no persistence for tests/Jenkins):

dmake/dmake/deepobuild.py

Lines 413 to 417 in 8d3134b

if hasattr(common.options, 'with_docker_links_volumes_persistence'):
volume_persistence = common.options.with_docker_links_volumes_persistence
else:
# skip host vols in non shell (i.e. in test: we don't want persistence)
volume_persistence = (common.command == "shell")

This feature is missing from Services.

See #439

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lifecycle/PAUSED Indicate a PR paused: temporarily closed, will be re-opened at a later date

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants