Add support for `extends` feature in Compose v3 / docker stack deploy

[shin-]

As can be seen in docker/compose#4315 , the extends feature that exists in docker-compose seems to be popular among users despite its flaws. However, it has so far not been added in the Engine's implementation of the Compose format. So far, we have advised users to simply flatten their Compose file structure when using v3, but is this the long-term solution we want to go with? How can we provide a clear upgrade path for users who have come to rely on this feature?

cc @dnephin @vdemeester

[shouze]

I've added some notes docker/compose#4315 (comment) to me bringing back extends as it exists up to docker compose file 2.1 version is not a good idea but the main feature I miss is to be able to declare abstract services (that should never be runned but could be used to group common properties of services for convenience).

[jakajancar]

Agreed on comments from docker/compose#4315 that the way extends worked was a bit spartan.

I won't recommend a solution, but FWIW, to show the extent of abuse, here are the conventions that adorn the top of our compose file:

#
# Docker Compose configuration
#
# Due to lack of "expressivity" in Compose, we define our own couple of service
# "pseudo-types":
#
#   - image-only services (name: *-image)
#
#     The only goal of these is to build images. No other services build images.
#
#     These have entrypoint overridden to exit immediately.
#
#   - base services (name: *-base)
#
#     These contain common configuration and are intended to be extended.
#
#     Their command (not entrypoint, to keep the original one) is overridden to
#     exit immediately. Service must support a command to exit immediately.
#
#   - task services (name: *-task)
#
#     These are intended for running one-off commands.
#
#     Their default command is overridden to exit immediately. Service must
#     support a command to exit immediately.
#
#   - "real" services
#
#     These are actual services that stay up and running.
#
version: '2'
services:
  ...

[nicodmf]

I think the extends as in v2.1 is a good choice. Extends is actually simple and understandable, this is too a good pratice for each environnement to have a little and readable transformation between dev, prod and env.

Actually, i have

• a common docker-compose file which present the base rules
• a dev docker compose extending it and with facilities for developper
• a staging docker compose extending common too and with facilities for this environnement
• a production docker compose extending common too and with facilities for this environnement, specially container replication, rules about restarting, etc...

This works, and i don't understand why we should search in this ticket a complete rewrite, but more the keep of an interessing feature. Extends is a good part with special use cases that other techniques can't resolv easily. I'm happy with extends possibilities.

[sambernet]

Blocking us from upgrading to v3.x format as well.

We keep our docker container definitions in a folder-per-instance layout, where each folder contains a docker-compose.yml that defines the env specifics for the container instance at hand. To factor out the common stuff (DRY) we use base service definitions in a parent folder and use extend.

So when I need to manage a particular container instance, I just need to cd into the right folder and can then directly run docker-compose commands without further configuration (no -f flags required that team members need to lookup or know, just works as expected out-of-the-box).

[SC7639]

I'm blocked from using version 3 compose files.

I use services.yml to define the base layout of my services and then extend that with dev.services.yml for my development evironment (mostly adding volumes) but I like the (DRY) re usability of extends when it was added. It's not a deal breaker but it would keep me from moving to version 3 unless there is a must have feature.

[sambernet]

I'm not against improving the v2.1 'extends' solution with something more appropriate though. Something like abstract services could be both more safe to use and more powerful.

For example (since it's abstract) I could imagine supporting abstract volumes/mountpoints/networks, where the abstract base service defines a local mount point or a required network for a service, without defining the host part - meaning a deriving service could then define/map the host part of the mounts/volumes and networks to whats appropriate in his host/stage environment.

That's an example of something we can't do now with 'extends' as far as I know, and would open some interesting possibilities.

[alwaysastudent]

Taking away extends feature was not helpful at all. We have a lot of web services started with the same volumes mapped, environment variables and labels set. They also have same healthcheck policies. And Not to mention ports. Combining compose files using multiple -f option is tedious and error prone if you are dealing with multiple projects in the same host.

[alwaysastudent]

@shin- will this be brought back on 3.2 schema ?

[efrecon]

I really, really would love to get extends back into the file format. I have been using it to further refine abstract services for a number of projects. I understand that multiple -f options fits almost the bill, but it doesn't always work. For example, I rather often change the name of the abstract service into something that is more meaningful in the context of the project at hand. This is something that the overriding that occurs with multiple -f options does not support.

I would not mind loosing the exact extends though, as long as their is some other way to "instantiate" an abstract service in a file.

[jazzfog]

I have a setup with a common service file and bunch of services extending it, changing mostly volumes, so I would say I rely on extends feature and do not see other good way to describe my setup.

--frustrated

[rationull]

+1
I can see the logic behind the recommendation to flatten docker-compose files but I don't like the idea of duplicating the code that defines our development service topology across multiple projects. We'll use the -f workaround for now, with shell scripts so that developers don't have to remember which services to include in which cases.

I'm hoping a satisfactory solution can be found here to enable better factoring of compose structures!