Extend an entire Compose file

[bfirsh]

It is a common pattern to define a base Compose file, and then define Compose files for different environments with a few small changes.

It is currently possible to extend single services, but it is very verbose to include a large number of services and make a small change to one of them (for example, setting RAILS_ENV=production).

It should be possible to extend a Compose file with a complete set of services from another Compose file. All of those services will be copied into the Compose file, as if you were extending each of the services individually:

• If you don't define a service in the child file, it is copied as-is.
• If you do define a service, it should behave as if you've extended that single service.
• There is no way of undefining a service (yet)

This is an intentionally simple first step, and I am intentionally not defining a syntax so we can discuss.

Design questions:

• This is the first top-level configuration we have added to Compose. How should we do this?

Related issues / suggested designs: #318 #1380 dcao-merge

(This is part of an initiative to define an app once in a way that can be used across dev, test and prod.)

[andrerom]

+100

#1 BUT, I guess you imply this but should be mentioned: it should keep the volume and link setup so you don't need to re define that.

#2 because of #1 it will have different semantics then extend. And personally I think it should allow composition, so that means "import" semantics. Example import application 1 and 2 and make them talk together by means of link/own-container..

[aanm]

@bfirsh Let's assume a dev has a Compose file comp-dev.yml and op has a different one comp-op.yml, will be possible to run compose has:
docker-compose --extends op-comp.yml -f comp-dev.yml up?

[dnephin]

If one config depends on another, I would expect that dependency to be declared as part of the contents of the file, instead of a command line argument.

[aanm]

It doesn't, it belongs to the operator. This way, the operator only needs to have its own compose file for its own infrastructure. Thus, for 10000 developers with 10000 different compose files, the single operator could easily override all options the he intended on any developer's compose file as:

for file in *-dev.yml; do
 docker-compose --extends op-comp.yml -f $file up -d
done

[aanand]

After some discussion, @dnephin and I came up with the following:

1. As @aanm suggests, extending at the command-line is more flexible than doing it in the file, because you can re-use the same overrides with multiple bases. It also means we don't have to do any disruptive new design on the file format.

   If you could pass the -f FILE flag multiple times, that would allow you to apply arbitrarily many overrides:

   $ docker-compose -f docker-compose.yml -f docker-compose.overrides.yml up
   

2. However, it's really verbose. If we want the idiomatic setup to be a docker-compose.yml with just the base configuration, plus a dev-specific file with overrides, that's a lot of typing, and users are going to have a terrible time.

   So there should be a sensible default: if docker-compose.overrides.yml exists, we apply it as if the user had typed the full command above; if not, we behave as we currently do (i.e. as if they'd just typed docker-compose -f docker-compose.yml up).

3. In non-development environments, whatever spins up docker-compose should explicitly pass the set of files in:

   # don't apply any overrides
   $ docker-compose -f docker-compose.yml up
   
   # apply production overrides
   $ docker-compose -f docker-compose.yml -f docker-compose.production.yml up
   

This caters to the use case of wanting to configure an app for multiple environments: put the core stuff in docker-compose.yml, the development overrides in docker-compose.overrides.yml, and any other environment-specific overrides in other files (e.g. docker-compose.production.yml).

It also caters to the use case of wanting to distribute an app's code with sensible defaults, but allowing people to override it when running it locally (for which one solution has already been proposed in #1999):

• put the defaults in docker-compose.yml
• add docker-compose.overrides.yml to .gitignore
• users can create docker-compose.overrides.yml and override stuff locally if they want to

One limitation is that it can't currently serve both use cases at once.

[shanielh]

I agree with @dnephin ,But I guess that both of the solutions are good.

The import inside the file is better for me because it will just keep my configuration more DRY and will not force me change my current deployment process.