Mount volume vs ADD and portability

[wernight]

Before being able to run a service there are often build steps. Example: Minimizing CSS/JS, compiling locales, aggregating static files... I'm considering websites with frequent docker-compose build && docker-compose up to push new changes.

There are two possibilities currently which fail short in some cases, even more with docker-compose:

1. Make a Dockerfile and use ADD to include your code, then do those operations during docker-compose build
   • Pro: Allows for example to give write access to www-data even if there is no such user/group in the host.
   • Pro: Doesn't pollute or even affect the host directory, which is usually really good during production.
   • Pro: Allow zero-downtime deployment (good for production).
2. Use a Docker image with writable mounted volumes, then do those operations during docker-compose run --rm ....
   • Pro: Changing a file locally and refreshing is usually enough, which is required during development!
   • Pro: Backing up modified data is easy as files are changed locally, which is good during development.

Using option (2) during development is good, and option (1) during production releases. However it's not just a flag to switch between both (unless you build an entire system). Which means that you would have to maintain two versions and break one or the other sooner or later (i.e., having something working if you run using (2) but breaking if you run via (1)).

Issues that could help:

• docker-compose should allow to tell which Docker running instance is used for a given service. This would allow to mix docker-compose and pure docker commands. To do for example: docker commit ... to that one could do option (2) (mount volumes, run some commands), and commit changes as a new image (that last step may be used only during production to fall to option (1)). So you may run all system tests and once everything's fine, commit as a new image and start that one on deployment ports. Also it'd simplify docker cp ... to retrieve data and make backups for example.
• docker-compose run and up should allow -v to mount during development only, allowing to setup option (1) but mount volumes only during development without changing the docker-compose.
• docker-compose.yml could have some for of conditional elements for example to have the ports assigned a bit differently during deployment than during development, or some folders not mounted...

[wernight]

Anyone here? This is my main problem overall with Docker and I tried forum and other places but I found no real answer or good discussion thread with actable outcomes.

[cpuguy83]

@wernight This is a short-coming of compose in general, IMO.

Being able to override settings (or add new ones) from the compose up CLI would be great.
Maybe docker-compose up -v <service>:<volume spec>

Or another possibility would be to allow the YAML file to have a concept of envs:

myservice:
    volumes:
        development:
            "~/mycode:/opt/myapp"

And have a env var DOCKER_COMPOSE_ENV....
Just thinking out loud.

[wernight]

I remember that one of the main feature in scope is supporting various environments nicely. Just didn't see a place where all news and discussions around this are gathered.

@cpuguy83 Having conditional elements in dev and release is good in some cases but for example if you had to have (conditional or not) ADD in Dockerfile for production and the same conditional mount for development, you'd end up having a risky system which may likely not work in production but works in development.

Case of the higher level

On a higher level if you could say what should be mounted where, docker-compose could either make a new image with those files in prod, or those mounted during development. This could be done so that path/files would be same in both environment and access rights as well.

If one can on top of that also run a few commands after mounting/add it'd allow a lot more, like doing some build.

That solution could work but sounds also pretty complex.

Case of the smart Dockerfile

In theory, given a Dockerfile that does ADD followed by other commands. docker-compose could auto-mount them during development, again managing the access rights. This could be a kind of docker-compose.yml mount option, a bit like volumes_from:.

[wernight]

Inheritance

Like SASS (or just object inheritance) docker-compose.yml could use inheritance which would allow small tweak without duplicating code. It would not exactly solve the ADD vs volumes: issue but it'd avoid other issues.

[wernight]

Seems ineritance is done in #1088

[dnephin]

I believe our changes in #2051 address this issue. What do you think?

[wernight]

The multiple files merge is interesting and probably better than the hierachy implementation.

It does not really solve the changes between volume .:/src:ro during development and ADD . /src in Dockerfile during production:

• Now it seems more and more production really requires an image with all the data, and mounting volumes only when really required for storage. I'm thinking about Kubernetes here a bit as well. This is also a good practice in general as it's good for deployment.
• Mounting volumes without ADD in Dockerfile is also crutial for fast development speed.

So #1088 doesn't really solve this issue.

[dnephin]

I think #2051 could solve the volumes vs ADD. If your base has ADD, but your override has a volume you get to use the volume in dev, but you use the added files in prod.

[thaJeztah]

@wernight does the solution @dnephin mentioned above solve your issue?

[wernight]

I guess it helps. It's not really a full solution but I guess there isn't one really, it's only possible in some cases to have some folders mounted during development. Probably close this ticket with #2051 as good enough solution.

Note that I'd suggest anyone to use COPY or ADD in Dockerfile always, and only mount some folders during development and not during CI.

[dnephin]

I agree, that's the way to do it