Ability to specify what .env file to use (instead of $CWD/.env)

[bafoed]

For example I have following file structure:

docker-configs/
|-- compose.base.yml

projects/
|-- project1/
|------ .env
|------ overrides.yml
|-- project2/
|------ .env
|------ overrides.yml

When my working directory is project1/, I can run compose with docker-compose -f ../../docker-configs/compose.base.yml -f overrides.yml to substitute variables in compose.base.yml using parameters from .env file. The situation changes when I want to automate some compose-related tasks, e.g. stop selected project. If I replace all relative paths with absolute ones, this will not be enough, because for some unknown reason docker-compose is able to read .env files only from the current working directory.

According to the documentation (https://docs.docker.com/compose/env-file/), this behavior can not be changed. Maybe it's worth to add some additional argument, such as --cli-env-file which could take full paths for purposes like mine? Or maybe there are other ways for this?

I wanted to send a pull request, but, unfortunately, there are a lot of mentions of various "environments" in the code and I could not figure out where cwd is hardcoded. I think this is a very simple feature that does not even cause any problems in backward compatibility - there is no need to change the default behavior, just add a new argument. Any hardcoded things get better when they can be changed.

This idea was already offered six months ago, but unfortunately it remained without attention: #3381 (comment).

And thank you very much for such a great work that has already been done.

[bafoed]

I just saw that in version 1.12 the argument --project-directory was added, which, theoretically, could solve this problem, but it does not do it. I do not know whether this can be considered a bug.

By the way, I can not use this argument in any way:
root@srv:/# docker-compose --project-directory /projects/project1 -f /docker-configs/compose.base.yml -f /projects/project1/overrides.yml config - undefined variables errors.

root@srv:/projects/project1# docker-compose --project-directory /projects/project1 -f /docker-configs/compose.base.yml -f overrides.yml config - ok (just as without --project-directory)

Created /projects/test/docker-compose.yml:

root@srv:/projects/test# cat docker-compose.yml
version: '2'

# do nothing

root@srv:/projects/test# docker-compose config
networks: {}
services: {}
version: '2.0'
volumes: {}

root@srv:/projects/test# cd /
root@srv:/# docker-compose --project-directory /projects/test config
ERROR: 
        Can't find a suitable configuration file in this directory or any
        parent. Are you in the right directory?

        Supported filenames: docker-compose.yml, docker-compose.yaml

[bafoed]

I tried to figure out what the --project-directory argument does at all, but did not fully understand it. But, at least, it became clear that it has nothing to do with the .env file.
So I want to say that the current description of this argument is a little misleading: on the documentation page there is the text "where the docker-compose command is executed (current working directory)", and in the description of the argument - "Specify an alternate working directory". However, they are not related.

It is very sad that the ability to change the working directory is made quite limited, so I decided to rebuild docker-compose for my needs. The patch that adds the functionality I need looks like this: https://gist.github.com/bafoed/b4fcd7acc8ab62f27120c65e1973c8cf/revisions#diff-5bc02cefb3ea9e27f1a6776eabd1935d
I'm not sure that I understood the meaning of the --project-directory argument correctly and it can be used for such purposes, so I send it as a gist, rather than pull request.
I hope someone more familiar with this code will give a comment.

[shin-]

Thank you for the report.

I agree that --project-directory should change where the .env file is retrieved. Feel free to submit a PR with your changes.

[loadx]

Hi All,

So I've had a quick look at the code for making project-directory flag work the way we agree it should.. in my opinion it should just override project_dir completly.

use cases:

1. command: docker-compose --project-directory /my_files
   Then .env is loaded from /my_files and so is docker-compose.yml.

2. command: docker-compose -f /compose_files/docker-compose.yaml --project-directory /my_files
   Then .env is loaded from /my_files along with the docker-compose.yaml. However it is over-ridden by the docker-compose.yaml loaded from /compose_files.

Do we agree this is the correct behavior? If so then I believe all we need to do is modify the project_dir to be over-written by the one specified in the flag.

https://github.com/docker/compose/blob/master/compose/cli/command.py#L24
@shin- Would you agree with this approach?

Cheers

[shin-]

Yes, there has been a couple PRs for it in #4709 and #4933 - I just haven't had time to test either thoroughly to make sure they don't break things in unexpected ways. It is on my radar though.

[loadx]

@shin- cheers for the heads up. I will have a look at the PR's and see If i can perhaps assist there. It looks like there hasn't been a comprehensive change to the tests to verify the various edge cases as mentioned above so I should be able to rectify that.

[schmunk42]

@estarter continued from #745 (comment)

It's very good that --project-directory is not affecting -f - otherwise imagine what a nightmare would be to use --project-directory and multiple docker-compose files all in different directories.

You would have to include the files with relative paths, that would be much more consistent than using project-directory only for the default files (docker-compose.yml, docker-compose.override.yml). When the option would use the .env from the specified directory you could specify your default compose files there file COMPOSE_FILE. If you want to include with absolute paths you can still do that.

@shin- I agree to this 100% #4841 (comment) - to my understanding --project-directory $DIR should be equivalent to:

pushd $DIR && docker-compose [...] && popd

[estarter]

You would have to include the files with relative paths, that would be much more consistent than using project-directory only for the default files (docker-compose.yml, docker-compose.override.yml).

I didn't mean to use --project-directory for the default files (docker-compose.yml, docker-compose.override.yml) actually, my only concern was .env file.

When the option would use the .env from the specified directory you could specify your default compose files there file COMPOSE_FILE. If you want to include with absolute paths you can still do that.

This is not quite a linux way. CLI tool that defines its own path convention is in practice difficult to use.
You propose to introduce a special path convention - when --project-directory is set than relative path is based on the parameter not current directory. That would not work with shell's path auto-complete feature, that would make relative path unusable for other commands (i can't re-use relative path with ls ), that makes reading of the path parameters of docker-compose a special thing.

@schmunk42 you described a perfect solution for the use-case when it's needed to change project directory:

pushd $DIR && docker-compose [...] && popd

I'm using this approach often working in shell for other commands, for example check on maven and its -f option.

---

What is missing for me is the option to define location of .env file.
Not sure if it'd be accepted to have a dedicated --env-file option?
Don't know how does that correlates with docker-compose.env plans?

@shin- WDYT?

[schmunk42]

@estarter Thank you for your valuable feedback.

You propose to introduce a special path convention - when --project-directory is set than relative path is based on the parameter not current directory.

Actually I meant relative paths should be relative from the file which uses them, but I also see that this raises other issues, ie. when merging files.

Before digging deeper into this topic, I have question @shin- ... Could you describe a bit more in detail what --project-directory is intended for? We looked through the sources and found ie. this:

:param working_dir: the directory to use for relative paths in the config

https://github.com/docker/compose/blob/master/compose/config/config.py#L151

Which paths are meant here, extends, env_file, volumes, ...? When playing around with the parameter we couldn't really figure out any effect at all?!

[zored]

Is there env-variable for --project-directory like $COMPOSE_FILE? Or pushd / popd is currently the solution? It would be great to change folders with fixed docker-compose behaviour. 😃

[schmunk42]

@zored The best I've found so far...

(cd project-directory && docker-compose up -d)

via #745 (comment)

[verglor]

Hello, I also think that adding --env-file option would be the best solution for specifying custom .env file without unnecessary (or even undesirable) change of project directory.

It would be even better if it is possible to specify multiple overriding env files (just like -f option does). Probably .env file from project directory should always be used and files specified in --env-file should just override it.