Make env_file optional

[akalipetis]

This makes sure:

• the actual file is not necessary, meaning that you don't have to create it each time you clone (given that usually .env files are ignored)
• you still get a warning if the file doesn't exist, so you can be aware of times where a file should have been loaded but eventually didn't, for example due to a typo

Fix #3560

Signed-off-by: Antonis Kalipetis akalipetis@gmail.com

[akalipetis]

I have also updated the failing test and rebased with master.

[akalipetis]

Hey, is this something we'd like to see in the codebase?
If not, I could just close it 😄

Happy new year!

[akalipetis]

Closing this one, since it didn't get much attention. Feel free to ping me to reopen it and rebase it if we believe it's a good fit 😃

[mariocesar]

This is useful, what it needs to be merged?

[atrauzzi]

Whoa whoa, would love to see this.

[lukad]

What is preventing this from getting merged?

[DracoBlue]

👍 please add this!

[akalipetis]

Reopening since it was requested by more people.

/cc @shin- since the discussion over at #3560 is kind of stalled but there are still people wanting this, is there anything I can do to either merge or decide against this?

[dariusgm]

Are there any new updates on this MR?

[derks]

Agree, this would be a nice feature to have... in our use-case, docker-compose.yml is built for local development to work out-of-the-box, but for different types of ci-testing (i.e. RAILS_ENV=test vs RAILS_ENV=development) we need to change/override the environment.

Could be made backward compatible if necessary so that both of the following would work:

Default/Required:

env_file: docker-compose.env

Backward/Forward Compatible:

env_file:
    path: docker-compose.env
    required: false

The above is similar to how the build directive is handled where the default accepts just a string/path:

build: .

But also accepts a nested dict:

build:
    context: .
    args:
        SOME_BUILD_ARG=1

Etc...

[lucasdf]

I would love to see this accepted.

[esco]

@akalipetis can this be merged?

[alanjds]

@dnephin
(hope you dont mind my spam here ;)

[nilbus]

This would fit my needs as implemented.

I'm of the opinion that we break backward-compatibility here, perhaps incrementing the docker-compose version number. The new behavior is the one I believe most people will want, so we shouldn't require that more verbose syntax (nested required: false) most of the time. In cases where ENV vars are required and cannot have defaults, applications can and often will fail meaningfully in their absence.

I defer to the compose team's opinion though. I'm not sure what this behavior change would require for docker-compose format versioning.

[Ilyes512]

I would really like to have this as well! +1!!!

Also: This is almost open for a year? Disappointed it's not merge earlier. Creating empty .env files just to satisfy the check is not acceptable.

[akalipetis]

If @shin- @dnephin or another maintainer is happy with either this (optional by default) or the proposed (dual definition, with either value or dict) implementation above, I'd be happy to rebase/update the PR and get it ready to be merged 😄

[nilbus]

Though unlikely, a missing file isn't the only reason a ConfigurationError could be thrown. When the path exists but is not a file (e.g. a directory), a ConfigurationError will also be thrown. In that case, a blocking error seems more appropriate than a passive warning.

Instead of a try/except, perhaps we should check file existence directly.

if os.path.exists(env_file):
  env.update(env_vars_from_file(env_file))
else:
  log.warn('env_file {} does not exist - ignoring'.format(env_file))

[rosskarchner]

If @akalipetis makes this change, would this be merge-able?

[AlJohri]

EDIT: removed. sorry missed to read the thread.

[nilbus]

Goal: Allow new contributors† to get set up by simply running docker-compose up, without required extra steps (that vary by platform) like touch .env.

Maintainers, do you agree that this is a worthy goal, in line with the project's philosophy?
Please confirm whether you think additional effort here is worthwhile.

If so, we might also consider also:

• combining the implementation and test commits, as per the FAQ for contributors
• updating the relevant documentation to explicitly document the new behavior

The most significant consideration for this PR may be the backward-compatibility issue. These are the options I see:

1. Change the behavior without a docker-compose.yml API version change, considering this to be a compatible change. Processes already handle missing ENV vars by providing defaults, erroring, or behaving incorrectly. Docker Compose needs not add extra protection against missing ENV vars, and it this behavior was never explicitly intended. The documentation does not specify the behavior when the env_file does not exist.

   This runs the (likely small) risk of offending users who rely on the assumption that docker-compose will never start containers with missing ENV vars, because they believe users will always create the env_file correctly, with all required variables. Alternatively, they may rely on the missing env_file error as a reminder to create the env_file with the required variables.

2. Change the behavior, and bump the Compose file version. Provide different implementations based on the version. Bumping the API version to 2.x and 3.x will encourage users to acknowledge this change.

3. Use the more verbose but definitely backward compatible syntax suggested by @derks:

   env_file:
       path: docker-compose.env
       required: false

Maintainers, you know the project needs best. Which option seems the most beneficial to you? We're willing to carry out the work but need a decision on the direction.

Thanks! 😄

†to projects that use docker-compose

[DracoBlue]

👍

[AlexSkrypnyk]

@KlaasH string interpolations would work, except for the cases when images use scripts that check if the variable is set rather than if it is empty. So, with AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID:-}, the AWS_ACCESS_KEY_ID variable will be set. If those images are 3rd party - there is no way to unset these empty variables. The only way would be to comment-out variable in docker-compose.yml.

As for using docker-compose.override.yml - environment variables may be set for all containers as a YAML reference like so:

x-environment:
  &default-environment
  XDEBUG_ENABLE: "${XDEBUG_ENABLE:-}"

services:
  cli:
    environment:
      <<: *default-environment
  php:
    environment:
      <<: *default-environment

It is not possible to extend the YAML reference section in docker-compose.override.yml.

[jlivermont]

Completely agree with @DarthNato

In our case we have frontend developers who connect to the dev database, and backend developers who connect to their local db. We use .env files to override the default connection settings. It's silly that one group currently has to create a blank .env file just to avoid breaking compose, since in our case, it truly is optional and not a required supporting file depending on your environment.

If anything, requiring an addendum to our README that you'll need to make a blank .env file in some cases makes the compose file less portable, not more.

At the very least can we support the more verbose syntax of

env_file:
    path: docker-compose.env
    required: false

That way you have to explicitly state it's an optional file which should mitigate many of the concerns expressed in rejecting this ticket.

Like others have stated, running a bunch of touch commands to create env files is cumbersome and annoying.

Can we reconsider @ytilis's suggestion --^ from Nov 8, 2017 to add a required attribute that marks an env files as required or optional? I think that that behavior should work for everyone. By default, missing env files cause docker-compose to error, but if marked as optional, a warning is printed and everything continues.

[tomasfejfar]

One important aspect of this issue is that it already behaves this way to some extent.

dev:
  build: .

This loads .env file if it exists. If it does not exist it does nothing.

dev:
  build: .
  envfile:
    - .env

This loads .env if exists, blows up if it does not...

[nhooey]

Currently there is no way to override environment variables with an optional environment variable file, unless it's explicitly created empty, or every environment variable is explicitly put in the docker-compose.yaml file, options both of which should be considered too tedious and unnecessary than providing a required option.

@ytilis suggests adding an option for env_file directives that let the user specify that the file is not required, so a .env file ignored in version control that doesn't exist will not break Docker Compose before a user creates it.

@ytilis' suggestion:

At the very least can we support the more verbose syntax of

env_file:
    path: docker-compose.env
    required: false

That way you have to explicitly state it's an optional file which should mitigate many of the concerns expressed in rejecting this ticket.

(Note that env_file should actually be a list of hashes, instead of just a single hash as stated in the example. I'm sure this is what @ytilis meant anyway.)

Could we get agreement from the maintainers that this approach will be approved, so someone can create a merge request?

@ulyssessouza: I'm tagging you since @shin-, @dnephin, and @aanand have all seem to quit the project entirely.

@jlivermont, @anujbiyani: You two may also be interested in this reply.

[saivan]

I have to agree with @tomasfejfar, this is inconsistent. Sometimes we have good reasons not to provide particular env files to particular users. Eg, an env file with development and production variables.

I'm fine with the required parameter, but not having this functionality at all is obviously a roadblock for a lot of users. So it would be great to see it added.

[umlaeute]

my ugly hack for now is to have an empty file in my repository (aptly named .empty), that is read as the env_file unless an ENV_FILE environment variable is set in the .env file.

version: '3'

services:
  test:
    image: alpine
    env_file:
      - ${ENV_FILE:-.empty}

using "no" env_file:

$ grep . .empty .env
grep: .env: No such file or directory
$ docker-compose run test sh -c set  | grep MYVAR
$

using .env as env_file:

$ grep . .empty .env
.env:ENV_FILE=.env
.env:MYVAR0=foo
.env:MYVAR1=bar
$ docker-compose run test sh -c set  | grep MYVAR
MYVAR0='foo'
MYVAR1='bar'
$

using some other env_file:

$ grep . .empty .env prod.env
.env:ENV_FILE=prod.env
prod.env:MYVAR0=hello
prod.env:MYVAR1=goodbye
$ docker-compose run test sh -c set  | grep MYVAR
MYVAR0='hello'
MYVAR1='goodbye'

it's hacky at best, but serves the purpose of having only a single simple file containing the environment for a container and which can be present or not (and be git-excluded)

[leonaves]

@ulyssessouza tagging again as this is a no-brainer in my eyes. If people are able to hack and workaround to achieve the same end then it might as well be explicitly supported. Doing it to protect consumers of these files only leads to put them at the mercy of workarounds. The people who want to setup their projects this way will find a way to do it supported or otherwise.

[tomasbedrich]

If your file is named exactly .env, you can just delete env_file directive, list all your variables in environment section (without values) and it works. 🎉 source

Example:

version: '3'
services:
  foo:
    image: foo
    build: .
    environment:
      - FLASK_RUN_PORT
      - FLASK_ENV
#    env_file: .env # <-- delete this

[bo5o]

What's the status on this? I have a list of env_files where some of them are optional. Right now, docker-compose complains that they are missing and throws an error. Current workaround is to manually touch missing files before.
It would help a lot if there was an option to output only a warning on missing env_file instead of an error.

[JonHolman]

Agreed, I encountered this today as well and I did the same thing touch the missing files before running compose. Seems ridiculous.

[remyduthu]

I am working on a "monorepo" with a single docker-compose.yaml file at the root of the projet. docker-compose raises an error because I don't have env_file for each service. It would be perfect if env_file could be optional (at least for services that are not concerned by docker-compose command)...

[leonaves]

@ulyssessouza Please take a look at this, and specifically @umlaeute's comment, this is possible with weird hacks already, so why not support it an official way, otherwise people are going to become dependent on methods not recommended by the official team that may end up breaking.

[mewtlu]

Appreciate tomasbedrich's workaround to this problem but I wholeheartedly agree with leonaves and others, I don't understand why a workaround/hack which dirties either git or the compose config would be seen as a better solution than simply supporting this?

[mmx86]

In dev environment for each service our team wants to have:
.default.env (default env settings)
.custom.env (custom overrides)

We want developer to:

1. git clone repository
2. docker-compose up
3. work

We don't want developer to:

1. Copy env files from example env files for each service.
2. Do anything else to be able start development.

We could just push into repository empty .custom.env files, but git doesn't allow to ignore created files. And custom overrides MUST be ignored by git.
One more option is to force every developer to run git update-index --assume-unchanged on these files. It is also unacceptable. And we don't want to deal with hooks.

It would be a life saver to have

    env_file:
      - path: service538/.default.env
        required: true
      - path: service538/.custom.env
        required: false

[bohdanyurov-gl]

So far every single tool or service I've worked with simply assumed all env files to be optional by default. I mean, if you need to have them as required they should be some config files or required parameters. I've got extremely confused noticing that docker-compose has this custom behavior that puts hundreds of people into a dirty-hack area.

[AlwinGarside]

@shin-

The same way that docker run --env-file doesnotexist ... results in an error, I don't think absent environment files should be ignored by Compose. I realize this PR proposes replacing the error with a warning, but those are too often ignored or misunderstood by users to have the desired impact.

Moreover, this would make Compose files as a whole less portable by encouraging people to share them without some of their supporting files, which feels like the wrong direction to take for our project.

Given these considerations, I'm sorry to say I have to refuse merging this PR. I appreciate the work @akalipetis put into it and the people who have reviewed and proposed changes, but the cons outweigh the pros here unfortunately.

Sorry for necrobumping, but I find the argument that this would make Compose files less portable is conjecture at best. In practice, the lack of an optional include path actually limits the portability of our project. Let me elaborate:

At work we maintain a fairly large (I think) docker-compose.yml with about 10 services. This docker-compose.yml (and related .env files) is used for all environments along our DTAP flow from development all the way up to production.

With a combination of extends and environment variable interpolation we've been able to make it so it can be easily configured to a certain environment. The development environment configuration works out of the box with sensible defaults. However, sometimes for certain external services alternate configuration need to be configured. Of course, this can be done by editing the original env files's or docker-compose.yml, however this wil leave local changes in git.

There is no elegant, fool-proof, convenient way to easily override environment values in files that are ignored in git by default. The only way to do this somewhat seamlessly is by using docker-compose.override.yml which is, in fact, optional... 🙃
However, docker-compose.override.yml is only optional as long as you don't define COMPOSE_FILE in your .env, which is something I do to easily include multiple docker-compose files without needing to use a handful of -f options. Not to mention overriding a compose file requires knowledge about the compose spec, but we have all kinds of people running our environment from QA to junior frontend devs.

As you can see, I'm trying to make my compose setup as portable as possible for other developers, and the lack of an optional env file either requires me to limit my options, jump through strange hoops or, ultimately, resort to managing files outside of my docker-compose setup. That is the opposite of compatibility.

[LiquidLemon]

I really wish this got merged. We often use .env-local files in conjunction with the usual .env to allow overriding the default configuration in our projects. Having to add a line of touch .env-local in every CI job which builds images using docker-compose is not ideal.

[chrisdlangton]

The same way that docker run --env-file doesnotexist ... results in an error, I don't think absent environment files should be ignored by Compose. I realize this PR proposes replacing the error with a warning, but those are too often ignored or misunderstood by users to have the desired impact.

Moreover, this would make Compose files as a whole less portable by encouraging people to share them without some of their supporting files, which feels like the wrong direction to take for our project.

Given these considerations, I'm sorry to say I have to refuse merging this PR. I appreciate the work @akalipetis put into it and the people who have reviewed and proposed changes, but the cons outweigh the pros here unfortunately.

This issue was closed with a comment that attempts to give a rationale, and that comment has an overwhelming response of thumbs down.

The comment gives a rationale that is not relevant to the OP issue.

The OP issue specifies that they have the file specified by env_file as an ignored file. i.e. a file called .env is the defacto file used for local development and it is mostly always ignored by git. It will not exist purposefully in non-local environments (like a CI environment) because that would not be secure but mosst importantly the git repo says this file can not exist due to the ignore rule.

The comment is completely ignorant to the OP specific requirement and issue, therefore the comment is not on topic and not a rational reason to close the issue.

[luxalpa]

The same way that docker run --env-file doesnotexist ... results in an error, I don't think absent environment files should be ignored by Compose. I realize this PR proposes replacing the error with a warning, but those are too often ignored or misunderstood by users to have the desired impact.

Moreover, this would make Compose files as a whole less portable by encouraging people to share them without some of their supporting files, which feels like the wrong direction to take for our project.

Given these considerations, I'm sorry to say I have to refuse merging this PR. I appreciate the work @akalipetis put into it and the people who have reviewed and proposed changes, but the cons outweigh the pros here unfortunately.

A "rationale" can not be based on feelings. This is an irrationale.

Are you telling your users that their problems won't be resolved in this project and that we should use a different project? Which one do you recommend? Or are you asking us to fork this? This is confusing.

[deviantintegral]

I'm normally loathe to create new issues when clearly there's active discussion elsewhere, but given there's been no updates from maintainers that I see since 2017 I've opened a new issue. Hopefully that gets on a maintainer's radar with either suggestions on alternatives, what would be required to see something like this merged, or an updated "won't fix" response.

#9181

[jarl-dk]

Why not just add a new feature?
Along with existing (required) env_file:, introduce optional_env_file:. Like

services:
  rails-app:
    env_file:
      - .env.development
    optional_env_file:
      - .env.development.local

[ledlamp]

by encouraging people to share them without some of their supporting files

Bruh I aint sharing my env file of secret keys 😤

[LudovicTOURMAN]

As a workaround, I'm using that syntax (pointing by default on /dev/null file):

  my-service:
    image: my-service:1.2.3
    env_file:
      - ${MY_SERVICE_ENV_FILE:-/dev/null}

You can then either use: docker compose up
or MY_SERVICE_ENV_FILE=my-service.env docker compose up

[ericbf]

Anyone still coming here (like I did) for this feature, it is implemented now, as of Docker Compose v2.24.0.

See this comment by BuriedStPatrick:

Came across this issue while googling for a solution for my highly specific use case. To anyone else coming across this issue, this IS actually implemented and no longer requires any hacks as of Docker Compose v2.24.0:

version: 3.9
services:
  my-service:
    env_file:
      - path: /path/to/optional/.env
        required: false # Makes the file optional. Default value is 'true'

Docs: https://docs.docker.com/compose/environment-variables/set-environment-variables/#additional-information