Allow env_files in docker-compose files to be optional

[deviantintegral]

Description

As a development leader, I would like teams to be able to spin up local docker-compose based environments without steps on the host beyond git clone.... As well, developers need to be able to optionally set or override environment variables such as connection strings or secrets. Since overriding variables is the 20% case, most developers should be able to run:

git clone <url>
docker-compose up -d

One of the most common patterns is to have a checked-in .env file of defaults, with .env.local or similar overriding the first file. Ideally, this could be achieved with:

service:
  web:
    env_file:
      - .env
      - .env.local

However, docker-compose requires .env.local to exist, and errors out. This is especially problematic with VS Code's "clone in named volume" functionality (key for working around Docker performance issues on macOS and Windows) in the remote containers extension, as there's no opportunity to run a script to create even a stub file before containers are started.

Additional information you deem important (e.g. issue happens only occasionally):

There was a previous declined PR at #3955 from 2017, along with a related issue at #3560. However, the PR has many comments since closing detailing common use cases and hacks around this missing functionality, as well as comparison to existing tools that do treat .env files as optional. I'm opening this issue for current feedback from current maintainers, as it seems that the original closer no longer works at Docker. I've likely missed suggestions and edge cases around this feature expressed in the PR comments, so please don't take the above as a complete description of community needs.

Open questions

• Is there planning underway or other issues which would actually solve this use case, but in a non-obvious way? I ask because there was a similar set of issues around making individual services optional that were eventually solved by profiles. I know that wasn't obvious to me until the profiles feature actually landed, so I have some hope there's something in compose v2 to address this.
• Otherwise, can we get an update from maintainers on current thinking around this issue? Even if that's to close this as "won't fix", given 5 years has passed that would still be a valuable signal that users should continue to find workarounds or alternate tools / methodologies to support this workflow.

[ndeloof]

Seems to me there's a confusion here between .env file used to pass variables to compose, and env_file used to set container environment, which is just the exact same as docker run --env-file. Making the later optional would trigger a bunch of new issues and confusion.

The recommended way to support "optionally set or override environment variables" is to declare those as service environment without a value, so that they get interpolated from environment variables and/or local .env file

service:
  web:
     - SOME_VAR

if SOME_VAR is not declared, container will be created without such a variable set. Otherwise current value will be set.

[ndeloof]

Assuming we want to add support for "optional env_file" as you describe, this should be debated under Compose specification, and probably will better be addressed with a custom syntax to mark file as optional, maybe using elvis operator ?:

service:
  web:
    env_file:
      - .env
      - ?:.env.local # optional file

[Kallys]

I'm interessed by this feature for same motivations (working docker compose up just after a git clone) as it may resolve my concerns about default env values.

Illustration

Let's look at this code for example (inspired from this one).

# docker-compose.yml
services:
  php:
    environment:
      DATABASE_URL: postgresql://${POSTGRES_USER:-api-platform}:${POSTGRES_PASSWORD:-!ChangeMe!}@database:5432/${POSTGRES_DB:-api}?serverVersion=${POSTGRES_VERSION:-13}

  database:
    image: postgres:${POSTGRES_VERSION:-13}-alpine
    environment:
      - POSTGRES_DB=${POSTGRES_DB:-api}
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-!ChangeMe!}
      - POSTGRES_USER=${POSTGRES_USER:-api-platform}

If someone modify POSTGRES_DB default value in "database" service forgetting doing the same in "php" service, it could lead to some difficulties...

Since there is no ability to define global constants in compose files and yaml anchors variables are not expanded, I tried using .env file as a way to both expose and provide environment default values.

# docker-compose.yml
services:
  php:
    environment:
      DATABASE_URL: postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@database:5432/${POSTGRES_DB}?serverVersion=${POSTGRES_VERSION}

  database:
    image: postgres:${POSTGRES_VERSION}-alpine
    environment:
      - POSTGRES_DB=${POSTGRES_DB}
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
      - POSTGRES_USER=${POSTGRES_USER}

# .env
POSTGRES_DB=api-platform
POSTGRES_PASSWORD=!ChangeMe!
POSTGRES_VERSION=13
POSTGRES_USER=api-platform

This solution works great ! But I came out to the same issue when I tried to commit this .env file and wanted to allow team members to locally override those values...

[deviantintegral]

Thanks @ndeloof , this is helpful. The above example really summarizes this issue with:

But I came out to the same issue when I tried to commit this .env file and wanted to allow team members to locally override those values...

Also:

declare those as service environment without a value

So it seems like this is somewhat doable as long as defaults aren't provided. That seems to break the 80% case of working "out of the clone" without additional configuration.

I think your proposal with the ? operator would work nicely.

[q0rban]

What about allowing glob patterns such as .env(.local|)

service:
  web:
    env_file:
      - '.env(.local|)'

[ndeloof]

Please, do not try to include some pseudo-programing langage / expression in compose file format :P

Hi I really want to have this option.

env_file:
  - path: ''./web/.env.development.local'
    required: false
  - path: ''./web/.env.local'
    required: true

How about this format. The advantage of this is capable of many env_files while some is not required and others are required. It happens many times when you have Unit Test CI while you also have E2E Test CI.

If maintainer agrees with it I could create PR.

[ndeloof]

This should first be debated on compose-specification

@ndeloof

yes, i saw you said that in previous comment as well, but no issue is submitted in compose-specification yet. so i wondered if there is still a problem.

you said 'this should be debated in compose-specification' but does it mean 'please post a issue in compose-specification' or 'please wait' ?

i'm just confused.

[ndeloof]

@ulwlu compose spec is public, anyone can open a proposal. If you feel this is un important topic, you're welcome to do

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.