Read MongoDB connection parameters from environment variables

[eecavanna]

On this branch, I made changes related to Docker, environment variables, and testing.

Main changes

• Environment variable-based configuration: I updated the server to use Pydantic's pydantic-settings package (documented here, and also used by microbiomedata/nmdc-server) to get the MongoDB connection parameters from environment variables. The precedence order (from highest to lowest) is: OS, .env file, class attribute defaults. Fixes Update FastAPI application obtain MongoDB credentials from environment (not hard-coded) #53.
• Example .env file: I included an example .env file named .env.example and instructions on creating one's own .env file. Here's an example .env file:

  MONGO_HOST=mongo
  MONGO_PORT=27017
  MONGO_DATABASE=bertron
  MONGO_USERNAME=admin
  MONGO_PASSWORD=root

• Name of test database: I added a pytest fixture that automatically sets the test database name in the context of all tests, overriding whatever the user might have in their .env file. The test database is always named bertron_test (we can make this, too, be read from an environment variable in a future PR). As a protective measure, tests will not run if the non-test database is also named bertron_test.
• New test: I added a test that involves accessing the MongoDB database. Previously, no tests did this. Fixes Add test that involves accessing the MongoDB database #75.

Tangential changes

• Fix bug involving silent failures: I fixed a bug in the GitHub Actions workflow, where the status code returned by the test container was not being propagated to the docker compose command, itself. With that bug in place, even if a test had failed, the GHA step would have succeeded. Now, GHA "sees" that status code, so the step fails when a test fails.
• Speed up container image builds: I created a .dockerignore file so Docker omits some unnecessary directories from the container image it builds. I did this while working on the GHA workflow (see next bullet point).
• Avoiding .venv/ collisions between host and container: I updated the GHA workflow to run the tests in the test container only, instead of also running them locally. That eliminated the need to handle collisions between the .venv/ directory created by the containers and the one the local uv program tries to create. This can be un-done, in case team members want the GHA workflow to also run the tests locally. Note: Previously, the "run tests locally" step was only running tests defined in the top-level tests/ directory, which was only the "hello world" test.
• Preventing redundant GHA workflow runs: Previously, whenever someone would add a commit to a PR, the CI GHA workflow would run twice—once, because a PR was being updated, and again, because a commit was being pushed to any branch. Now, whenever someone adds a commit to a PR, the CI GHA workflow will only run once.

[eecavanna]

This was not being used. Rather than update it to make it reference the newly-introduced environment variables (but still not be used), I opted to delete it.

[eecavanna]

Screenshot from before I made this change:

[eecavanna]

There is some overlap between this PR and #77; for example, both introduce a test involving accessing the MongoDB database.

[jeff-cohere]

I assume the -mongo here is a default value for the environment variable? If so, do you think we should support defaults, or put our efforts toward making it clear what environment variables everyone needs to have defined? Just asking the question so it's been asked. :-)

[eecavanna]

Thanks for reviewing this! I can confirm the first part immediately:

I assume the -mongo here is a default value for the environment variable?

Yes (reference). If—in the host environment—the MONGO_HOST environment variable is either not defined or its value is an empty string, then—within the container—that environment variable will be given the value mongo.

I'll think about the second part of your question now and reply after a break. I like the question. 👍

[eecavanna]

If so, do you think we should support defaults, or put our efforts toward making it clear what environment variables everyone needs to have defined?

Thanks for asking.

Here are my thoughts:

1. I do like the idea of there being only one layer at which the configuration parameters' values are defined. By "layer," I mean (a) the host environment, (b) an .env file, (c) the docker-compose.yml file, or (d) the Pydantic Settings class definition. I think limiting it to one layer would make it easier for us to trace down the source of misconfigurations.
2. I don't think I can declare a configuration parameter in the Pydantic Settings class definition without either giving it a default value there, or passing its value into the Settings() constructor via a kwarg. This is based upon the red squiggly lines VS Code is showing and what I am/am not seeing in the pydantic-settings documentation. I could set its type to Optional[str], for example, but—if I do—Pydantic won't warn me about its value not being a str.
3. I can update the docker-compose.yml file to require that the host environment (or .env file) have a given environment variable defined, instead of docker-compose.yml containing default values. I think this would be a step in the right direction, and I plan to make this change right after I post this comment. Here's how the syntax used in docker-compose.yml would change:

   - MONGO_HOST: ${MONGO_HOST:-mongo}
   + MONGO_HOST: ${MONGO_HOST:?}

   When an environment variable is missing and someone runs docker compose up, they'll see a message like this:

   $ docker compose up --detach
   error while interpolating services.app.environment.MONGO_HOST: required variable MONGO_HOST is missing a value

4. The end result of me making the change described in bullet point 3 is that the configuration parameters will get their values from one of three layers, instead of one of all four layers listed in bullet point 1; i.e. the docker-compose.yml file would be off the list.

[jeff-cohere]

Very nicely done and carefully documented, @eecavanna . I posed one question, but this looks good to me.