Fidesops requires a minimal config TOML file to start and several config variables are missing safe defaults

[NevilleS]

Bug Description

In our 1.8.0 release, we removed the default config TOML from our published Docker image. In doing so, we fixed one issue where there were unexpected defaults being set for a variety of config variables, but we also broke any deployments that relied on configuring the service using ENV variables exclusively.

Steps to Reproduce

1. Run docker run ethyca/fidesops:1.8.0 (without any env variables, any volume mounts, etc.)
2. Observe the following error:

~/git/fidesops% docker run ethyca/fidesops:1.8.0
Startup configuration: reloading = False, dev_mode = False
Startup configuration: pii logging = False
fidesops.toml or fidesctl.toml or fides.toml could not be loaded:
ValidationError: 8 validation errors for FidesopsConfig
database
  field required (type=value_error.missing)
redis
  field required (type=value_error.missing)
security
  field required (type=value_error.missing)
execution
  field required (type=value_error.missing)
root_user
  field required (type=value_error.missing)
admin_ui
  field required (type=value_error.missing)
notifications
  field required (type=value_error.missing)
port
  field required (type=value_error.missing)

Expected behavior

If you read the error carefully above, it's actually attempting to load a fidesops.toml and failing, so it becomes unable to load any of the sub-configurations for database, redis, security, etc. Because of this, it never gets to the stage where it can attempt to read ENV variables to set those values.

In other words: it should be possible to simply docker run ethyca/fidesops and for the resulting error should be that a small handful of required config variables need to be set, including:

• FIDESOPS__SECURITY__APP_ENCRYPTION_KEY
• FIDESOPS__SECURITY__OAUTH_ROOT_CLIENT_ID
• FIDESOPS__SECURITY__OAUTH_ROOT_CLIENT_SECRET
• FIDESOPS__DATABASE__SERVER
• FIDESOPS__DATABASE__USER
• FIDESOPS__DATABASE__PASSWORD
• FIDESOPS__DATABASE__DB
• FIDESOPS__DATABASE__PORT
• FIDESOPS__REDIS__HOST
• FIDESOPS__REDIS__PORT
• FIDESOPS__REDIS__PASSWORD

These are the minimum ENV variables documented in the deployment guide here: https://ethyca.github.io/fidesops/deployment/#install-fidesops-via-docker

I believe all other config variables should be either optional (i.e. if not set, the features that rely on them can't be used at runtime) or have safe defaults. There may be a few exceptions which we can discuss and document, but ultimately the goal here should be to make it possible to run this again via docker.

Workaround

Today, I found that the safest workaround is two parts.

First, to workaround the required TOML loading, provide this minimal config TOML to the container (via a mount, e.g. the --mount or -v option to docker run):

port = 8080

[database]
enabled = true

[redis]
db_index = 0

[security]

[execution]
task_retry_count = 0
task_retry_delay = 1
task_retry_backoff = 1

[root_user]

[admin_ui]

[notifications]

Second, to workaround a few other config variables that don't have safe defaults, provide some ENV vars:

• FIDESOPS__SECURITY__DRP_JWT_SECRET=<some secret> (this should be optional)

Lastly, to workaround the requirement that a TOML file exists and is writeable for analytics purposes, provide a few more ENV vars:

• FIDESOPS__ROOT_USER__ANALYTICS_OPT_OUT=true
• FIDESOPS__ROOT_USER__ANALYTICS_ID=<some guid>

I don't like this third workaround, but note that it's related to a different issue we are tracking here: ethyca/fides#1721. I only document it here for temporary purposes, but solving that "read-only" TOML problem should be addressed in ethyca/fides#1721, not here.

Environment

• Version: 1.8.0
• OS: Mac OSX
• Python Version: 3.9
• Docker Version: 20.10.17