Variable substitution in DataSource JSON files

[bajtos]

To support https://12factor.net/ application loading their database connection strings from ENV variables, we should enhance our DataSource booter to provide variable substitution similar to what https://github.com/strongloop/loopback-boot already provides for LB3 applications.

Example LB3 config file:

// server/datasources.json
{
  "db": {
    "connector": "mysql",
    "url": "${MYSQL_URL}",
  }
}

Proposed format for LB4:

// src/datasources/db.json
{
  "name": "db",
  "connector": "mysql",
  "url": "${ENV.MYSQL_URL}",
}

By using ENV. prefix, we are keeping doors open to add other variable sources in the future, e.g. APP. prefix to access application-level bindings.

To support numeric values, allow the following format (notice the + sign after the opening bracket):

{
  "port": "${+ENV.PORT}",
}

Acceptance criteria

• A new helper function accepting configuration object loaded from the JSON file, an object with values to use for substitutions (conceptually {ENV: process.env}), and returning modified configuration object with substitutions resolved according to rules described above. Include comprehensive test coverage.
• Substitution should be applied to values at all levels, including deeply nested properties.
• Support conversion to numeric values as described above.
• @loopback/boot recognizes and replaces variables in datasource configuration. Leverage the already implemented helper function.
• process.env must not be accessed directly from individual booters. Instead, the environment object should be provided via app.boot() arguments. For example, we can add a new property to BootExecutionOptions.
• Modify examples and CLI templates to pass process.env value to app.boot(), so that variable substitution can be performed.
• Documentation
• Blog post

Out of scope

• Allow extensions to contribute custom convention for environment-specific operational configuration - see the spike [Spike] Allow extensions to contribute custom convention for environment-specific operational configuration #1464

[marioestradarosa]

@bajtos , I was analyzing the datasource booter and found that this can be done before it calls the repositoryMixin.dataSource easily. I am suggesting that we use a convention based on the datasource name, without specifying any variable in the .json file.

// src/datasources/db.json
{
  "name": "myDBName",
  "connector": "mysql",
  "host": "xxxx",
  "port": "yyyy",
  "user": "xxxxxkkd",
...
}

Then in the booter it will evaluate ENV.MYDBNAME_USER , if null then it will use the "xxxxxkkd" property value already there, otherwise it will use the value specified by ENV.MYDBNAME_USER.

thoughts?

[emonddr]

we need a JSON utility that can perform dependency injection post boot.

[emonddr]

The team brought up a few questions:

1. How to declare a variable for non-string values. for example PORT NUMBER.? (Can't use string)
2. How do we specify a default value?
3. Impossible to add a comment in the file.

We figure we may need a file format different from .json. Perhaps .config.

Please clarify the Acceptance Criteria in light of our comments. Thank you.

[emonddr]

@bajtos ^

[bajtos]

@marioestradarosa Thank you for chiming in!

I was analyzing the datasource booter and found that this can be done before it calls the repositoryMixin.dataSource easily.

Yes, that was my roughly my idea too.

I am suggesting that we use a convention based on the datasource name, without specifying any variable in the .json file.

Then in the booter it will evaluate ENV.MYDBNAME_USER , if null then it will use the "xxxxxkkd" property value already there, otherwise it will use the value specified by ENV.MYDBNAME_USER.

I see this as an alternative/complementary solution to what I have originally proposed.

In many environments, the application developer does not control the environment variables. For example, Heroku uses DATABASE_URL environment variable - see https://devcenter.heroku.com/articles/heroku-postgresql#connecting-in-node-js.

How to declare a variable for non-string values. for example PORT NUMBER.? (Can't use string)

Good question! I am proposing to use the following syntax for numeric values (notice the + sign after the opening bracket):

// a number
"port": "${+ENV.PORT}",
// a string
"host": "${ENV.HOST}",

Compare this to regular javascript:

const port = +'0'; 
const host = 'localhost';

How do we specify a default value?

That's out of scope of this story. When the environment value is not set, then the config property is set to undefined.

Impossible to add a comment in the file.
We figure we may need a file format different from .json. Perhaps .config.

That's out of scope of this story. Please open a new issue to discuss this idea.

[bajtos]

Updated the acceptance criteria:

• A new helper function accepting configuration object loaded from the JSON file, an object with values to use for substitutions (conceptually {ENV: process.env}), and returning modified configuration object with substitutions resolved according to rules described above. Include comprehensive test coverage.
• Substitution should be applied to values at all levels, including deeply nested properties.
• Support conversion to numeric values as described above.
• @loopback/boot recognizes and replaces variables in datasource configuration. Leverage the already implemented helper function.
• process.env must not be accessed directly from individual booters. Instead, the environment object should be provided via app.boot() arguments. For example, we can add a new property to BootExecutionOptions.
• Modify examples and CLI templates to pass process.env value to app.boot(), so that variable substitution can be performed.
• Documentation
• Blog post