Add Environment Variable DynamicConfigProvider by bananaaggle · Pull Request #11377 · apache/druid

bananaaggle · 2021-06-22T10:39:09Z

As #9351 described, DynamicConfigProvider should replace PasswordProvider. PasswordProvider has an implementation named EnvironmentVariablePasswordProvider, which allow users to read config from environment variable. This PR provides a same implementation for DynamicConfigProvider. There is an example for its usage:

"DynamicConfigProvider": {
  "type": "environment",
  "variables":
         {
            "userNameVariable": "MY_USER_NAME_VAR",
            "passwordVariable": "MY_PASSWORD_VAR"
         }
}

This PR has:

been self-reviewed.
- using the concurrency checklist (Remove this item if the PR doesn't have any relation to concurrency.)
added documentation for new or modified features or behaviors.
added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
added or updated version, license, or notice information in licenses.yaml
added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
added integration tests.
been tested in a test Druid cluster.

jihoonson

@bananaaggle thanks for the PR. I left some comments.

jihoonson · 2021-06-25T06:32:13Z

+      @JsonProperty("variables") Map<String, String> config
+  )
+  {
+    this.variables = ImmutableMap.copyOf(config);


config can be null if there is something wrong in the configuration. This will throw NPE in that case. It would be better to tell what is wrong than NPE with no message.

jihoonson · 2021-06-25T06:34:19Z

+import java.util.Map;
+import java.util.Objects;
+
+public class EnvironmentVariableDynamicConfigProvider implements DynamicConfigProvider


Suggested change

public class EnvironmentVariableDynamicConfigProvider implements DynamicConfigProvider

public class EnvironmentVariableDynamicConfigProvider implements DynamicConfigProvider<String>

jihoonson · 2021-06-25T06:34:33Z

+  }
+
+  @Override
+  public Map getConfig()


Suggested change

public Map getConfig()

public Map<String> getConfig()

jihoonson · 2021-06-25T06:41:14Z

+  @Override
+  public int hashCode()
+  {
+    return variables != null ? variables.hashCode() : 0;


variables doesn't seem to be able to null.

jihoonson · 2021-06-25T20:45:20Z

+    try {
+      this.variables = ImmutableMap.copyOf(config);
+    }
+    catch (NullPointerException e) {
+      log.error(e, "Can not parse config by EnvironmentVariableDynamicConfigProvider! Please check your config file.");
+      throw e;
+    }


It would be nice to propagate the error to users so that they don't have to look up logs when this happens.

Suggested change

try {

this.variables = ImmutableMap.copyOf(config);

}

catch (NullPointerException e) {

log.error(e, "Can not parse config by EnvironmentVariableDynamicConfigProvider! Please check your config file.");

throw e;

}

this.variables = ImmutableMap.copyOf(Preconditions.checkNotNull(config, "config"));

clintropolis

👍

clintropolis · 2021-06-25T23:57:40Z

+
+  private static final Logger log = new Logger(EnvironmentVariableDynamicConfigProvider.class);
+
+  private ImmutableMap<String, String> variables;


nit: this can be final

jihoonson · 2021-06-30T02:51:03Z

@bananaaggle thanks for the quick response. The code change LGTM, but there are 2 things missing here, one is documentation and another is tests. For documentation, I think you can add some in docs/operations/dynamic-config-provider.md. For tests, you can add some unit tests in EnvironmentVariableDynamicConfigProviderTest that should run with a proper set of environment variables. To let Travis CI run those new tests successfully, you will need to add those environment variables in .travis.yaml too. I can help you with adding these, let me know if you have questions.

bananaaggle · 2021-06-30T04:49:42Z

@bananaaggle thanks for the quick response. The code change LGTM, but there are 2 things missing here, one is documentation and another is tests. For documentation, I think you can add some in docs/operations/dynamic-config-provider.md. For tests, you can add some unit tests in EnvironmentVariableDynamicConfigProviderTest that should run with a proper set of environment variables. To let Travis CI run those new tests successfully, you will need to add those environment variables in .travis.yaml too. I can help you with adding these, let me know if you have questions.

Document added. I'll add more tests in week.

bananaaggle · 2021-07-01T10:47:19Z

@bananaaggle thanks for the quick response. The code change LGTM, but there are 2 things missing here, one is documentation and another is tests. For documentation, I think you can add some in docs/operations/dynamic-config-provider.md. For tests, you can add some unit tests in EnvironmentVariableDynamicConfigProviderTest that should run with a proper set of environment variables. To let Travis CI run those new tests successfully, you will need to add those environment variables in .travis.yaml too. I can help you with adding these, let me know if you have questions.

Hi, @jihoonson. I add unit test for getConfig(). To set environment variable, I find a method named setEnv(Map<String, String> newenv) in stackoverflow which can change jvm environment variable by reflection. Do you think it is a proper solution? Is necessary to add environment variables in .travis.yaml? If add environment variables in .travis.yaml, this unit test will work in travis, but users can not pass it without travis. What unit tests you think is necessary to be added in next step?

jihoonson

@bananaaggle thank you for adding a documentation! I left some comments on it.

For unit tests, I was thinking of some tests that should run with a set of certain environment variables. So, for Travis, those variables should be set in .travis.yaml. To run those tests locally, those variables should be set and passed properly to the process that runs the tests. I thought this would be the easiest way to add tests, but if you think there is a better way, I'm OK with it as long as those tests are not flaky.

jihoonson · 2021-07-02T03:54:51Z

+
+`EnvironmentVariableDynamicConfigProvider` can be used to replace `EnvironmentVariablePasswordProvider`. This class allow users to avoid exposing passwords or other secret information in the runtime.properties file. You can set environment variables in the following example:
+```json
+druid.metadata.storage.connector.dynamicConfigProvider={"type": "environment","variables":{"user": "MY_USER_NAME_VAR","password": "MY_PASSWORD_VAR"}


We don't support dynamicConfig for metadata yet.

Suggested change

druid.metadata.storage.connector.dynamicConfigProvider={"type": "environment","variables":{"user": "MY_USER_NAME_VAR","password": "MY_PASSWORD_VAR"}

druid.some.config.dynamicConfigProvider={"type": "environment","variables":{"secret1": "SECRET1_VAR","secret2": "SECRET2_VAR"}

Fixed. In #11389, I make a design to replace PasswordProvider. I think it can work with metadata and other classes which use PasswordProvider.

Cool, I will take a look at that PR too. Thanks!

jihoonson · 2021-07-02T03:57:19Z


+## EnvironmentVariableDynamicConfigProvider
+
+`EnvironmentVariableDynamicConfigProvider` can be used to replace `EnvironmentVariablePasswordProvider`. This class allow users to avoid exposing passwords or other secret information in the runtime.properties file. You can set environment variables in the following example:


Suggested change

`EnvironmentVariableDynamicConfigProvider` can be used to replace `EnvironmentVariablePasswordProvider`. This class allow users to avoid exposing passwords or other secret information in the runtime.properties file. You can set environment variables in the following example:

`EnvironmentVariableDynamicConfigProvider` can be used to avoid exposing credentials or other secret information in the configuration files using environment variables. An example to use this configProvider is:

jihoonson

Hey @bananaaggle, thank you for adding unit tests. I have some concerns about the new code added. Please see my comments.

jihoonson · 2021-07-11T17:51:10Z

+      theEnvironmentField.setAccessible(true);
+      Map<String, String> env = (Map<String, String>) theEnvironmentField.get(null);
+      env.putAll(newenv);
+      Field theCaseInsensitiveEnvironmentField = processEnvironmentClass.getDeclaredField("theCaseInsensitiveEnvironment");


What is this theCaseInsensitiveEnvironment variable? I don't see it in java.lang.ProcessEnvironment.

jihoonson · 2021-07-11T17:51:48Z

+    catch (NoSuchFieldException e) {
+      Class[] classes = Collections.class.getDeclaredClasses();
+      Map<String, String> env = System.getenv();
+      for (Class cl : classes) {
+        if ("java.util.Collections$UnmodifiableMap".equals(cl.getName())) {
+          Field field = cl.getDeclaredField("m");
+          field.setAccessible(true);
+          Object obj = field.get(env);
+          Map<String, String> map = (Map<String, String>) obj;
+          map.clear();
+          map.putAll(newenv);
+        }
+      }
+    }
+  }


Can you elaborate more on what this catch clause is doing?

jihoonson · 2021-07-11T17:59:48Z

+    Assert.assertEquals("123", ((EnvironmentVariableDynamicConfigProvider) provider).getConfig().get("password"));
+  }
+
+  protected static void setEnv(Map<String, String> newenv) throws Exception


This method seems to have a couple of issues.

This method seems to be copied from the code snippet in https://stackoverflow.com/a/7201825/4127682. If this is true, we cannot use it directly because it violates the ASF license policy. You can get some hint from the stack overflow, but should not copy from it.

As noted in the stack overflow link, the environment variables should be reset because they are shared by all tests run by the same JVM process.

It would be nice to add some comments to help others understand the code. I left some comments for this.

I've changed this unit test. Separated setting and getting environment variables. I use a map to record which environment variables are changed and add a method to recover those system environment variables after test. Add comment for getENVMap().

bananaaggle · 2021-07-21T05:37:11Z

Hi @jihoonson! I've changed this unit test but CI tests failed. It seems caused by travis error, can you help me rerun it? And is there something in code needed to change? Thanks!

jihoonson · 2021-08-04T14:26:58Z

@bananaaggle sorry, I forgot about this PR. I restarted the timed-out tests. Will finish my review today.

jihoonson

LGTM. Thanks @bananaaggle!

bananaaggle · 2021-08-05T08:09:08Z

LGTM. Thanks @bananaaggle!

Thank you very much for review! In 11389, I make a design to replace PasswordProvider, but I'm not sure this design is proper or not. Can you help review it? If design is proper, I will involve more classes which use PasswordProvider in that PR. I also comment about it in 9351.

add_environment_variable_DynamicConfigProvider

a95949b

bananaaggle mentioned this pull request Jun 25, 2021

Adding CredentialsProvider, deprecating PasswordProvider #9351

Open

jihoonson reviewed Jun 25, 2021

View reviewed changes

fix code

72a8a32

jihoonson reviewed Jun 25, 2021

View reviewed changes

clintropolis added the Area - Operations label Jun 25, 2021

clintropolis reviewed Jun 25, 2021

View reviewed changes

yuanyi added 2 commits June 26, 2021 12:35

code fixed

a6f2042

code fixed

02a4397

add document

7b0f972

yuanyi added 4 commits June 30, 2021 12:50

fix doc

5f297c7

fix doc

18c22f1

add more unit test

60fdd53

fix style

cc774cc

jihoonson reviewed Jul 2, 2021

View reviewed changes

yuanyi added 2 commits July 2, 2021 12:20

fix document

6d352bc

bug fixed

13d62fe

jihoonson reviewed Jul 11, 2021

View reviewed changes

yuanyi added 3 commits July 12, 2021 19:44

fix unit test

b32d3e0

fix comment

47c7f44

fix test

86be099

jihoonson approved these changes Aug 5, 2021

View reviewed changes

jihoonson merged commit 23d7d71 into apache:master Aug 5, 2021

clintropolis added this to the 0.22.0 milestone Aug 12, 2021

clintropolis mentioned this pull request Sep 3, 2021

[Draft] 0.22.0 Release Notes #11657

Closed

	public class EnvironmentVariableDynamicConfigProvider implements DynamicConfigProvider
	public class EnvironmentVariableDynamicConfigProvider implements DynamicConfigProvider<String>


		private static final Logger log = new Logger(EnvironmentVariableDynamicConfigProvider.class);

		private ImmutableMap<String, String> variables;

	druid.metadata.storage.connector.dynamicConfigProvider={"type": "environment","variables":{"user": "MY_USER_NAME_VAR","password": "MY_PASSWORD_VAR"}
	druid.some.config.dynamicConfigProvider={"type": "environment","variables":{"secret1": "SECRET1_VAR","secret2": "SECRET2_VAR"}


		## EnvironmentVariableDynamicConfigProvider

		`EnvironmentVariableDynamicConfigProvider` can be used to replace `EnvironmentVariablePasswordProvider`. This class allow users to avoid exposing passwords or other secret information in the runtime.properties file. You can set environment variables in the following example:

Conversation

bananaaggle commented Jun 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jihoonson left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

clintropolis left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jihoonson commented Jun 30, 2021

Uh oh!

bananaaggle commented Jun 30, 2021

Uh oh!

bananaaggle commented Jul 1, 2021

Uh oh!

jihoonson left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jihoonson left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bananaaggle Jul 12, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bananaaggle commented Jul 21, 2021

Uh oh!

jihoonson commented Aug 4, 2021

Uh oh!

jihoonson left a comment

Choose a reason for hiding this comment

Uh oh!

bananaaggle commented Aug 5, 2021

Uh oh!

Reviewers

bananaaggle commented Jun 22, 2021 •

edited

Loading

bananaaggle Jul 12, 2021 •

edited

Loading