TLS support

common/src/main/java/io/druid/metadata/PasswordProvider.java

@@ -24,7 +24,7 @@
 
 
 /**
- * Implement this for different ways to (optionally securely) access db passwords.
+ * Implement this for different ways to (optionally securely) access secrets.
  */
 @JsonTypeInfo(use = JsonTypeInfo.Id.NAME, property = "type", defaultImpl = DefaultPasswordProvider.class)
 @JsonSubTypes(value = {

common/src/test/java/io/druid/metadata/MetadataStorageConnectorConfigTest.java

@@ -76,7 +76,7 @@ public void testEquals() throws IOException
   private static final ObjectMapper jsonMapper = new ObjectMapper();
 
   @Test
-  public void testMetadaStorageConnectionConfigSimplePassword() throws Exception
+  public void testMetadataStorageConnectionConfigSimplePassword() throws Exception
   {
     testMetadataStorageConnectionConfig(
         true,
@@ -90,7 +90,7 @@ public void testMetadaStorageConnectionConfigSimplePassword() throws Exception
   }
 
   @Test
-  public void testMetadaStorageConnectionConfigWithDefaultProviderPassword() throws Exception
+  public void testMetadataStorageConnectionConfigWithDefaultProviderPassword() throws Exception
   {
     testMetadataStorageConnectionConfig(
         true,

distribution/pom.xml

@@ -110,6 +110,8 @@
                                 <argument>io.druid.extensions:druid-stats</argument>
                                 <argument>-c</argument>
                                 <argument>io.druid.extensions:druid-examples</argument>
+                                <argument>-c</argument>
+                                <argument>io.druid.extensions:simple-client-sslcontext</argument>
                                 <argument>${druid.distribution.pulldeps.opts}</argument>
                             </arguments>
                         </configuration>

docs/content/configuration/broker.md

@@ -15,7 +15,8 @@ The broker node uses several of the global configs in [Configuration](../configu
 |Property|Description|Default|
 |--------|-----------|-------|
 |`druid.host`|The host for the current node. This is used to advertise the current processes location as reachable from another node and should generally be specified such that `http://${druid.host}/` could actually talk to this process|InetAddress.getLocalHost().getCanonicalHostName()|
-|`druid.port`|This is the port to actually listen on; unless port mapping is used, this will be the same port as is on `druid.host`|8082|
+|`druid.plaintextPort`|This is the port to actually listen on; unless port mapping is used, this will be the same port as is on `druid.host`|8082|
+|`druid.tlsPort`|TLS port for HTTPS connector, if [druid.server.http.tls](../operations/tls-support.html) is set then this config will be used. If `druid.host` contains port then that port will be ignored. This should be a non-negative Integer.|8282|
 |`druid.service`|The name of the service. This is used as a dimension when emitting metrics and alerts to differentiate between the various services|druid/broker|
 
 ### Query Configs

docs/content/configuration/coordinator.md

@@ -15,7 +15,8 @@ The coordinator node uses several of the global configs in [Configuration](../co
 |Property|Description|Default|
 |--------|-----------|-------|
 |`druid.host`|The host for the current node. This is used to advertise the current processes location as reachable from another node and should generally be specified such that `http://${druid.host}/` could actually talk to this process|InetAddress.getLocalHost().getCanonicalHostName()|
-|`druid.port`|This is the port to actually listen on; unless port mapping is used, this will be the same port as is on `druid.host`|8081|
+|`druid.plaintextPort`|This is the port to actually listen on; unless port mapping is used, this will be the same port as is on `druid.host`|8081|
+|`druid.tlsPort`|TLS port for HTTPS connector, if [druid.server.http.tls](../operations/tls-support.html) is set then this config will be used. If `druid.host` contains port then that port will be ignored. This should be a non-negative Integer.|8281|
 |`druid.service`|The name of the service. This is used as a dimension when emitting metrics and alerts to differentiate between the various services|druid/coordinator|
 
 ### Coordinator Operation

docs/content/configuration/historical.md

@@ -15,7 +15,8 @@ The historical node uses several of the global configs in [Configuration](../con
 |Property|Description|Default|
 |--------|-----------|-------|
 |`druid.host`|The host for the current node. This is used to advertise the current processes location as reachable from another node and should generally be specified such that `http://${druid.host}/` could actually talk to this process|InetAddress.getLocalHost().getCanonicalHostName()|
-|`druid.port`|This is the port to actually listen on; unless port mapping is used, this will be the same port as is on `druid.host`|8083|
+|`druid.plaintextPort`|This is the port to actually listen on; unless port mapping is used, this will be the same port as is on `druid.host`|8083|
+|`druid.tlsPort`|TLS port for HTTPS connector, if [druid.server.http.tls](../operations/tls-support.html) is set then this config will be used. If `druid.host` contains port then that port will be ignored. This should be a non-negative Integer.|8283|
 |`druid.service`|The name of the service. This is used as a dimension when emitting metrics and alerts to differentiate between the various services|druid/historical|
 
 ### General Configuration

docs/content/configuration/index.md

@@ -234,7 +234,7 @@ These properties specify the jdbc connection and other configuration around the
 |`druid.metadata.storage.type`|The type of metadata storage to use. Choose from "mysql", "postgresql", or "derby".|derby|
 |`druid.metadata.storage.connector.connectURI`|The jdbc uri for the database to connect to|none|
 |`druid.metadata.storage.connector.user`|The username to connect with.|none|
-|`druid.metadata.storage.connector.password`|The password provider or String password used to connect with.|none|
+|`druid.metadata.storage.connector.password`|The [Password Provider](../operations/password-provider.html) or String password used to connect with.|none|
 |`druid.metadata.storage.connector.createTables`|If Druid requires a table and it doesn't exist, create it?|true|
 |`druid.metadata.storage.tables.base`|The base name for tables.|druid|
 |`druid.metadata.storage.tables.segments`|The table to use to look for segments.|druid_segments|
@@ -246,26 +246,6 @@ These properties specify the jdbc connection and other configuration around the
 |`druid.metadata.storage.tables.supervisors`|Used by the indexing service to store supervisor configurations.|druid_supervisors|
 |`druid.metadata.storage.tables.audit`|The table to use for audit history of configuration changes e.g. Coordinator rules.|druid_audit|
 
-#### Password Provider
-
-Environment variable password provider provides password by looking at specified environment variable. Use this in order to avoid specifying password in runtime.properties file.
-e.g 
-
-```json
-{ 
-    "type": "environment",
-    "variable": "METADATA_STORAGE_PASSWORD"   
-}
-```
-
-The values are described below. 
-
-|Field|Type|Description|Required|
-|-----|----|-----------|--------|
-|`type`|String|password provider type|Yes: `environment`|
-|`variable`|String|environment variable to read password from|Yes|
-
-
 ### Deep Storage
 
 The configurations concern how to push and pull [Segments](../design/segments.html) from deep storage.

docs/content/configuration/indexing-service.md

@@ -9,14 +9,24 @@ The indexing service uses several of the global configs in [Configuration](../co
 
 ### Must be set on Overlord and Middle Manager
 
-#### Node Configs
+#### Overlord Node Configs
 
 |Property|Description|Default|
 |--------|-----------|-------|
 |`druid.host`|The host for the current node. This is used to advertise the current processes location as reachable from another node and should generally be specified such that `http://${druid.host}/` could actually talk to this process|InetAddress.getLocalHost().getCanonicalHostName()|
-|`druid.port`|This is the port to actually listen on; unless port mapping is used, this will be the same port as is on `druid.host`|8090|
+|`druid.plaintextPort`|This is the port to actually listen on; unless port mapping is used, this will be the same port as is on `druid.host`|8090|
+|`druid.tlsPort`|TLS port for HTTPS connector, if [druid.server.http.tls](../operations/tls-support.html) is set then this config will be used. If `druid.host` contains port then that port will be ignored. This should be a non-negative Integer.|8290|
 |`druid.service`|The name of the service. This is used as a dimension when emitting metrics and alerts to differentiate between the various services|druid/overlord|
 
+#### MiddleManager Node Configs
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.host`|The host for the current node. This is used to advertise the current processes location as reachable from another node and should generally be specified such that `http://${druid.host}/` could actually talk to this process|InetAddress.getLocalHost().getCanonicalHostName()|
+|`druid.plaintextPort`|This is the port to actually listen on; unless port mapping is used, this will be the same port as is on `druid.host`|8091|
+|`druid.tlsPort`|TLS port for HTTPS connector, if [druid.server.http.tls](../operations/tls-support.html) is set then this config will be used. If `druid.host` contains port then that port will be ignored. This should be a non-negative Integer.|8291|
+|`druid.service`|The name of the service. This is used as a dimension when emitting metrics and alerts to differentiate between the various services|druid/middlemanager|
+
 #### Task Logging
 
 If you are running the indexing service in remote mode, the task logs must be stored in S3, Azure Blob Store, Google Cloud Storage or HDFS.
@@ -297,8 +307,9 @@ Middle managers pass their configurations down to their child peons. The middle
 |`druid.indexer.runner.javaOpts`|*DEPRECATED* A string of -X Java options to pass to the peon's JVM. Quotable parameters or parameters with spaces are encouraged to use javaOptsArray|""|
 |`druid.indexer.runner.javaOptsArray`|A json array of strings to be passed in as options to the peon's jvm. This is additive to javaOpts and is recommended for properly handling arguments which contain quotes or spaces like `["-XX:OnOutOfMemoryError=kill -9 %p"]`|`[]`|
 |`druid.indexer.runner.maxZnodeBytes`|The maximum size Znode in bytes that can be created in Zookeeper.|524288|
-|`druid.indexer.runner.startPort`|The port that peons begin running on.|8100|
-|`druid.indexer.runner.separateIngestionEndpoint`|*Deprecated.* Use separate server and consequently separate jetty thread pool for ingesting events|false|
+|`druid.indexer.runner.startPort`|Starting port used for peon processes, should be greater than 1023.|8100|
+|`druid.indexer.runner.tlsStartPort`|Starting TLS port for peon processes, should be greater than 1023.|8300|
+|`druid.indexer.runner.separateIngestionEndpoint`|*Deprecated.* Use separate server and consequently separate jetty thread pool for ingesting events. Not supported with TLS.|false|
 |`druid.worker.ip`|The IP of the worker.|localhost|
 |`druid.worker.version`|Version identifier for the middle manager.|0|
 |`druid.worker.capacity`|Maximum number of tasks the middle manager can accept.|Number of available processors - 1|

docs/content/configuration/realtime.md

@@ -16,7 +16,8 @@ The realtime node uses several of the global configs in [Configuration](../confi
 |Property|Description|Default|
 |--------|-----------|-------|
 |`druid.host`|The host for the current node. This is used to advertise the current processes location as reachable from another node and should generally be specified such that `http://${druid.host}/` could actually talk to this process|InetAddress.getLocalHost().getCanonicalHostName()|
-|`druid.port`|This is the port to actually listen on; unless port mapping is used, this will be the same port as is on `druid.host`|8084|
+|`druid.plaintextPort`|This is the port to actually listen on; unless port mapping is used, this will be the same port as is on `druid.host`|8084|
+|`druid.tlsPort`|TLS port for HTTPS connector, if [druid.server.http.tls](../operations/tls-support.html) is set then this config will be used. If `druid.host` contains port then that port will be ignored. This should be a non-negative Integer.|8284|
 |`druid.service`|The name of the service. This is used as a dimension when emitting metrics and alerts to differentiate between the various services|druid/realtime|
 
 ### Realtime Operation

docs/content/development/extensions-core/simple-client-sslcontext.md

@@ -0,0 +1,22 @@
+---
+layout: doc_page
+---
+
+## Simple SSLContext Provider Module
+
+This module contains a simple implementation of [SSLContext](http://docs.oracle.com/javase/8/docs/api/javax/net/ssl/SSLContext.html)
+that will be injected to be used with HttpClient that Druid nodes use internally to communicate with each other. To learn more about
+Java's SSL support, please refer to [this](http://docs.oracle.com/javase/8/docs/technotes/guides/security/jsse/JSSERefGuide.html) guide.
+
+# Configuration
+
+|Property|Description|Default|Required|
+|--------|-----------|-------|--------|
+|`druid.client.https.protocol`|SSL protocol to use.|`TLSv1.2`|no|
+|`druid.client.https.trustStoreType`|The type of the key store where trusted root certificates are stored.|`java.security.KeyStore.getDefaultType()`|no|
+|`druid.client.https.trustStorePath`|The file path or URL of the TLS/SSL Key store where trusted root certificates are stored.|none|yes|
+|`druid.client.https.trustStoreAlgorithm`|Algorithm to be used by TrustManager to validate certificate chains|`javax.net.ssl.TrustManagerFactory.getDefaultAlgorithm()`|no|
+|`druid.client.https.trustStorePassword`|The [Password Provider](../../operations/password-provider.html) or String password for the Trust Store.|none|yes|
+
+This [document](http://docs.oracle.com/javase/8/docs/technotes/guides/security/StandardNames.html) lists all the possible
+values for the above mentioned configs among others provided by Java implementation.

docs/content/development/extensions.md

@@ -37,6 +37,7 @@ Core extensions are maintained by Druid committers.
 |druid-stats|Statistics related module including variance and standard deviation.|[link](../development/extensions-core/stats.html)|
 |mysql-metadata-storage|MySQL metadata store.|[link](../development/extensions-core/mysql.html)|
 |postgresql-metadata-storage|PostgreSQL metadata store.|[link](../development/extensions-core/postgresql.html)|
+|simple-client-sslcontext|Simple SSLContext provider module to be used by internal HttpClient talking to other nodes over HTTPS.|[link](../development/extensions-core/simple-client-sslcontext.html)|
 
 # Community Extensions
 

docs/content/development/modules.md

@@ -178,6 +178,23 @@ Adding new Jersey resources to a module requires calling the following code to b
 Jerseys.addResource(binder, NewResource.class);
 ```
 
+### Adding a new Password Provider implementation
+
+You will need to implement `io.druid.metadata.PasswordProvider` interface. For every place where Druid uses PasswordProvider, a new instance of the implementation will be created,
+thus make sure all the necessary information required for fetching each password is supplied during object instantiation.
+In your implementation of `io.druid.initialization.DruidModule`, `getJacksonModules` should look something like this -
+
+``` java
+    return ImmutableList.of(
+        new SimpleModule("SomePasswordProviderModule")
+            .registerSubtypes(
+                new NamedType(SomePasswordProvider.class, "some")
+            )
+    );
+```
+
+where `SomePasswordProvider` is the implementation of `PasswordProvider` interface, you can have a look at `io.druid.metadata.EnvironmentVariablePasswordProvider` for example.
+
 ### Bundle your extension with all the other Druid extensions
 
 When you do `mvn install`, Druid extensions will be packaged within the Druid tarball and `extensions` directory, which are both underneath `distribution/target/`.

docs/content/operations/password-provider.md

@@ -0,0 +1,31 @@
+#### Password Provider
+
+Druid needs some passwords for accessing various secured systems like metadata store, Key Store containing server certificates etc.
+All these passwords have corresponding runtime properties associated with them, for example `druid.metadata.storage.connector.password` corresponds to the metadata store password.
+
+By default users can directly set the passwords in plaintext for these runtime properties, for example `druid.metadata.storage.connector.password=pwd` sets the metadata store password
+to be used by Druid to connect to metadata store to `pwd`. Apart from this, users can use environment variables to get password in following way -
+
+Environment variable password provider provides password by looking at specified environment variable. Use this in order to avoid specifying password in runtime.properties file.
+e.g
+
+```json
+{ "type": "environment", "variable": "METADATA_STORAGE_PASSWORD" }
+```
+
+The values are described below.
+
+|Field|Type|Description|Required|
+|-----|----|-----------|--------|
+|`type`|String|password provider type|Yes: `environment`|
+|`variable`|String|environment variable to read password from|Yes|
+
+However, many times users may want their own way to optionally securely fetch password during runtime of the Druid process.
+Druid allows this by users to implement their own `PasswordProvider` interface and create a Druid extension to register this implementation at Druid process startup.
+Please have a look at "Adding a new Password Provider implementation" on this [page](../development/modules.html) to learn more.
+
+To use this implementation, simply set the relevant password runtime property to something similar as was done for Environment variable password provider like -
+
+```json
+{ "type": "<registered_password_provider_name>", "<jackson_property>": "<value>", ... }
+```