Extension points for authentication/authorization

NOTICE

@@ -77,3 +77,9 @@ This product contains a modified version of The Guava Authors's Closer class fro
    * https://github.com/google/guava
  * COMMIT TAG:
    * https://github.com/google/guava/blob/c462d69329709f72a17a64cb229d15e76e72199c
+
+This product contains code adapted from Apache Hadoop
+  * LICENSE:
+    * https://github.com/apache/hadoop/blob/trunk/LICENSE.txt (Apache License, Version 2.0)
+  * HOMEPAGE:
+    * http://hadoop.apache.org/

api/src/main/java/io/druid/guice/annotations/EscalatedGlobal.java

@@ -0,0 +1,37 @@
+/*
+ * Licensed to Metamarkets Group Inc. (Metamarkets) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. Metamarkets licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package io.druid.guice.annotations;
+
+import com.google.inject.BindingAnnotation;
+
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+
+/**
+ */
+@BindingAnnotation
+@Target({ElementType.FIELD, ElementType.PARAMETER, ElementType.METHOD})
+@Retention(RetentionPolicy.RUNTIME)
+@PublicApi
+public @interface EscalatedGlobal
+{
+}

benchmarks/src/main/java/io/druid/benchmark/query/SqlBenchmark.java

@@ -39,6 +39,8 @@
 import io.druid.query.dimension.DimensionSpec;
 import io.druid.query.groupby.GroupByQuery;
 import io.druid.segment.QueryableIndex;
+import io.druid.server.security.AuthConfig;
+import io.druid.server.security.AuthTestUtils;
 import io.druid.sql.calcite.planner.DruidPlanner;
 import io.druid.sql.calcite.planner.PlannerConfig;
 import io.druid.sql.calcite.planner.PlannerFactory;
@@ -112,13 +114,15 @@ public void setup() throws Exception
     final PlannerConfig plannerConfig = new PlannerConfig();
 
     this.walker = new SpecificSegmentsQuerySegmentWalker(conglomerate).add(dataSegment, index);
-
     plannerFactory = new PlannerFactory(
         CalciteTests.createMockSchema(walker, plannerConfig),
         CalciteTests.createMockQueryLifecycleFactory(walker),
         CalciteTests.createOperatorTable(),
         CalciteTests.createExprMacroTable(),
         plannerConfig,
+        new AuthConfig(),
+        AuthTestUtils.TEST_AUTHENTICATOR_MAPPER,
+        AuthTestUtils.TEST_AUTHORIZER_MAPPER,
         CalciteTests.getJsonMapper()
     );
     groupByQuery = GroupByQuery

docs/content/configuration/auth.md

@@ -0,0 +1,102 @@
+---
+layout: doc_page
+---
+
+# Authentication and Authorization
+
+|Property|Type|Description|Default|Required|
+|--------|-----------|--------|--------|--------|
+|`druid.auth.authenticationChain`|JSON List of Strings|List of Authenticator type names|["allowAll"]|no|
+|`druid.auth.escalatedAuthenticator`|String|Type of the Authenticator that should be used for internal Druid communications. This Authenticator must be present in `druid.auth.authenticationChain`.|"allowAll"|no|
+|`druid.auth.authorizers`|JSON List of Strings|List of Authorizer type names |["allowAll"]|no|
+
+## Enabling Authentication/Authorization
+
+## Authentication Chain
+Authentication decisions are handled by a chain of Authenticator instances. A request will be checked by Authenticators in the sequence defined by the `druid.auth.authenticationChain`.
+
+Authenticator implementions are provided by extensions.
+
+For example, the following authentication chain definition enables the Kerberos and HTTP Basic authenticators, from the `druid-kerberos` and `druid-basic-security` core extensions, respectively:
+
+```
+druid.auth.authenticationChain=["kerberos", "basic"]
+```
+
+A request will pass through all Authenticators in the chain, until one of the Authenticators successfully authenticates the request or sends an HTTP error response. Authenticators later in the chain will be skipped after the first successful authentication or if the request is terminated with an error response.
+
+If no Authenticator in the chain successfully authenticated a request or sent an HTTP error response, an HTTP error response will be sent at the end of the chain.
+
+Druid includes a built-in Authenticator, used for the default unsecured configuration.
+
+### AllowAll Authenticator
+
+This built-in Authenticator authenticates all requests, and always directs them to an Authorizer named "allowAll". It is not intended to be used for anything other than the default unsecured configuration.
+
+## Internal Authenticator
+The `druid.auth.escalatedAuthenticator` property determines what authentication scheme should be used for internal Druid cluster communications (such as when a broker node communicates with historical nodes for query processing).
+
+The Authenticator chosen for this property must also be present in `druid.auth.authenticationChain`.
+
+## Authorizers
+Authorization decisions are handled by an Authorizer. The `druid.auth.authorizers` property determines what Authorizer implementations will be active.
+
+There are two built-in Authorizers, "default" and "noop". Other implementations are provided by extensions.
+
+For example, the following authorizers definition enables the "basic" implementation from `druid-basic-security`:
+
+```
+druid.auth.authorizers=["basic"]
+```
+
+
+Only a single Authorizer will authorize any given request.
+
+Druid includes one built in authorizer:
+
+### AllowAll Authorizer
+The Authorizer with type name "allowAll" accepts all requests.
+
+## Default Unsecured Configuration
+
+When `druid.auth.authenticationChain` is left empty or unspecified, Druid will create an authentication chain with a single AllowAll Authenticator named "allowAll".
+
+When `druid.auth.authorizers` is left empty or unspecified, Druid will create a single AllowAll Authorizer named "allowAll".
+
+The default value of `druid.auth.escalatedAuthenticator` is "allowAll" to match the default unsecured Authenticator/Authorizer configurations.
+
+## Authenticator to Authorizer Routing
+
+When an Authenticator successfully authenticates a request, it must attach a AuthenticationResult to the request, containing an information about the identity of the requester, as well as the name of the Authorizer that should authorize the authenticated request.
+
+An Authenticator implementation should provide some means through configuration to allow users to select what Authorizer(s) the Authenticator should route requests to.
+
+## Internal System User
+
+Internal requests between Druid nodes (non-user initiated communications) need to have authentication credentials attached. 
+
+These requests should be run as an "internal system user", an identity that represents the Druid cluster itself, with full access permissions.
+
+The details of how the internal system user is defined is left to Authorizer and Authenticator implementations.
+
+### Authorizer Internal System User Handling
+
+Authorizers implementations must recognize and authorize an identity for the "internal system user", with full access permissions.
+
+### Authenticator Internal System User Handling
+
+Authenticators must implement three methods related to the internal system user:
+
+```java
+  public HttpClient createEscalatedClient(HttpClient baseClient);
+
+  public org.eclipse.jetty.client.HttpClient createEscalatedJettyClient(org.eclipse.jetty.client.HttpClient baseClient);
+
+  public AuthenticationResult createEscalatedAuthenticationResult();
+```
+
+`createEscalatedClient` returns an wrapped HttpClient that attaches the credentials of the "internal system user" to requests.
+
+`createEscalatedJettyClient` is similar to `createEscalatedClient`, except that it operates on a Jetty HttpClient.
+
+`createEscalatedAuthenticationResult` returns an AuthenticationResult containing the identity of the "internal system user".

docs/content/development/extensions-core/druid-kerberos.md

@@ -5,29 +5,46 @@ layout: doc_page
 # Druid-Kerberos
 
 Druid Extension to enable Authentication for Druid Nodes using Kerberos.
-This extension adds AuthenticationFilter which is used to protect HTTP Endpoints using the simple and protected GSSAPI negotiation mechanism [SPNEGO](https://en.wikipedia.org/wiki/SPNEGO). 
+This extension adds an Authenticator which is used to protect HTTP Endpoints using the simple and protected GSSAPI negotiation mechanism [SPNEGO](https://en.wikipedia.org/wiki/SPNEGO). 
 Make sure to [include](../../operations/including-extensions.html) `druid-kerberos` as an extension.
 
 
 ## Configuration
 
+### Creating an Authenticator
+```
+druid.auth.authenticatorChain=["MyKerberosAuthenticator"]
+
+druid.auth.authenticator.MyKerberosAuthenticator.type=kerberos
+```
+
+To use the Kerberos authenticator, add an authenticator with type `kerberos` to the authenticatorChain. The example above uses the name "MyKerberosAuthenticator" for the Authenticator.
+
+Configuration of the named authenticator is assigned through properties with the form:
+
+```
+druid.auth.authenticator.<authenticatorName>.<authenticatorProperty>
+```
+
+The configuration examples in the rest of this document will use "kerberos" as the name of the authenticator being configured.
+
+### Properties
 |Property|Possible Values|Description|Default|required|
 |--------|---------------|-----------|-------|--------|
-|`druid.hadoop.security.kerberos.principal`|`druid@EXAMPLE.COM`| Principal user name, used for internal node communication|empty|Yes|
-|`druid.hadoop.security.kerberos.keytab`|`/etc/security/keytabs/druid.keytab`|Path to keytab file used for internal node communication|empty|Yes|
-|`druid.hadoop.security.spnego.principal`|`HTTP/_HOST@EXAMPLE.COM`| SPNego service principal used by druid nodes|empty|Yes|
-|`druid.hadoop.security.spnego.keytab`|`/etc/security/keytabs/spnego.service.keytab`|SPNego service keytab used by druid nodes|empty|Yes|
-|`druid.hadoop.security.spnego.authToLocal`|`RULE:[1:$1@$0](druid@EXAMPLE.COM)s/.*/druid DEFAULT`|It allows you to set a general rule for mapping principal names to local user names. It will be used if there is not an explicit mapping for the principal name that is being translated.|DEFAULT|No|
-|`druid.hadoop.security.spnego.excludedPaths`|`['/status','/health']`| Array of HTTP paths which which does NOT need to be authenticated.|None|No|
-|`druid.hadoop.security.spnego.cookieSignatureSecret`|`secretString`| Secret used to sign authentication cookies. It is advisable to explicitly set it, if you have multiple druid ndoes running on same machine with different ports as the Cookie Specification does not guarantee isolation by port.|<Random value>|No|
+|`druid.auth.authenticator.kerberos.internalClientPrincipal`|`druid@EXAMPLE.COM`| Principal user name, used for internal node communication|empty|Yes|
+|`druid.auth.authenticator.kerberos.internalClientKeytab`|`/etc/security/keytabs/druid.keytab`|Path to keytab file used for internal node communication|empty|Yes|
+|`druid.auth.authenticator.kerberos.serverPrincipal`|`HTTP/_HOST@EXAMPLE.COM`| SPNego service principal used by druid nodes|empty|Yes|
+|`druid.auth.authenticator.kerberos.serverKeytab`|`/etc/security/keytabs/spnego.service.keytab`|SPNego service keytab used by druid nodes|empty|Yes|
+|`druid.auth.authenticator.kerberos.authToLocal`|`RULE:[1:$1@$0](druid@EXAMPLE.COM)s/.*/druid DEFAULT`|It allows you to set a general rule for mapping principal names to local user names. It will be used if there is not an explicit mapping for the principal name that is being translated.|DEFAULT|No|
+|`druid.auth.authenticator.kerberos.excludedPaths`|`['/status','/health']`| Array of HTTP paths which which does NOT need to be authenticated.|None|No|
+|`druid.auth.authenticator.kerberos.cookieSignatureSecret`|`secretString`| Secret used to sign authentication cookies. It is advisable to explicitly set it, if you have multiple druid ndoes running on same machine with different ports as the Cookie Specification does not guarantee isolation by port.|<Random value>|No|
+|`druid.auth.authenticator.kerberos.authorizerName`|`secretString`| Secret used to sign authentication cookies. It is advisable to explicitly set it, if you have multiple druid ndoes running on same machine with different ports as the Cookie Specification does not guarantee isolation by port.|<Random value>|No|
 
 As a note, it is required that the SPNego principal in use by the druid nodes must start with HTTP (This specified by [RFC-4559](https://tools.ietf.org/html/rfc4559)) and must be of the form "HTTP/_HOST@REALM". 
 The special string _HOST will be replaced automatically with the value of config `druid.host`
 
 ### Auth to Local Syntax
-
-
-`druid.hadoop.security.spnego.authToLocal` allows you to set a general rules for mapping principal names to local user names.
+`druid.auth.authenticator.kerberos.authToLocal` allows you to set a general rules for mapping principal names to local user names.
 The syntax for mapping rules is `RULE:\[n:string](regexp)s/pattern/replacement/g`. The integer n indicates how many components the target principal should have. If this matches, then a string will be formed from string, substituting the realm of the principal for $0 and the n‘th component of the principal for $n. e.g. if the principal was druid/admin then `\[2:$2$1suffix]` would result in the string `admindruidsuffix`.
 If this string matches regexp, then the s//\[g] substitution command will be run over the string. The optional g will cause the substitution to be global over the string, instead of replacing only the first match in the string.
 If required, multiple rules can be be joined by newline character and specified as a String.