Remove Escalator jetty http client escalation method by jon-wei · Pull Request #5322 · apache/druid

jon-wei · 2018-02-02T04:08:28Z

The jetty HttpClient used in the Router should not have any "internal system user" escalation applied to it, as the proxy requests sent by the router should be sent with the same credentials in the original requests that the Router receives.

This did not occur in practice with the existing druid-basic-security extension because its implementation added an AuthenticationStore to the jetty HttpClient, which is only used when the client receives an authentication challenge (see https://www.eclipse.org/jetty/documentation/9.4.x/http-client-authentication.html). The challenge does not occur when using HTTP basic auth with the Router because ProxyServlet copies the original request's headers into the proxy request and the original request must have had valid HTTP basic auth credentials to be accepted by the Router, bypassing the need for the authentication challenge.

However, any Escalator implementations that function by unconditionally adding credential headers to outgoing requests could incorrectly attach the privileged internal system user's credentials.

The escalated client was also being used for query cancellations sent by the Router, which is incorrect behavior as these cancellation requests should use the credentials of the requester, not the privileged internal system user.

This PR removes the unnecessary/incorrect createEscalatedJettyClient method from the Escalator interface, fixes the broadcastClient such that it copies the headers of the original request, and also fixes a crash in AuthorizerMapperModule when the authorizers list is null.

gianm

👍 LGTM

b-slim · 2018-04-21T16:01:12Z

@jon-wei I think this broke the Kerberos/Spnego Authentication work flow.
In nutshell , the credentials received by the router and handed off to broker are not valid because the token is generated for a given hostname (here the Router) thus when the Broker tries to validate the token the request gets rejected. Am not sure if rolling back the this patch is the way to go, or if we should add another mechanism for such a case where the credentials are host based and not transferable. Let me know what you think, am still going through the code to understand how it works internally...

pjain1 · 2018-04-21T16:01:51Z

Hello, This employee can no longer access email on this account. Your email will not be forwarded. If you have a sales inquiry, please contact sales@yahoo-inc.com If you require assistance with a legal matter, please contact legal-notices@yahoo-inc.com Thank you!

b-slim · 2018-04-23T23:11:28Z

@jon-wei can you please take a look at this question? am still unsure why can't we consider the hop from router to broker is like the hop from broker to historical?
Also in all cases before the query gets to the proxyServelet the Router did run all the authentication chains so am not sure what is the security issue?

gianm · 2018-04-23T23:20:10Z

@b-slim The hops are different because the broker verifies permissions on the request, but the router doesn't. So it's ok for the broker to use an escalated connection to talk to historicals (since it has already verified that the end user access is authorized). But it's not ok for the router to use an escalated connection to talk to brokers. The only thing the router has done is verify that the end user is authenticated, but it has not verified that it is authorized.

jon-wei · 2018-04-23T23:22:46Z

@b-slim

am still unsure why can't we consider the hop from router to broker is like the hop from broker to historical?

The router only passes on requests without performing any authorization checks (like, does a user have access rights to query datasource X?), the router isn't aware of the semantics of specific requests so it can't really check that, so the original identity of the requester needs to be passed on to the forwarding destination

jon-wei · 2018-04-23T23:24:39Z

@b-slim does the router -> broker work with Kerberos if you set druid.auth.authenticator.kerberos.cookieSignatureSecret to the same value on both router and broker?

I seem to recall needing to do that when I was testing this patch

b-slim · 2018-04-23T23:27:36Z

@gianm why can't the router do the authorized stuff, it can do that as well?
seems like the answer is #5322 (comment)
NVM.

b-slim · 2018-04-23T23:30:20Z

@jon-wei i will try your suggestion and update this.

b-slim · 2018-04-23T23:44:39Z

@jon-wei that config did not fix the issue still getting denied when the query reach the Broker.
I think this means we are left with 2 options:

1 to escalate with a Do As client kind of thingy not sure.
2 Escalate as root Druid and attache the verified ClientId and let the Broker do the Authorize stuff.

In both cases we need to add hook to create an escalated client to be used between Router and Broker.

jon-wei · 2018-04-23T23:46:15Z

@b-slim In theory the router could do all the authorization and forward requests as an escalated user, but I think that puts too much responsibility on the router in terms of needing to understand how to authorize requests. For example, suppose the router needs to forward a SQL query. To authorize such a query the SQL query needs to be parsed and planned into Druid queries, so the router would need to run a SQL planner, that workload/burden of knowledge is better left to the actual endpoints.

Or suppose the router is forwarding coordinator/overlord requests (#5369), I don't think its right to make the router need to understand the semantics of every possible request.

b-slim · 2018-04-23T23:54:53Z

@jon-wei you bring very good points but i think that adding the hook and giving the implementor of the module the choice of doing every thing at the Router or at Broker is a valid design choice as well. For instance the current Kerberos implementation doesn't do any authorization stuff thus there is no need really to make the Broker do the work.

jon-wei · 2018-04-24T00:22:56Z

For instance the current Kerberos implementation doesn't do any authorization stuff thus there is no need really to make the Broker do the work.

Suppose the Kerberos authenticator is being used in conjunction with an authorization system, e.g. a role-based access control system, it would be incorrect for the Kerberos authenticator to use an escalated identity to forward requests (i.e., it's a big security hole that would bypass all authorization checks).

I think this means we are left with 2 options:
1 to escalate with a Do As client kind of thingy not sure.
2 Escalate as root Druid and attache the verified ClientId and let the Broker do the Authorize stuff.
In both cases we need to add hook to create an escalated client to be used between Router and Broker.

If it's the only option, I suppose we could build a way to send escalated requests with a subidentity, where the subidentity is used for authorization checks at the forwarding endpoint but I think it'd be good to see if there are other options.

I'm being wary of adding escalated communication paths because of potential for privilege escalation bugs (ideally I think the broker -> historical request path should inherit the original request credentials as well, vs. using escalated comms, but that's not implemented right now).

I'm not deeply familiar with Kerberos, is there something we could do with ticket forwarding here (https://web.mit.edu/kerberos/krb5-1.13/doc/user/tkt_mgmt.html)? Is this related?

As a side note, how do you test your Kerberos stuff? Do you have any testing VMs/dockerfiles that are publicly available or you can share?

I set up a kerberos environment from scratch sometime last year, it was quite a pain, wondering if you know of any more convenient tools.

b-slim · 2018-04-24T00:29:17Z

As a side note, how do you test your Kerberos stuff? Do you have any testing VMs/dockerfiles that are publicly available or you can share?

I set up a kerberos environment from scratch sometime last year, it was quite a pain, wondering if you know of any more convenient tools.

@jon-wei yes this task is very painful, i tried to do the docker thingy but it did not really work kerberos depends on lot of other services like DNS and other stuff thus am relying on a pre-deployed internal test cluster where i attache a remote debugger....

jon-wei · 2018-04-24T00:38:46Z

In both cases we need to add hook to create an escalated client to be used between Router and Broker.

To clarify what I was saying above, maybe we do need to add additional hooks to the auth system to support this use case, but I think it'd be good to minimize "send as internal superuser" cases when possible.

b-slim · 2018-04-24T00:45:52Z

but I think it'd be good to minimize "send as internal superuser" cases when possible.

@jon-wei I do agree, am looking at how this can be done with a clean design.
am thinking one way is to decouple Authorization and Authentication, what you think?

jon-wei · 2018-04-24T03:51:06Z

am thinking one way is to decouple Authorization and Authentication, what you think?

Hm, I think the current system is pretty decoupled, it was intended to allow multiple authentication schemes to interoperate with various authorization mechanisms, the request flow is at a high level like the following:

Authenticator checks incoming request, determines and validates the identity of the user, generates an AuthenticationResult (contains a string that identifies the user and the name of the authorizer that should handle the request, the result is intended to be fed into any valid Authorizer implementation)
Authorizer only looks at the AuthenticationResult attached to the request to make authorization decisions, it doesn't care how the Authenticator determined that result

jon-wei · 2018-04-24T04:12:35Z

Well, decoupled as much as I think they can be, authorization needs some kind of authenticated
"user identity" to make any sense

gianm · 2018-04-24T14:39:11Z

Due to reasons raised by @jon-wei in #5322 (comment), I think we should keep the responsibility for verifying authorization at the final destination, not the router. The router would need the ability to understand every possible endpoint, and there are too many.

The DoAs / subidentity stuff you are talking about sounds related to #5436 -- we might be able to solve two issues with one approach. Although for #5436 I didn't have it being security critical in mind. I was just thinking of it being some extra metadata for metrics, rather than being used for downstream authorization.

@b-slim does the router -> broker work with Kerberos if you set
druid.auth.authenticator.kerberos.cookieSignatureSecret to the same value on both router and broker?

I seem to recall needing to do that when I was testing this patch

@jon-wei were you testing on a single machine with druid.auth.authenticator.kerberos.serverPrincipal set the same on both the broker and router? Or were the values different? From my reading of how SPNEGO works, the method we are using today may only work if the broker and router have the same server principal name (SPN).

I'm being wary of adding escalated communication paths because of potential for privilege escalation bugs (ideally I think the broker -> historical request path should inherit the original request credentials as well, vs. using escalated comms, but that's not implemented right now).

Reading over this ticket it doesn't sound like inheriting original credentials makes sense with Kerberos. The SPN (server principal name) is part of the negotiation and so the same ticket will not work for more than one server. So it sounds like we need to do something beyond just forwarding the negotiated ticket from the original requester.

b-slim · 2018-04-24T16:24:12Z

@jon-wei and @gianm seems we both agree that currently the Kerberos Auth module is broken when using Router in a Different machine with Different host name. In the Ker/Spnego terms this is called a double Hop
Thus i think we agree that we need a way to decorate the Http Client thus we need to add something like the escalated or call it decorated jetty client.

As i said above probably the most kosher way to do this is to use the Do As (here the Router acting on behalf of client). But looking at the current Kerberos Auth module this do As is not even needed because there is no Authorization at all, thus a first fix will be to use the bare escalated as Router.
IMO this is a legitimate use case and common pattern we see, where only admins access the Druid Cluster and delegate the Authorization to something like Superset or any other entity like Hive Server, thus the need for Authentication only.

Again your concerns are totally valid and i think we can achieve both patterns by letting the implementor of Auth module decide whether it is okay or not to escalate as root-druid.

gianm · 2018-04-24T17:24:34Z

But looking at the current Kerberos Auth module this do As is not even needed because there is no Authorization at all, thus a first fix will be to use the bare escalated as Router.
IMO this is a legitimate use case and common pattern we see, where only admins access the Druid Cluster and delegate the Authorization to something like Superset or any other entity like Hive Server, thus the need for Authentication only.

That makes the router a security hole that completely bypasses the authorization mechanisms. I believe you when you say you have use cases where this is okay, but if we support this behavior, I really believe it has to be off by default and very clear from the documentation that it is not a fully secure configuration. I would also prefer it to be a router-specific configuration rather than something more general.

We should also keep researching how to support this properly, since that workaround would not be usable for anyone that actually wants to enforce permissions at a more granular level than "you're in" vs. "you're out".

jon-wei · 2018-04-24T17:49:13Z

My first preference would be to see if there's a way to use existing Kerberos features/mechanisms to handle this use case, I'll do some research on this.

If that's not possible, maybe we could add in some kind of impersonation path (like, router signs the request it forwards with some secret shared by the druid nodes, the request contains the identity of the sending user, the request hits a new Authenticator implementation at the broker that checks for such signed messages and authenticates that impersonated user by checking the signature), rather than a full-on "send as superuser" path.

jon-wei · 2018-04-24T20:32:24Z

@b-slim Do you have a logs or stack trace of the error that happens in the router->broker case? I don't have a setup to reproduce this yet.

jon-wei · 2018-04-24T21:03:38Z

Hm, I tried having my broker and router bind to different IP addresses in my testing setup, not able to reproduce the failure, my query to the router succeeds (192.168.56.1 is an interface on a virtual box network)

2018-04-24T20:46:56,741 INFO [qtp127269549-75] io.druid.security.kerberos.KerberosAuthenticator - Login using keytab /Users/jw/kerberos/http2-rc4-hmac.keytab, for principal HTTP/192.168.56.1@VM
2018-04-24T20:46:56,786 INFO [qtp127269549-75] io.druid.security.kerberos.KerberosAuthenticator - Initialized, principal HTTP/192.168.56.1@VM from keytab /Users/jw/kerberos/http2-rc4-hmac.keytab

2018-04-24T20:46:56,967 INFO [qtp1339630217-119] io.druid.security.kerberos.KerberosAuthenticator - Login using keytab /Users/jw/kerberos/http2-rc4-hmac.keytab, for principal HTTP/127.0.0.1@VM
2018-04-24T20:46:56,969 INFO [qtp1339630217-119] io.druid.security.kerberos.KerberosAuthenticator - Initialized, principal HTTP/127.0.0.1@VM from keytab /Users/jw/kerberos/http2-rc4-hmac.keytab

b-slim · 2018-04-24T21:21:28Z

@jon-wei i get the following

java.security.PrivilegedActionException: GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)
stackTrace = {StackTraceElement[36]@9389} 
 0 = {StackTraceElement@9391} "sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:856)"
 1 = {StackTraceElement@9392} "sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342)"
 2 = {StackTraceElement@9393} "sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285)"
 3 = {StackTraceElement@9394} "io.druid.security.kerberos.DruidKerberosAuthenticationHandler$2.run(DruidKerberosAuthenticationHandler.java:183)"
 4 = {StackTraceElement@9395} "io.druid.security.kerberos.DruidKerberosAuthenticationHandler$2.run(DruidKerberosAuthenticationHandler.java:161)"
 5 = {StackTraceElement@9396} "java.security.AccessController.doPrivileged(Native Method)"
 6 = {StackTraceElement@9397} "javax.security.auth.Subject.doAs(Subject.java:422)"
 7 = {StackTraceElement@9398} "io.druid.security.kerberos.DruidKerberosAuthenticationHandler.authenticate(DruidKerberosAuthenticationHandler.java:160)"
 8 = {StackTraceElement@9399} "io.druid.security.kerberos.KerberosAuthenticator$1.doFilterSuper(KerberosAuthenticator.java:295)"
 9 = {StackTraceElement@9400} "io.druid.security.kerberos.KerberosAuthenticator$1.doFilter(KerberosAuthenticator.java:265)"
 10 = {StackTraceElement@9401} "io.druid.server.security.AuthenticationWrappingFilter.doFilter(AuthenticationWrappingFilter.java:60)"
 11 = {StackTraceElement@9402} "org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)"
 12 = {StackTraceElement@9403} "io.druid.server.security.SecuritySanityCheckFilter.doFilter(SecuritySanityCheckFilter.java:86)"
 13 = {StackTraceElement@9404} "org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759)"
 14 = {StackTraceElement@9405} "org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)"
 15 = {StackTraceElement@9406} "org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224)"
 16 = {StackTraceElement@9407} "org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)"
 17 = {StackTraceElement@9408} "org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)"
 18 = {StackTraceElement@9409} "org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)"
 19 = {StackTraceElement@9410} "org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)"
 20 = {StackTraceElement@9411} "org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)"
 21 = {StackTraceElement@9412} "org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493)"
 22 = {StackTraceElement@9413} "org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52)"
 23 = {StackTraceElement@9414} "org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)"
 24 = {StackTraceElement@9415} "org.eclipse.jetty.server.Server.handle(Server.java:534)"
 25 = {StackTraceElement@9416} "org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)"
 26 = {StackTraceElement@9417} "org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)"
 27 = {StackTraceElement@9418} "org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)"
 28 = {StackTraceElement@9419} "org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)"
 29 = {StackTraceElement@9420} "org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)"
 30 = {StackTraceElement@9421} "org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)"
 31 = {StackTraceElement@9422} "org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)"
 32 = {StackTraceElement@9423} "org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)"
 33 = {StackTraceElement@9424} "org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)"
 34 = {StackTraceElement@9425} "org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)"
 35 = {StackTraceElement@9426} "java.lang.Thread.run(Thread.java:748)"

b-slim · 2018-04-24T21:22:27Z

i have two similar Routers if i hit one colocated with broker it works if i hit the second it does not.

b-slim · 2018-04-24T21:27:59Z

@jon-wei FYI i am running out of 0.12.0 i don;t see any major code changes since then.
on the Druid logs i see the followings

2018-04-24T21:11:25,320 ERROR [qtp2006056784-185] io.druid.security.kerberos.KerberosAuthenticator - Could not find matching key from server creds.
2018-04-24T21:11:25,320 DEBUG [qtp2006056784-185] io.druid.security.kerberos.KerberosAuthenticator - Request [{http://ctr-e138-1518143905142-237396-01-000003.hwx.site:8082/druid/v2/}] triggering authentication
2018-04-24T21:11:25,694 DEBUG [qtp2006056784-185] io.druid.security.kerberos.KerberosAuthenticator - Authentication exception: GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)

jon-wei · 2018-04-24T21:29:48Z

Cool, thanks for the info.

I'll try spinning up non-colocated brokers and routers to see if I can reproduce.

b-slim · 2018-04-24T21:36:32Z

@jon-wei am looking at io.druid.security.kerberos.KerberosAuthenticator#getPrincipalFromRequestNew and am wondering if this a common logic used by other projects or it is a Druid specific thingy?

jon-wei · 2018-04-24T21:43:41Z

that's Druid specific, I added it to get the principal name

b-slim · 2018-04-24T22:45:15Z

@jon-wei here the full stack with more debug flags

2018-04-24T22:42:43,745 INFO [qtp409957428-173] io.druid.security.kerberos.KerberosAuthenticator - Initialized, principal HTTP/ctr-e138-1518143905142-237396-01-000003.hwx.site@EXAMPLE.COM from keytab /etc/security/keytabs/spnego.service.keytab
2018-04-24T22:42:43,747 ERROR [qtp409957428-173] io.druid.security.kerberos.KerberosAuthenticator - Could not find matching key from server creds.
2018-04-24T22:42:43,748 DEBUG [qtp409957428-173] io.druid.security.kerberos.KerberosAuthenticator - Request [{http://ctr-e138-1518143905142-237396-01-000003.hwx.site:8082/druid/v2/}] triggering authentication
Found KeyTab /etc/security/keytabs/spnego.service.keytab for HTTP/ctr-e138-1518143905142-237396-01-000003.hwx.site@EXAMPLE.COM
Found KeyTab /etc/security/keytabs/spnego.service.keytab for HTTP/ctr-e138-1518143905142-237396-01-000003.hwx.site@EXAMPLE.COM
Entered Krb5Context.acceptSecContext with state=STATE_NEW
>>> KeyTabInputStream, readName(): EXAMPLE.COM
>>> KeyTabInputStream, readName(): HTTP
>>> KeyTabInputStream, readName(): ctr-e138-1518143905142-237396-01-000003.hwx.site
>>> KeyTab: load() entry length: 108; type: 16
>>> KeyTabInputStream, readName(): EXAMPLE.COM
>>> KeyTabInputStream, readName(): HTTP
>>> KeyTabInputStream, readName(): ctr-e138-1518143905142-237396-01-000003.hwx.site
>>> KeyTab: load() entry length: 116; type: 18
>>> KeyTabInputStream, readName(): EXAMPLE.COM
>>> KeyTabInputStream, readName(): HTTP
>>> KeyTabInputStream, readName(): ctr-e138-1518143905142-237396-01-000003.hwx.site
>>> KeyTab: load() entry length: 100; type: 23
>>> KeyTabInputStream, readName(): EXAMPLE.COM
>>> KeyTabInputStream, readName(): HTTP
>>> KeyTabInputStream, readName(): ctr-e138-1518143905142-237396-01-000003.hwx.site
>>> KeyTab: load() entry length: 92; type: 3
>>> KeyTabInputStream, readName(): EXAMPLE.COM
>>> KeyTabInputStream, readName(): HTTP
>>> KeyTabInputStream, readName(): ctr-e138-1518143905142-237396-01-000003.hwx.site
>>> KeyTab: load() entry length: 100; type: 17
Looking for keys for: HTTP/ctr-e138-1518143905142-237396-01-000003.hwx.site@EXAMPLE.COM
Added key: 17version: 2
Found unsupported keytype (3) for HTTP/ctr-e138-1518143905142-237396-01-000003.hwx.site@EXAMPLE.COM
Added key: 23version: 2
Added key: 18version: 2
Added key: 16version: 2
>>> EType: sun.security.krb5.internal.crypto.Des3CbcHmacSha1KdEType
2018-04-24T22:42:43,765 ERROR [qtp409957428-173] io.druid.security.kerberos.KerberosAuthenticator - WTF
org.apache.hadoop.security.authentication.client.AuthenticationException: GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)
        at io.druid.security.kerberos.DruidKerberosAuthenticationHandler.authenticate(DruidKerberosAuthenticationHandler.java:221) ~[druid-kerberos-0.12.0-rc2-SNAPSHOT.jar:0.12.0-rc2-SNAPSHOT]
        at io.druid.security.kerberos.KerberosAuthenticator$1.doFilterSuper(KerberosAuthenticator.java:295) [druid-kerberos-0.12.0-rc2-SNAPSHOT.jar:0.12.0-rc2-SNAPSHOT]
        at io.druid.security.kerberos.KerberosAuthenticator$1.doFilter(KerberosAuthenticator.java:265) [druid-kerberos-0.12.0-rc2-SNAPSHOT.jar:0.12.0-rc2-SNAPSHOT]
        at io.druid.server.security.AuthenticationWrappingFilter.doFilter(AuthenticationWrappingFilter.java:60) [druid-server-0.12.0.3.0.0.0-1195.jar:0.12.0.3.0.0.0-1195]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) [jetty-servlet-9.3.19.v20170502.jar:9.3.19.v20170502]
        at io.druid.server.security.SecuritySanityCheckFilter.doFilter(SecuritySanityCheckFilter.java:86) [druid-server-0.12.0.3.0.0.0-1195.jar:0.12.0.3.0.0.0-1195]
        at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) [jetty-servlet-9.3.19.v20170502.jar:9.3.19.v20170502]
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) [jetty-servlet-9.3.19.v20170502.jar:9.3.19.v20170502]
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:224) [jetty-server-9.3.19.v20170502.jar:9.3.19.v20170502]
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) [jetty-server-9.3.19.v20170502.jar:9.3.19.v20170502]
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) [jetty-servlet-9.3.19.v20170502.jar:9.3.19.v20170502]
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) [jetty-server-9.3.19.v20170502.jar:9.3.19.v20170502]
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) [jetty-server-9.3.19.v20170502.jar:9.3.19.v20170502]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.3.19.v20170502.jar:9.3.19.v20170502]
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:493) [jetty-server-9.3.19.v20170502.jar:9.3.19.v20170502]
        at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:52) [jetty-server-9.3.19.v20170502.jar:9.3.19.v20170502]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) [jetty-server-9.3.19.v20170502.jar:9.3.19.v20170502]
        at org.eclipse.jetty.server.Server.handle(Server.java:534) [jetty-server-9.3.19.v20170502.jar:9.3.19.v20170502]
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) [jetty-server-9.3.19.v20170502.jar:9.3.19.v20170502]
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [jetty-server-9.3.19.v20170502.jar:9.3.19.v20170502]
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283) [jetty-io-9.3.19.v20170502.jar:9.3.19.v20170502]
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108) [jetty-io-9.3.19.v20170502.jar:9.3.19.v20170502]
        at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) [jetty-io-9.3.19.v20170502.jar:9.3.19.v20170502]
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) [jetty-util-9.3.19.v20170502.jar:9.3.19.v20170502]
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) [jetty-util-9.3.19.v20170502.jar:9.3.19.v20170502]
        at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) [jetty-util-9.3.19.v20170502.jar:9.3.19.v20170502]
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) [jetty-util-9.3.19.v20170502.jar:9.3.19.v20170502]
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) [jetty-util-9.3.19.v20170502.jar:9.3.19.v20170502]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: org.ietf.jgss.GSSException: Failure unspecified at GSS-API level (Mechanism level: Checksum failed)
        at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:856) ~[?:1.8.0_161]
        at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342) ~[?:1.8.0_161]
        at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285) ~[?:1.8.0_161]
        at io.druid.security.kerberos.DruidKerberosAuthenticationHandler$2.run(DruidKerberosAuthenticationHandler.java:183) ~[?:?]
        at io.druid.security.kerberos.DruidKerberosAuthenticationHandler$2.run(DruidKerberosAuthenticationHandler.java:161) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161]
        at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_161]
        at io.druid.security.kerberos.DruidKerberosAuthenticationHandler.authenticate(DruidKerberosAuthenticationHandler.java:160) ~[?:?]
        ... 28 more
Caused by: sun.security.krb5.KrbCryptoException: Checksum failed
        at sun.security.krb5.internal.crypto.Des3CbcHmacSha1KdEType.decrypt(Des3CbcHmacSha1KdEType.java:96) ~[?:1.8.0_161]
        at sun.security.krb5.internal.crypto.Des3CbcHmacSha1KdEType.decrypt(Des3CbcHmacSha1KdEType.java:88) ~[?:1.8.0_161]
        at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:175) ~[?:1.8.0_161]
        at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:281) ~[?:1.8.0_161]
        at sun.security.krb5.KrbApReq.<init>(KrbApReq.java:149) ~[?:1.8.0_161]
        at sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:108) ~[?:1.8.0_161]
        at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:829) ~[?:1.8.0_161]
        at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342) ~[?:1.8.0_161]
        at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285) ~[?:1.8.0_161]
        at io.druid.security.kerberos.DruidKerberosAuthenticationHandler$2.run(DruidKerberosAuthenticationHandler.java:183) ~[?:?]
        at io.druid.security.kerberos.DruidKerberosAuthenticationHandler$2.run(DruidKerberosAuthenticationHandler.java:161) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161]
        at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_161]
        at io.druid.security.kerberos.DruidKerberosAuthenticationHandler.authenticate(DruidKerberosAuthenticationHandler.java:160) ~[?:?]
        ... 28 more
Caused by: java.security.GeneralSecurityException: Checksum failed
        at sun.security.krb5.internal.crypto.dk.DkCrypto.decrypt(DkCrypto.java:362) ~[?:1.8.0_161]
        at sun.security.krb5.internal.crypto.Des3.decrypt(Des3.java:79) ~[?:1.8.0_161]
        at sun.security.krb5.internal.crypto.Des3CbcHmacSha1KdEType.decrypt(Des3CbcHmacSha1KdEType.java:94) ~[?:1.8.0_161]
        at sun.security.krb5.internal.crypto.Des3CbcHmacSha1KdEType.decrypt(Des3CbcHmacSha1KdEType.java:88) ~[?:1.8.0_161]
        at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:175) ~[?:1.8.0_161]
        at sun.security.krb5.KrbApReq.authenticate(KrbApReq.java:281) ~[?:1.8.0_161]
        at sun.security.krb5.KrbApReq.<init>(KrbApReq.java:149) ~[?:1.8.0_161]
        at sun.security.jgss.krb5.InitSecContextToken.<init>(InitSecContextToken.java:108) ~[?:1.8.0_161]
        at sun.security.jgss.krb5.Krb5Context.acceptSecContext(Krb5Context.java:829) ~[?:1.8.0_161]
        at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:342) ~[?:1.8.0_161]
        at sun.security.jgss.GSSContextImpl.acceptSecContext(GSSContextImpl.java:285) ~[?:1.8.0_161]
        at io.druid.security.kerberos.DruidKerberosAuthenticationHandler$2.run(DruidKerberosAuthenticationHandler.java:183) ~[?:?]
        at io.druid.security.kerberos.DruidKerberosAuthenticationHandler$2.run(DruidKerberosAuthenticationHandler.java:161) ~[?:?]
        at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_161]
        at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_161]
        at io.druid.security.kerberos.DruidKerberosAuthenticationHandler.authenticate(DruidKerberosAuthenticationHandler.java:160) ~[?:?]
        ... 28 more

b-slim · 2018-04-24T22:48:10Z

and this what a happy ending looks like when use colocated broker/router

2018-04-24T22:46:17,795 ERROR [qtp409957428-188] io.druid.security.kerberos.KerberosAuthenticator - Could not find matching key from server creds.
2018-04-24T22:46:17,796 DEBUG [qtp409957428-188] io.druid.security.kerberos.KerberosAuthenticator - Request [{http://ctr-e138-1518143905142-237396-01-000003.hwx.site:8082/druid/v2/}] triggering authentication
Found KeyTab /etc/security/keytabs/spnego.service.keytab for HTTP/ctr-e138-1518143905142-237396-01-000003.hwx.site@EXAMPLE.COM
Found KeyTab /etc/security/keytabs/spnego.service.keytab for HTTP/ctr-e138-1518143905142-237396-01-000003.hwx.site@EXAMPLE.COM
Entered Krb5Context.acceptSecContext with state=STATE_NEW
Looking for keys for: HTTP/ctr-e138-1518143905142-237396-01-000003.hwx.site@EXAMPLE.COM
Added key: 17version: 2
Found unsupported keytype (3) for HTTP/ctr-e138-1518143905142-237396-01-000003.hwx.site@EXAMPLE.COM
Added key: 23version: 2
Added key: 18version: 2
Added key: 16version: 2
>>> EType: sun.security.krb5.internal.crypto.Des3CbcHmacSha1KdEType
Using builtin default etypes for permitted_enctypes
default etypes for permitted_enctypes: 18 17 16 23.
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
MemoryCache: add 1524609977/563031/EE1A4FB089925135C370E3646612BD84/druid@EXAMPLE.COM to druid@EXAMPLE.COM|HTTP/ctr-e138-1518143905142-237396-01-000003.hwx.site@EXAMPLE.COM
>>> KrbApReq: authenticate succeed.
Krb5Context setting peerSeqNumber to: 583816652
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
Krb5Context setting mySeqNumber to: 449475208
2018-04-24T22:46:17,812 DEBUG [qtp409957428-188] io.druid.security.kerberos.KerberosAuthenticator - Request [{http://ctr-e138-1518143905142-237396-01-000003.hwx.site:8082/druid/v2/}] user [{druid}] authenticated

b-slim · 2018-04-25T21:06:34Z

@jon-wei this is where the magic is happening io.druid.security.kerberos.DruidKerberosUtil#kerberosChallenge
as you can see the challenge is tight to the serverName hence can not be read by broker if it was originally generated to Router. I am not sure how you can get around this?

      // GSS name for server
      GSSName serverName = manager.createName("HTTP@" + server, GSSName.NT_HOSTBASED_SERVICE);
      // Create a GSSContext for authentication with the service.

jon-wei · 2018-04-25T21:33:41Z

@b-slim Can you try a test build where you put that escalated router -> broker path back in, and see if that fixes the issue? We can see where to go from there. I don't have a setup where I can reproduce the issue yet.

b-slim · 2018-04-25T21:52:45Z

@jon-wei i did that and it is not working, in fact that client does't try to authenticate at all. am still looking what is the best way to fix this currently am thinking about attaching the authToken since it is tight to the user id and every node can decrypt it if they share the same secret ... will keep you posted am still hacking around.

b-slim · 2018-04-26T01:36:57Z

@jon-wei and @gianm thanks for feedback sent PR

Remove Escalator jetty http client escalation method

2e48d6d

jon-wei added Bug Security Area - Router labels Feb 2, 2018

jon-wei added this to the 0.12.0 milestone Feb 2, 2018

gianm approved these changes Feb 2, 2018

View reviewed changes

himanshug merged commit c9e7c0a into apache:master Feb 2, 2018

jon-wei added Incompatible Release Notes labels Feb 5, 2018

jon-wei added a commit to jon-wei/druid that referenced this pull request Feb 5, 2018

Remove Escalator jetty http client escalation method (apache#5322)

8ae644a

jon-wei mentioned this pull request Feb 5, 2018

[Backport] Remove Escalator jetty http client escalation method (#5322) #5348

Merged

jon-wei added a commit that referenced this pull request Feb 5, 2018

Remove Escalator jetty http client escalation method (#5322) (#5348)

b5e1108

gianm pushed a commit to implydata/druid-public that referenced this pull request Feb 6, 2018

Remove Escalator jetty http client escalation method (apache#5322)

929cd01

jon-wei mentioned this pull request Feb 15, 2018

[WIP] Druid 0.12.0 release notes #5211

Closed

b-slim mentioned this pull request Apr 26, 2018

Kerberos Spnego Authentication Router Issue #5706

Merged

Conversation

jon-wei commented Feb 2, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gianm left a comment

Choose a reason for hiding this comment

Uh oh!

b-slim commented Apr 21, 2018

Uh oh!

pjain1 commented Apr 21, 2018 via email

Uh oh!

b-slim commented Apr 23, 2018

Uh oh!

gianm commented Apr 23, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jon-wei commented Apr 23, 2018

Uh oh!

jon-wei commented Apr 23, 2018

Uh oh!

b-slim commented Apr 23, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

b-slim commented Apr 23, 2018

Uh oh!

b-slim commented Apr 23, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jon-wei commented Apr 23, 2018

Uh oh!

b-slim commented Apr 23, 2018

Uh oh!

jon-wei commented Apr 24, 2018

Uh oh!

b-slim commented Apr 24, 2018

Uh oh!

jon-wei commented Apr 24, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

b-slim commented Apr 24, 2018

Uh oh!

jon-wei commented Apr 24, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jon-wei commented Apr 24, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gianm commented Apr 24, 2018

Uh oh!

b-slim commented Apr 24, 2018

Uh oh!

gianm commented Apr 24, 2018

Uh oh!

jon-wei commented Apr 24, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jon-wei commented Apr 24, 2018

Uh oh!

jon-wei commented Apr 24, 2018

Uh oh!

b-slim commented Apr 24, 2018

Uh oh!

b-slim commented Apr 24, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

b-slim commented Apr 24, 2018

Uh oh!

jon-wei commented Apr 24, 2018

Uh oh!

b-slim commented Apr 24, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jon-wei commented Apr 24, 2018

Uh oh!

b-slim commented Apr 24, 2018

Uh oh!

b-slim commented Apr 24, 2018

Uh oh!

b-slim commented Apr 25, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

jon-wei commented Feb 2, 2018 •

edited

Loading

gianm commented Apr 23, 2018 •

edited

Loading

b-slim commented Apr 23, 2018 •

edited

Loading

b-slim commented Apr 23, 2018 •

edited

Loading

jon-wei commented Apr 24, 2018 •

edited

Loading

jon-wei commented Apr 24, 2018 •

edited

Loading

jon-wei commented Apr 24, 2018 •

edited

Loading

jon-wei commented Apr 24, 2018 •

edited

Loading

b-slim commented Apr 24, 2018 •

edited

Loading

b-slim commented Apr 24, 2018 •

edited

Loading

b-slim commented Apr 25, 2018 •

edited

Loading