[SPARK-29082][core] Skip delegation token generation if no credentials are available. #25805

vanzin · 2019-09-16T16:33:39Z

This situation can happen when an external system (e.g. Oozie) generates
delegation tokens for a Spark application. The Spark driver will then run
against secured services, have proper credentials (the tokens), but no
kerberos credentials. So trying to do things that requires a kerberos
credential fails.

Instead, if no kerberos credentials are detected, just skip the whole
delegation token code.

Tested with an application that simulates Oozie; fails before the fix,
passes with the fix. Also with other DT-related tests to make sure other
functionality keeps working.

…s are available. This situation can happen when an external system (e.g. Oozie) generates delegation tokens for a Spark application. The Spark driver will then run against secured services, have proper credentials (the tokens), but no kerberos credentials. So trying to do things that requires a kerberos credential fails. Instead, if no kerberos credentials are detected, just skip the whole delegation token code. Tested with an application that simulates Oozie; fails before the fix, passes with the fix. Also with other DT-related tests to make sure other functionality keeps working.

gaborgsomogyi

LGTM (pending tests).

SparkQA · 2019-09-16T22:26:09Z

Test build #110629 has finished for PR 25805 at commit e6efb06.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

Different scenario because the tests doesn't go through spark-submit, which logs in the user with the keytab. Just need to tweak the check so that a keytab is considered as "having kerberos creds".

SparkQA · 2019-09-17T01:25:40Z

Test build #110671 has finished for PR 25805 at commit 29b069d.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

dongjoon-hyun · 2019-09-17T02:56:05Z

Retest this please.

SparkQA · 2019-09-17T03:18:09Z

Test build #110723 has finished for PR 25805 at commit 29b069d.

This patch fails to generate documentation.
This patch merges cleanly.
This patch adds no public classes.

HyukjinKwon · 2019-09-17T06:51:42Z

Doc failure seems unrelated.

HyukjinKwon · 2019-09-17T06:51:47Z

retest this please

SparkQA · 2019-09-17T07:05:02Z

Test build #110744 has finished for PR 25805 at commit 29b069d.

This patch fails due to an unknown error code, -9.
This patch merges cleanly.
This patch adds no public classes.

gaborgsomogyi · 2019-09-17T11:25:48Z

[error] running /home/jenkins/workspace/SparkPullRequestBuilder@2/build/sbt -Phadoop-2.7 -Pkubernetes -Phive-thriftserver -Phadoop-cloud -Pkinesis-asl -Pyarn -Pspark-ganglia-lgpl -Phive -Pmesos test:package streaming-kinesis-asl-assembly/assembly ; process was terminated by signal 9

gaborgsomogyi · 2019-09-17T11:25:58Z

retest this please

SparkQA · 2019-09-17T13:07:01Z

Test build #110775 has finished for PR 25805 at commit 29b069d.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

gaborgsomogyi · 2019-09-17T13:35:28Z

Maybe it's related since No UGI and Proxy user tests are failing.

vanzin · 2019-09-17T16:16:48Z

Hmm, that's a new test IIRC, I'll take a look at what's going on.

vanzin · 2019-09-17T17:26:41Z

The Hive tests passed locally for me. Maybe just needed a merge with master...

vanzin · 2019-09-17T18:05:16Z

retest this please

SparkQA · 2019-09-17T18:59:04Z

Test build #110814 has finished for PR 25805 at commit c46d76d.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2019-09-17T18:59:27Z

Test build #110815 has finished for PR 25805 at commit c150238.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2019-09-17T19:15:10Z

retest this please

SparkQA · 2019-09-17T19:20:19Z

Test build #110820 has finished for PR 25805 at commit c150238.

This patch fails build dependency tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2019-09-17T19:22:31Z

retest this please

vanzin · 2019-09-17T19:22:47Z

(Looks like maven metadata is corrupt everywhere after Jenkins was restarted...)

SparkQA · 2019-09-17T22:11:38Z

Test build #110798 has finished for PR 25805 at commit c46d76d.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2019-09-17T22:15:47Z

Ack. Updated the wrong branch...

SparkQA · 2019-09-17T22:42:11Z

Test build #110823 has finished for PR 25805 at commit c150238.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

vanzin · 2019-09-18T20:29:33Z

Ok, no more comments, I'll merge to master.

dongjoon-hyun · 2019-09-19T20:35:49Z

Hi, All.
This seems to break our Jenkins one profile consistently. Could you guys take a look please?

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-3.2-jdk-11/

- zero-partition RDD *** FAILED ***
  java.io.IOException: Can't get Master Kerberos principal for use as renewer
  at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:134)
  at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:102)
  at org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:81)
  at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:216)
  at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:325)
  at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:205)
  at org.apache.spark.rdd.RDD.$anonfun$partitions$2(RDD.scala:256)
  at scala.Option.getOrElse(Option.scala:189)
  at org.apache.spark.rdd.RDD.partitions(RDD.scala:254)
  at org.apache.spark.rdd.MapPartitionsRDD.getPartitions(MapPartitionsRDD.scala:49)

After this PR, all run failed consistently since 433 ...

dongjoon-hyun · 2019-09-20T00:54:09Z

Sorry guys. Until now, I cannot find a solution for this. I'll revert this one to unblock our JDK11 monitoring. Please make another PR and test with [test-hadoop3.2][test-java11].

dongjoon-hyun · 2019-09-20T02:19:10Z

FYI, all the failures are recovered and the Jenkins is continuing to the next step. If this didn't hide the other failure during the outage, I guess the Jenkins will pass.

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-3.2-jdk-11/443/

srowen · 2019-09-20T14:15:24Z

Yeah, I found this caused tests to fail locally too, so it's at least not specific to Jenkins (or Java 11)

dongjoon-hyun · 2019-09-20T14:57:37Z

In addition to this, one another big issue is that one Python UT failure blocks spark-master-test-sbt-hadoop-2.7 for a long time until now.

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-sbt-hadoop-2.7/

gaborgsomogyi · 2019-09-20T16:16:10Z

Having a look but jdk11 has to be set up locally where I have some difficulty...

gaborgsomogyi · 2019-09-20T19:25:30Z

We've found the issue with the help of @squito so going to file a PR soon with the required change...

dongjoon-hyun · 2019-09-20T19:39:42Z

Great! Thank you, @gaborgsomogyi and @squito !

…s are available This PR is an enhanced version of #25805 so I've kept the original text. The problem with the original PR can be found in comment. This situation can happen when an external system (e.g. Oozie) generates delegation tokens for a Spark application. The Spark driver will then run against secured services, have proper credentials (the tokens), but no kerberos credentials. So trying to do things that requires a kerberos credential fails. Instead, if no kerberos credentials are detected, just skip the whole delegation token code. Tested with an application that simulates Oozie; fails before the fix, passes with the fix. Also with other DT-related tests to make sure other functionality keeps working. Closes #25901 from gaborgsomogyi/SPARK-29082. Authored-by: Gabor Somogyi <gabor.g.somogyi@gmail.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com>

gaborgsomogyi approved these changes Sep 16, 2019

View reviewed changes

Fix Kafka unit test.

29b069d

Different scenario because the tests doesn't go through spark-submit, which logs in the user with the keytab. Just need to tweak the check so that a keytab is considered as "having kerberos creds".

dongjoon-hyun added the SPARK CORE label Sep 17, 2019

Marcelo Vanzin added 2 commits September 17, 2019 09:36

Merge branch 'master' into SPARK-29082

b274593

Add a unit test.

c46d76d

Add bug ref.

c150238

vanzin force-pushed the SPARK-29082 branch from a6aca84 to c150238 Compare September 17, 2019 22:16

vanzin closed this in f32f16f Sep 18, 2019

gaborgsomogyi mentioned this pull request Sep 23, 2019

[SPARK-29082][CORE] Skip delegation token generation if no credentials are available #25901

Closed

vanzin deleted the SPARK-29082 branch October 23, 2019 15:54

[SPARK-29082][core] Skip delegation token generation if no credentials are available. #25805

[SPARK-29082][core] Skip delegation token generation if no credentials are available. #25805

Uh oh!

Conversation

vanzin commented Sep 16, 2019

Uh oh!

gaborgsomogyi left a comment

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Sep 16, 2019

Uh oh!

SparkQA commented Sep 17, 2019

Uh oh!

dongjoon-hyun commented Sep 17, 2019

Uh oh!

SparkQA commented Sep 17, 2019

Uh oh!

HyukjinKwon commented Sep 17, 2019

Uh oh!

HyukjinKwon commented Sep 17, 2019

Uh oh!

SparkQA commented Sep 17, 2019

Uh oh!

gaborgsomogyi commented Sep 17, 2019

Uh oh!

gaborgsomogyi commented Sep 17, 2019

Uh oh!

SparkQA commented Sep 17, 2019

Uh oh!

gaborgsomogyi commented Sep 17, 2019

Uh oh!

vanzin commented Sep 17, 2019

Uh oh!

vanzin commented Sep 17, 2019

Uh oh!

vanzin commented Sep 17, 2019

Uh oh!

SparkQA commented Sep 17, 2019

Uh oh!

SparkQA commented Sep 17, 2019

Uh oh!

vanzin commented Sep 17, 2019

Uh oh!

SparkQA commented Sep 17, 2019

Uh oh!

vanzin commented Sep 17, 2019

Uh oh!

vanzin commented Sep 17, 2019

Uh oh!

SparkQA commented Sep 17, 2019

Uh oh!

vanzin commented Sep 17, 2019

Uh oh!

SparkQA commented Sep 17, 2019

Uh oh!

vanzin commented Sep 18, 2019

Uh oh!

dongjoon-hyun commented Sep 19, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented Sep 20, 2019

Uh oh!

dongjoon-hyun commented Sep 20, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

srowen commented Sep 20, 2019

Uh oh!

dongjoon-hyun commented Sep 20, 2019

Uh oh!

gaborgsomogyi commented Sep 20, 2019

Uh oh!

gaborgsomogyi commented Sep 20, 2019

Uh oh!

dongjoon-hyun commented Sep 20, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

dongjoon-hyun commented Sep 19, 2019 •

edited

Loading

dongjoon-hyun commented Sep 20, 2019 •

edited

Loading