[SPARK-1825] Fixes cross-platform submit problem #899

zeodtr · 2014-05-28T04:56:37Z

This is my attempt to fix https://issues.apache.org/jira/browse/SPARK-1825.

Tested on Windows 7 and Hortonworks HDP 2.1 Sandbox, and it works.
Two more problems reported in SPARK-1825(SPARK_HOME, %HADOOP_MAPRED_HOME%) were gone, perhaps by other commits that took after rc5.

WARNING:
But this fix is Hadoop 2.4.0+ only, since it uses new APIs introduced by https://issues.apache.org/jira/browse/YARN-1824.
So, version checking may be needed, but my knowledge for the Spark source code is limited, so I don't know how to do it.

AmplabJenkins · 2014-05-28T04:57:58Z

Can one of the admins verify this patch?

srowen · 2014-05-28T09:15:36Z

yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala

As you say, this value only appeared in Hadoop 2.4.0:

http://hadoop.apache.org/docs/r2.4.0/api/org/apache/hadoop/yarn/api/ApplicationConstants.html
http://hadoop.apache.org/docs/r2.3.0/api/org/apache/hadoop/yarn/api/ApplicationConstants.html

File.pathSeparator should already be ":" vs ";", which is what you intend right?
http://docs.oracle.com/javase/7/docs/api/java/io/File.html#pathSeparator

I'm missing what this changes then. I understand that the char may vary on the client vs cluster, and that's why it's right to reference a symbolic constant, but these seem to be the same in that respect.

The value of ApplicationConstants.CLASS_PATH_SEPARATOR is "<CPS>" - neither ":" nor ";".
The point is that the separator will be chosen by the cluster(in my case, linux machine), rather than the client(in my case, Windows machine) if ApplicationConstants.CLASS_PATH_SEPARATOR is used.
That is, the server hadoop module will find "<CPS>" string in the path string and replace it with the real separator appropriate to its OS.
But current Spark 1.0 code 'hardcodes' the separator on the client side, by using File.pathSeparator. Then the Windows-style path string (that contains ';' which confuses the linux shell script interpreter) will be sent to the linux cluster, in my case.

BTW, maybe this is a 'yarn-client' mode-only problem.
I have not tested the 'yarn-cluster' mode.

Oh I get it. Because the client forms the path but sends it to the server, so it's produced and consumed in different places. The bad news is that you can't use this constant, but, I suppose you can literally specify <CPS>. But will that fail for Hadoop < 2.4?

I think it will fail for Hadoop < 2.4.
The hadoop server code that recognizes <CPS> was committed at Mar 17 2014, to resolve YARN-1824. The code is in org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch. (see this github link)
So, maybe the best solution that does not require Hadoop 2.4.0 is to build the environment variables on the cluster side. (I don't know how to do that - is it even possible?)

andrewor14 · 2014-09-03T19:23:58Z

ok to test

andrewor14 · 2014-09-03T19:24:12Z

@zeodtr Can you update the title to make it something more descriptive?

SparkQA · 2014-09-03T19:30:04Z

QA tests have started for PR 899 at commit b768fc6.

This patch merges cleanly.

SparkQA · 2014-09-03T20:54:09Z

QA tests have finished for PR 899 at commit b768fc6.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

zeodtr · 2014-09-04T03:18:56Z

@andrewor14 Updated the title.

andrewor14 · 2014-09-04T04:33:36Z

Sorry, could you update the title to use this format [SPARK-1825] Fixes cross platform...? We use a tool that parses the JIRA number to link Spark PRs with their associated issues. Thanks.

zeodtr · 2014-09-05T00:51:18Z

@andrewor14 Updated the title.

vanzin · 2014-09-05T16:58:59Z

@zeodtr does this compile with anything < hadoop 2.4? If it doesn't, this is a no-go.

andrewor14 · 2014-09-05T20:25:26Z

@zeodtr Thanks for updating the title. Just so I understand the issue, for HDP 2.1 on Windows we need these changes for Spark to run, is that correct? However, with this patch other hadoop versions < 2.4 won't even compile, so it seems that we need to do figure out which fields to call during runtime through reflection. When you have a chance can you fix this up so it's compatible across other hadoop versions as well?

andrewor14 · 2014-09-05T20:26:15Z

Also, I notice that this is opened against branch-1.0. It would be better if you could open it against the master branch so the latest Spark releases will also benefit from your changes.

SparkQA · 2014-09-05T23:46:30Z

Can one of the admins verify this patch?

andrewor14 · 2014-09-06T00:53:32Z

ok to test

SparkQA · 2014-09-06T01:43:29Z

QA tests have started for PR 899 at commit b768fc6.

This patch merges cleanly.

zeodtr · 2014-09-06T02:36:54Z

@andrewor14 @vanzin As I already mentioned in another comment, This won't compile on Hadoop < 2.4. So I suggested another method, but currently I have not enough knowledge to implement the idea.
Sorry if this PR confuses you.

SparkQA · 2014-09-06T03:10:05Z

QA tests have finished for PR 899 at commit b768fc6.

This patch passes unit tests.
This patch merges cleanly.
This patch adds no public classes.

andrewor14 · 2014-09-06T18:11:55Z

@zeodtr No worries. We can build on top of your patch to make this work for hadoop versions < 2.4. Thanks for digging through this code.

andrewor14 · 2015-02-06T19:54:45Z

Hey @zeodtr I believe this is fixed in #3943 would you mind closing this PR?

zeodtr · 2015-02-08T11:29:38Z

@andrewor14 Ok, Thanks for fixing!

andrewor14 · 2015-02-08T18:37:46Z

I didn't fix it, @tsudukim did, so thank him :)

…for Spark 3.1.2 (apache#899)

…OOM (#898) * Fix Driver OOM * Fix * Fix * Fix (#899) * Update DynamicDataPruningSuite.scala * Update DynamicDataPruningSuite.scala

…for Spark 3.1.2 (apache#899)

fixes https://issues.apache.org/jira/browse/SPARK-1825

b768fc6

srowen reviewed May 28, 2014
View reviewed changes

zeodtr changed the title ~~fixes https://issues.apache.org/jira/browse/SPARK-1825~~ fixes crossplatform submit problem (see https://issues.apache.org/jira/browse/SPARK-1825) Sep 4, 2014

zeodtr changed the title ~~fixes crossplatform submit problem (see https://issues.apache.org/jira/browse/SPARK-1825)~~ [SPARK-1825] Fixes cross-platform submit problem Sep 5, 2014

zeodtr closed this Feb 8, 2015

agirish pushed a commit to HPEEzmeral/apache-spark that referenced this pull request May 5, 2022

An attempt to hide progressbar in output of whitesource scan on CI / …

6b9d34d

…for Spark 3.1.2 (apache#899)

wangyum added a commit that referenced this pull request May 26, 2023

[CARMEL-5902] Take dynamicPruningMaxInsetNum of rows to avoid Driver …

0825e22

…OOM (#898) * Fix Driver OOM * Fix * Fix * Fix (#899) * Update DynamicDataPruningSuite.scala * Update DynamicDataPruningSuite.scala

udaynpusa pushed a commit to mapr/spark that referenced this pull request Jan 30, 2024

An attempt to hide progressbar in output of whitesource scan on CI / …

d0a92b6

…for Spark 3.1.2 (apache#899)

mapr-devops pushed a commit to mapr/spark that referenced this pull request May 8, 2025

An attempt to hide progressbar in output of whitesource scan on CI / …

8cd4a3f

…for Spark 3.1.2 (apache#899)

[SPARK-1825] Fixes cross-platform submit problem #899

[SPARK-1825] Fixes cross-platform submit problem #899

Uh oh!

Conversation

zeodtr commented May 28, 2014

Uh oh!

AmplabJenkins commented May 28, 2014

Uh oh!

srowen May 28, 2014

Choose a reason for hiding this comment

Uh oh!

zeodtr May 28, 2014

Choose a reason for hiding this comment

Uh oh!

zeodtr May 28, 2014

Choose a reason for hiding this comment

Uh oh!

srowen May 28, 2014

Choose a reason for hiding this comment

Uh oh!

zeodtr May 29, 2014

Choose a reason for hiding this comment

Uh oh!

andrewor14 commented Sep 3, 2014

Uh oh!

andrewor14 commented Sep 3, 2014

Uh oh!

SparkQA commented Sep 3, 2014

Uh oh!

SparkQA commented Sep 3, 2014

Uh oh!

zeodtr commented Sep 4, 2014

Uh oh!

andrewor14 commented Sep 4, 2014

Uh oh!

zeodtr commented Sep 5, 2014

Uh oh!

vanzin commented Sep 5, 2014

Uh oh!

andrewor14 commented Sep 5, 2014

Uh oh!

andrewor14 commented Sep 5, 2014

Uh oh!

SparkQA commented Sep 5, 2014

Uh oh!

andrewor14 commented Sep 6, 2014

Uh oh!

SparkQA commented Sep 6, 2014

Uh oh!

zeodtr commented Sep 6, 2014

Uh oh!

SparkQA commented Sep 6, 2014

Uh oh!

andrewor14 commented Sep 6, 2014

Uh oh!

andrewor14 commented Feb 6, 2015

Uh oh!

zeodtr commented Feb 8, 2015

Uh oh!

andrewor14 commented Feb 8, 2015

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants