Skip to content

Conversation

@maropu
Copy link
Member

@maropu maropu commented Sep 27, 2016

What changes were proposed in this pull request?

This pr is just to fix the document of spark-kinesis-integration.
Since SPARK-17418 prevented all the kinesis stuffs (including kinesis example code)
from publishing, bin/run-example streaming.KinesisWordCountASL and bin/run-example streaming.JavaKinesisWordCountASL does not work.
Instead, it fetches the kinesis jar from the Spark Package.

@SparkQA
Copy link

SparkQA commented Sep 27, 2016

Test build #65962 has finished for PR 15260 at commit ce4640e.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@srowen
Copy link
Member

srowen commented Sep 28, 2016

Although it should still work (?) to build with -Pkinesis-asl I think, because the assembly still exists it's just not published, I do think we should make this doc change, because it reflects how users would really use this with a Spark distribution.

@maropu
Copy link
Member Author

maropu commented Sep 28, 2016

In master, I tried;

$  mvn -Pkinesis-asl -DskipTests package; ./bin/run-example streaming.KinesisWordCountASL
java.lang.ClassNotFoundException: org.apache.spark.examples.streaming.KinesisWordCountASL
        at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
        at java.security.AccessController.doPrivileged(Native Method

Also, in the spark-2.0 branch, I hit the same issue.
IIUC classpath does not include the kinesis jar in external/kinesis-asl-assembly/target when running this example.

@maropu
Copy link
Member Author

maropu commented Sep 28, 2016

Aha, I see. However, some users possibly think they'd just like to download compiled Spark jars and run the kinesis example for quick tests because compiling the whole spark package takes much time.
In other examples (e.g., SparkPi), since spark compiled binaries in the download site bundle them, we can download them and just run the examples. But, we currently cannot run the kinesis example because they donot bundle it now.

So, how about staying this current doc as it is and appending a new direction like this pr?
Also, in a follow-up pr, it is okay to me that I'll fix the run-example script so that it can add external/kinesis-asl-assembly/target in classpath.

@srowen
Copy link
Member

srowen commented Sep 28, 2016

I think your current change sounds fine. It shows how people should use this now.

@maropu
Copy link
Member Author

maropu commented Sep 28, 2016

yea, thx,
Also, should we add external/kinesis-asl-assembly/target in classpath so that we can run $ mvn -Pkinesis-asl -DskipTests package; ./bin/run-example streaming.KinesisWordCountASL?

@srowen
Copy link
Member

srowen commented Sep 29, 2016

Oh, add it to run-example? I see why you say that although it won't be necessary for people using it the 'right way' and the docs will now show that way.

@maropu
Copy link
Member Author

maropu commented Sep 29, 2016

yea, okay.

@srowen
Copy link
Member

srowen commented Sep 29, 2016

Merged to master/2.0

asfgit pushed a commit that referenced this pull request Sep 29, 2016
## What changes were proposed in this pull request?
This pr is just to fix the document of `spark-kinesis-integration`.
Since `SPARK-17418` prevented all the kinesis stuffs (including kinesis example code)
from publishing,  `bin/run-example streaming.KinesisWordCountASL` and `bin/run-example streaming.JavaKinesisWordCountASL` does not work.
Instead, it fetches the kinesis jar from the Spark Package.

Author: Takeshi YAMAMURO <linguin.m.s@gmail.com>

Closes #15260 from maropu/DocFixKinesis.

(cherry picked from commit b2e9731)
Signed-off-by: Sean Owen <sowen@cloudera.com>
@asfgit asfgit closed this in b2e9731 Sep 29, 2016
@maropu maropu deleted the DocFixKinesis branch July 5, 2017 11:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants