Skip to content

[SPARK-18461][DOCS][StructuredStreaming] Added more information about monitoring streaming queries #15897

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
tdas wants to merge 2 commits into from
Closed

[SPARK-18461][DOCS][StructuredStreaming] Added more information about monitoring streaming queries #15897

tdas wants to merge 2 commits into from

Conversation

@tdas
Copy link
Contributor

@tdas tdas commented Nov 16, 2016 •
edited
Loading

What changes were proposed in this pull request?

screen shot 2016-11-15 at 6 27 32 pm

screen shot 2016-11-15 at 6 27 45 pm

@tdas
Copy link
Contributor Author

tdas commented Nov 16, 2016

@zsxwing @marmbrus

@tdas tdas changed the title [SPARK-18461][DOCS][Structured Streaming] Added more information about monitoring streaming queries [SPARK-18461][DOCS][StructuredStreaming] Added more information about monitoring streaming queries Nov 16, 2016
@SparkQA
Copy link

SparkQA commented Nov 16, 2016

Test build #68686 has finished for PR 15897 at commit 4958caf.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Nov 16, 2016

Test build #68687 has finished for PR 15897 at commit 1348470.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@marmbrus
Copy link
Contributor

marmbrus commented Nov 16, 2016

Thanks, merging to master and 2.1.

@asfgit asfgit closed this in bb6cdfd Nov 16, 2016
asfgit pushed a commit that referenced this pull request Nov 16, 2016
@tdas @marmbrus
[SPARK-18461][DOCS][STRUCTUREDSTREAMING] Added more information about…
3d4756d 
… monitoring streaming queries

## What changes were proposed in this pull request?
<img width="941" alt="screen shot 2016-11-15 at 6 27 32 pm" src="https://cloud.githubusercontent.com/assets/663212/20332521/4190b858-ab61-11e6-93a6-4bdc05105ed9.png">
<img width="940" alt="screen shot 2016-11-15 at 6 27 45 pm" src="https://cloud.githubusercontent.com/assets/663212/20332525/44a0d01e-ab61-11e6-8668-47f925490d4f.png">

Author: Tathagata Das <tathagata.das1565@gmail.com>

Closes #15897 from tdas/SPARK-18461.

(cherry picked from commit bb6cdfd)
Signed-off-by: Michael Armbrust <michael@databricks.com>
zsxwing
zsxwing reviewed Nov 16, 2016
View reviewed changes
docs/structured-streaming-programming-guide.md
{% highlight java %}
StreamingQuery query = ...

System.out.println(query.status);
Copy link
Member

@zsxwing zsxwing Nov 16, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: query.status()

docs/structured-streaming-programming-guide.md
{% highlight python %}
query = ... // a StreamingQuery

print(query.status)
Copy link
Member

@zsxwing zsxwing Nov 16, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there is no status method in Python now... We need to add it.

docs/structured-streaming-programming-guide.md
spark.streams.addListener(new StreamingQueryListener() {

@Overrides void onQueryStarted(QueryStartedEvent queryStarted) {
System.out.println("Query started: " + queryTerminated.queryStatus.name);
Copy link
Member

@zsxwing zsxwing Nov 16, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: queryTerminated.queryStatus.name -> queryTerminated.queryStatus().name()

docs/structured-streaming-programming-guide.md
System.out.println("Query started: " + queryTerminated.queryStatus.name);
}
@Overrides void onQueryTerminated(QueryTerminatedEvent queryTerminated) {
System.out.println("Query terminated: " + queryTerminated.queryStatus.name);
Copy link
Member

@zsxwing zsxwing Nov 16, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: queryTerminated.queryStatus.name -> queryTerminated.queryStatus().name()

docs/structured-streaming-programming-guide.md
System.out.println("Query terminated: " + queryTerminated.queryStatus.name);
}
@Overrides void onQueryProgress(QueryProgressEvent queryProgress) {
System.out.println("Query made progress: " + queryProgress.queryStatus);
Copy link
Member

@zsxwing zsxwing Nov 16, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: queryTerminated.queryStatus.name -> queryTerminated.queryStatus().name()

docs/structured-streaming-programming-guide.md

spark.streams.addListener(new StreamingQueryListener() {

@Overrides void onQueryStarted(QueryStartedEvent queryStarted) {
Copy link
Member

@zsxwing zsxwing Nov 16, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: @ Overrides -> @Override

docs/structured-streaming-programming-guide.md
@Overrides void onQueryStarted(QueryStartedEvent queryStarted) {
System.out.println("Query started: " + queryTerminated.queryStatus.name);
}
@Overrides void onQueryTerminated(QueryTerminatedEvent queryTerminated) {
Copy link
Member

@zsxwing zsxwing Nov 16, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: @ Overrides -> @Override

docs/structured-streaming-programming-guide.md
@Overrides void onQueryTerminated(QueryTerminatedEvent queryTerminated) {
System.out.println("Query terminated: " + queryTerminated.queryStatus.name);
}
@Overrides void onQueryProgress(QueryProgressEvent queryProgress) {
Copy link
Member

@zsxwing zsxwing Nov 16, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: @ Overrides -> @Override

docs/structured-streaming-programming-guide.md
{% highlight java %}
SparkSession spark = ...

spark.streams.addListener(new StreamingQueryListener() {
Copy link
Member

@zsxwing zsxwing Nov 16, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: spark.streams -> spark.streams()

@tdas
Copy link
Contributor Author

tdas commented Nov 16, 2016

I am making a PR to add status to python. I will fix these doc stuff in that.

uzadude pushed a commit to uzadude/spark that referenced this pull request Jan 27, 2017
@tdas @uzadude
[SPARK-18461][DOCS][STRUCTUREDSTREAMING] Added more information about…
351f8b8 
… monitoring streaming queries

## What changes were proposed in this pull request?
<img width="941" alt="screen shot 2016-11-15 at 6 27 32 pm" src="https://cloud.githubusercontent.com/assets/663212/20332521/4190b858-ab61-11e6-93a6-4bdc05105ed9.png">
<img width="940" alt="screen shot 2016-11-15 at 6 27 45 pm" src="https://cloud.githubusercontent.com/assets/663212/20332525/44a0d01e-ab61-11e6-8668-47f925490d4f.png">

Author: Tathagata Das <tathagata.das1565@gmail.com>

Closes apache#15897 from tdas/SPARK-18461.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zsxwing zsxwing zsxwing left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@tdas @SparkQA @marmbrus @zsxwing