Skip to content

MINOR: Update stream documentation #8622

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
bbejeck merged 5 commits into from
May 19, 2020
Merged

MINOR: Update stream documentation #8622

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
e0cab36
update stream documentation
showuon May 6, 2020
c0e3a97
Merge branch 'trunk' of https://github.com/apache/kafka into stream_d…
showuon May 14, 2020
3165ae7
fix broken links and HTML format error
showuon May 14, 2020
460768e
revert back to version 2.3 since since 2.4, we have cooperative rebal…
showuon May 19, 2020
a3accf6
addressing reviewer's comments
showuon May 19, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/streams/architecture.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ <h3><a id="streams_architecture_state" href="#streams_architecture_state">Local

<p>
Kafka Streams provides so-called <b>state stores</b>, which can be used by stream processing applications to store and query data,
which is an important capability when implementing stateful operations. The <a href="/{{version}}/documentation/streams/developer-guide#streams_dsl">Kafka Streams DSL</a>, for example, automatically creates
which is an important capability when implementing stateful operations. The <a href="/{{version}}/documentation/streams/developer-guide/dsl-api.html">Kafka Streams DSL</a>, for example, automatically creates
and manages such state stores when you are calling stateful operators such as <code>join()</code> or <code>aggregate()</code>, or when you are windowing a stream.
</p>

Expand Down
2 changes: 1 addition & 1 deletion docs/streams/developer-guide/dsl-api.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3738,7 +3738,7 @@ <h5><a class="toc-backref" href="#id34">KTable-KTable Foreign-Key
</div>
</div>
<div class="section" id="testing-a-streams-app">
<a class="headerlink" href="#testing-a-streams-app" title="Permalink to this headline"><h2>Testing a Streams application</h2></a>
<h2><a class="headerlink" href="#testing-a-streams-app" title="Permalink to this headline">Testing a Streams application</a></h2>
Copy link
Copy Markdown
Member Author

@showuon showuon May 14, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, the TESTING A STREAMS APPLICATION header is in blue, which is different from others. Turns out that it's because we put <h2> tag inside <a>, and cause to apply the wrong css format.
image

Kafka Streams comes with a <code>test-utils</code> module to help you test your application <a href="testing.html">here</a>.
</div>
</div>
Expand Down
2 changes: 1 addition & 1 deletion docs/streams/developer-guide/running-app.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@
<span id="streams-developer-guide-execution"></span><h1>Running Streams Applications<a class="headerlink" href="#running-streams-applications" title="Permalink to this headline"></a></h1>
<p>You can run Java applications that use the Kafka Streams library without any additional configuration or requirements. Kafka Streams
also provides the ability to receive notification of the various states of the application. The ability to monitor the runtime
status is discussed in <a class="reference internal" href="../monitoring.html#streams-monitoring"><span class="std std-ref">the monitoring guide</span></a>.</p>
status is discussed in <a class="reference internal" href="/documentation/#kafka_streams_monitoring"><span class="std std-ref">the monitoring guide</span></a>.</p>
Copy link
Copy Markdown
Member Author

@showuon showuon May 14, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fix the broken link.

<div class="contents local topic" id="table-of-contents">
<p class="topic-title first"><b>Table of Contents</b></p>
<ul class="simple">
Expand Down
4 changes: 2 additions & 2 deletions docs/streams/developer-guide/write-streams.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,7 @@ <h2>Using Kafka Streams within your application code<a class="headerlink" href="
</div>
<p>If there are other instances of this stream processing application running elsewhere (e.g., on another machine), Kafka
Streams transparently re-assigns tasks from the existing instances to the new instance that you just started.
For more information, see <a class="reference internal" href="../architecture.html#streams_architecture_tasks"><span class="std std-ref">Stream Partitions and Tasks</span></a> and <a class="reference internal" href="../architecture.html#streams-architecture-threads"><span class="std std-ref">Threading Model</span></a>.</p>
For more information, see <a class="reference internal" href="../architecture.html#streams_architecture_tasks"><span class="std std-ref">Stream Partitions and Tasks</span></a> and <a class="reference internal" href="../architecture.html#streams_architecture_threads"><span class="std std-ref">Threading Model</span></a>.</p>
Copy link
Copy Markdown
Member Author

@showuon showuon May 14, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix the broken link from wrong href="../architecture.html#streams-architecture-threads" to href="../architecture.html#streams_architecture_threads" (the one with underscore)

<p>To catch any unexpected exceptions, you can set an <code class="docutils literal"><span class="pre">java.lang.Thread.UncaughtExceptionHandler</span></code> before you start the
application. This handler is called whenever a stream thread is terminated by an unexpected exception:</p>
<div class="highlight-java"><div class="highlight"><pre><span></span><span class="c1">// Java 8+, using lambda expressions</span>
Expand Down Expand Up @@ -208,7 +208,7 @@ <h2>Using Kafka Streams within your application code<a class="headerlink" href="
</div>

<div class="section" id="testing-a-streams-app">
<a class="headerlink" href="#testing-a-streams-app" title="Permalink to this headline"><h2>Testing a Streams application</a></h2>
<h2><a class="headerlink" href="#testing-a-streams-app" title="Permalink to this headline">Testing a Streams application</a></h2>
Copy link
Copy Markdown
Member Author

@showuon showuon May 14, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

wrong HTML formatting. The starting/ending <h2> tag and <a> tag are mixed places.

Kafka Streams comes with a <code>test-utils</code> module to help you test your application <a href="testing.html">here</a>.
</div>
</div>
Expand Down
6 changes: 3 additions & 3 deletions docs/streams/upgrade-guide.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ <h1>Upgrade Guide and API Changes</h1>
</div>

<p>
Upgrading from any older version to {{fullDotVersion}} is possible: you will need to do two rolling bounces, where during the first rolling bounce phase you set the config <code>upgrade.from="older version"</code>
Upgrading from any older version to {{fullDotVersion}} is possible: if upgrading from 2.3 or below, you will need to do two rolling bounces, where during the first rolling bounce phase you set the config <code>upgrade.from="older version"</code>
(possible values are <code>"0.10.0" - "2.3"</code>) and during the second you remove it. This is required to safely upgrade to the new cooperative rebalancing protocol of the embedded consumer. Note that you will remain using the old eager
Copy link
Copy Markdown
Member

@ableegoldman ableegoldman May 19, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@showuon Can we clarify that you only need to do this if you're upgrading from 2.3 or below? I know this seems implied by the fact that the config's possible values stop at 2.3 but there are always creative interpretations of seemingly obvious things 🙂

Copy link
Copy Markdown
Member Author

@showuon showuon May 19, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hi @ableegoldman , After reading the whole paragraph again, I think you're right.

(possible values are "0.10.0" - "2.3") and during the second you remove it. ....
This is required to safely upgrade to the new cooperative rebalancing protocol of the embedded consumer. ....
you can safely switch over to cooperative at any time once the entire group is on 2.4+ by removing the config value and bouncing.

So, Because we explicitly said since 2.4+, there'll be cooperative rebalancing protocol available, I think here we keep it as 2.3 is fine and correct.

Or do you have any other suggestion?
Thank you.

Copy link
Copy Markdown
Member

@ableegoldman ableegoldman May 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we just add a small qualifier in the first line?
you will need to do two rolling bounces --> if upgrading from 2.3 or below, you will need to do two rolling bounces

Copy link
Copy Markdown
Member Author

@showuon showuon May 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for suggestion, @ableegoldman , it makes it more clear! I've updated in this commit a3accf6. Thank you.

Copy link
Copy Markdown
Member

@ableegoldman ableegoldman May 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome, thank you!

Copy link
Copy Markdown
Member Author

@showuon showuon May 19, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've also updated in the kafka-site repo: apache/kafka-site@513a820. Thank you.

rebalancing protocol if you skip or delay the second rolling bounce, but you can safely switch over to cooperative at any time once the entire group is on 2.4+ by removing the config value and bouncing. For more details please refer to
<a href="https://cwiki.apache.org/confluence/x/vAclBg">KIP-429</a>:
Expand Down Expand Up @@ -100,7 +100,7 @@ <h3><a id="streams_api_changes_250" href="#streams_api_changes_250">Streams API
We add a new <code>cogroup()</code> operator (via <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-150+-+Kafka-Streams+Cogroup">KIP-150</a>>)
that allows to aggregate multiple streams in a single operation.
Cogrouped streams can also be windowed before they are aggregated.
We refer to the <a href="/{{version}}/documentation/streams/developer-guide/dsl-api.html">developer guide</a> for more details.
Please refer to the <a href="/{{version}}/documentation/streams/developer-guide/dsl-api.html">developer guide</a> for more details.
Copy link
Copy Markdown
Member Author

@showuon showuon May 6, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Rephrase this sentence.
Before: We refer to the developer guide for more details.
After: Please refer to the developer guide for more details.

</p>
<p>
We added a new <code>KStream.toTable()</code> API to translate an input event stream into a changelog stream as per
Expand Down Expand Up @@ -711,7 +711,7 @@ <h3><a id="streams_api_changes_100" href="#streams_api_changes_100">Streams API
If you are monitoring on task level or processor-node / state store level Streams metrics, please note that the metrics sensor name and hierarchy was changed:
The task ids, store names and processor names are no longer in the sensor metrics names, but instead are added as tags of the sensors to achieve consistent metrics hierarchy.
As a result you may need to make corresponding code changes on your metrics reporting and monitoring tools when upgrading to 1.0.0.
Detailed metrics sensor can be found in the <a href="#kafka_streams_monitoring">Streams Monitoring</a> section.
Detailed metrics sensor can be found in the <a href="/{{version}}/documentation/#kafka_streams_monitoring">Streams Monitoring</a> section.
Copy link
Copy Markdown
Member Author

@showuon showuon May 6, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#kafka_streams_monitoring is located in /documentation, not the current path (i.e. documentation/streams/upgrade-guide)

</p>

<p>
Expand Down