Merging Apache trunk to Microsoft trunk#1
Merged
Conversation
Three main tests: 1. Setup: Producer (0.8) → Kafka Cluster → Consumer (0.8) First rolling bounce: Set inter.broker.protocol.version = 0.8 and message.format.version = 0.8 Second rolling bonus, use latest (default) inter.broker.protocol.version and message.format.version 2. Setup: Producer (0.9) → Kafka Cluster → Consumer (0.9) First rolling bounce: Set inter.broker.protocol.version = 0.9 and message.format.version = 0.9 Second rolling bonus, use latest (default) inter.broker.protocol.version and message.format.version 3. Setup: Producer (0.9) → Kafka Cluster → Consumer (0.9) First rolling bounce: Set inter.broker.protocol.version = 0.9 and message.format.version = 0.9 Second rolling bonus: use inter.broker.protocol.version = 0.10 and message.format.version = 0.9 Plus couple of variations of these tests using old/new consumer and no compression / snappy compression. Author: Anna Povzner <anna@confluent.io> Reviewers: Ewen Cheslack-Postava <ewen@confluent.io> Closes #980 from apovzner/kafka-3201-02
This patch reuse max.in.flight.request.per.connection. When it equals to one, we take it as user wants order protection. The current approach is make sure there is only one batch per partition on the fly. Author: Jiangjie Qin <becket.qin@gmail.com> Reviewers: Aditya Auradkar <aauradkar@linkedin.com>, Jason Gustafson <jason@confluent.io>, Grant Henke <granthenke@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Joel Koshy <jjkoshy.w@gmail.com>, Jun Rao <junrao@gmail.com> Closes #857 from becketqin/KAFKA-3197
…s used. Author: Jiangjie Qin <becket.qin@gmail.com> Reviewers: Ismael Juma <ismael@juma.me.uk>, Jun Rao <junrao@gmail.com> Closes #1023 from becketqin/KAFKA-3343
Per discussion with guozhangwang, `ignore` failing streams system tests until fix for KAFKA-3354 is checked in. Author: Geoff Anderson <geoff@confluent.io> Reviewers: Guozhang Wang Closes #1031 from granders/ignore-streams-systest
Remove test cases testInvalidDefaultRange() and testInvalidDefaultString(). Defaults if not overridden will get checked on parse. Testing the defaults is unnecessary. This allows you to set that a parameter is required while setting a validator for that parameter. Added a test case testNullDefaultWithValidator that allows a null default with a validator for certain strings. Author: Jeremy Custenborder <jcustenborder@gmail.com> Reviewers: Grant Henke <granthenke@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io> Closes #936 from jcustenborder/KAFKA-3237
… for block.on.buffer.full default value is "false" and not "true" See: https://stackoverflow.com/questions/35578519/kafka-block-on-buffer-full-default-value and https://github.com/apache/kafka/blob/d5b43b19bb06e9cdc606312c8bcf87ed267daf44/clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java#L232 Author: mjsax <mjsax@apache.org> Reviewers: Ismael Juma <ismael@juma.me.uk>, Grant Henke <granthenke@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io> Closes #954 from mjsax/hotfix-docu
JAAS configuration may be set using other methods and hence the check for System property doesn't always match where the actual configuration used by Kafka is loaded from. Author: Rajini Sivaram <rajinisivaram@googlemail.com> Reviewers: Ismael Juma <ismael@juma.me.uk>, Sriharsha Chintalapani <harsha@hortonworks.com>, Flavio Junqueira <fpj@apache.org>, Ewen Cheslack-Postava <ewen@confluent.io> Closes #967 from rajinisivaram/KAFKA-3279
When invoking `gradle` on a recent version, it updates `gradlew.bat` to fix a typo. It's an annoyance at development time as it causes a diff on whatever branch one is working on. Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Grant Henke <granthenke@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io> Closes #1034 from ijuma/update-gradlew.bat
By using `getHostString` (introduced in Java 7) instead of `getHostName`. Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Jason Gustafson, Grant Henke Closes #1030 from ijuma/kafka-3352-avoid-dns-reverse-look-ups
Scala 2.11.8 has been released with a number of bug fixes: * http://www.scala-lang.org/news/2.11.8/ There are a few important collection fixes: * https://issues.scala-lang.org/browse/SI-9497 * scala/scala#4714 * scala/scala#4693 And also some pattern matcher fixes. Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Grant Henke, Ashish Singh Closes #1032 from ijuma/kafka-3357-update-to-scala-2.11.8
- Moves all generated docs under /docs/generated - Generates docs for Protocol, Errors, and ApiKeys - Adds new protocol.html page Author: Grant Henke <granthenke@gmail.com> Reviewers: Gwen Shapira Closes #970 from granthenke/protocol-doc-wip
Author: Grant Henke <granthenke@gmail.com> Reviewers: Gwen Shapira Closes #1037 from granthenke/protocol-minor
Author: Ashish Singh <asingh@cloudera.com> Reviewers: Ewen Cheslack-Postava <ewen@confluent.io> Closes #1024 from SinghAsDev/KAFKA-3344
Because protocol.html is going to be in its own page it needs the header and footer included. Author: Grant Henke <granthenke@gmail.com> Reviewers: Gwen Shapira Closes #1043 from granthenke/protocol-docs-style
… format … format Author: Gwen Shapira <cshapi@gmail.com> Reviewers: Ewen Cheslack-Postava <ewen@confluent.io> Closes #1039 from gwenshap/minor-consumer-groups
Author: Jason Gustafson <jason@confluent.io> Reviewers: Ismael Juma Closes #1036 from hachikuji/KAFKA-3318
AWS instance size increased to m3.xlarge to allow all system tests to pass. ijuma ewencp have a look please. Author: Eno Thereska <eno.thereska@gmail.com> Reviewers: Gwen Shapira Closes #1046 from enothereska/minor-aws
Author: Guozhang Wang <wangguoz@gmail.com> Reviewers: Yasuhiro Matsuda <yasuhiro@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io> Closes #1047 from guozhangwang/KInternal
Author: Manikumar reddy O <manikumar.reddy@gmail.com> Reviewers: Ismael Juma, Guozhang Wang Closes #1040 from omkreddy/MINOR-DOC
Author: Jason Gustafson <jason@confluent.io> Author: Ismael Juma <ismael@juma.me.uk> Author: hachikuji <jason@confluent.io> Reviewers: Grant Henke, Ismael Juma, Gwen Shapira, Flavio Junquiera Closes #988 from hachikuji/KAFKA-2073
… follower Author: Jiangjie Qin <becket.qin@gmail.com> Reviewers: Aditya Auradkar <aauradkar@linkedin.com>, Ismael Juma <ismael@juma.me.uk>, Joel Koshy <jjkoshy.w@gmail.com>, Jun Rao <junrao@gmail.com>, Guozhang Wang <wangguoz@gmail.com> Closes #1018 from becketqin/KAFKA-2960
Author: Ashish Singh <asingh@cloudera.com> Reviewers: Gwen Shapira Closes #1048 from SinghAsDev/KAFKA-3380
This fix was suggested by Maciek Makowski, who also reported the problem. Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Guozhang Wang <wangguoz@gmail.com> Closes #1029 from ijuma/KAFKA-3047-log-append-can-corrupt-the-log
Author: Manikumar reddy O <manikumar.reddy@gmail.com> Reviewers: Ismael Juma <ismael@juma.me.uk>, Sriharsha Chintalapani <mail@harsha.io> Closes #1054 from omkreddy/KAFKA-2551
guozhangwang Author: Yasuhiro Matsuda <yasuhiro@confluent.io> Reviewers: Guozhang Wang <wangguoz@gmail.com> Closes #1062 from ymatsuda/k3395
…d final for object used in sychronization guozhangwang Very minor cleanup. Author: Liquan Pei <liquanpei@gmail.com> Reviewers: Guozhang Wang <wangguoz@gmail.com> Closes #1063 from Ishiihara/minor-cleanup
Author: Jiangjie Qin <becket.qin@gmail.com> Reviewers: Gwen Shapira Closes #1049 from becketqin/KAFKA-3373
…ompiler settings * Fix and suppress number of unchecked warnings (except for Kafka Streams) * Add `SafeVarargs` annotation to fix warnings * Suppress unfixable deprecation warnings * Replace deprecated by non-deprecated usage where possible * Avoid reflective calls via structural types in Scala * Tweak compiler settings for scalac and javac Once we drop Java 7 and Scala 2.10, we can tweak the compiler settings further so that they warn us about more things. Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Grant Henke, Gwen Shapira, Guozhang Wang Closes #1042 from ijuma/kafka-3375-suppress-depreccated-tweak-compiler
…expired batches in recordAccumulator Added topic-partition information to the exception message on batch expiry in RecordAccumulator Author: MayureshGharat <gharatmayuresh15@gmail.com> Reviewers: Gwen Shapira, Lin Dong, Ismael Juma Closes #695 from MayureshGharat/kafka-3013
Please see https://cwiki.apache.org/confluence/display/KAFKA/KIP-36+Rack+aware+replica+assignment for the overall design. The update to TopicMetadataRequest/TopicMetadataResponse will be done in a different PR. Author: Allen Wang <awang@netflix.com> Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>, Grant Henke <granthenke@gmail.com>, Jun Rao <junrao@gmail.com> Closes #132 from allenxwang/KAFKA-1215
Author: Manikumar reddy O <manikumar.reddy@gmail.com> Reviewers: Sriharsha Chintalapani <schintalapani@hortonworks.com>, Ismael Juma <ismael@juma.me.uk> Closes #1230 from omkreddy/KAFKA-3406
Author: Manikumar reddy O <manikumar.reddy@gmail.com> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #1237 from omkreddy/KAFKA-3580
guozhangwang Author: Damian Guy <damian.guy@gmail.com> Reviewers: Matthias J. Sax, Guozhang Wang Closes #1272 from dguy/kstreamimpl-to-npe and squashes the following commits: 49d48fb [Damian Guy] actually commit the fix 07ce589 [Damian Guy] fix npe in KStreamImpl.to(..) 74d396d [Damian Guy] fix npe in KStreamImpl.to(..)
The patch does the following. 1. Adds ApiVersionsRequest/Response. 2. Adds UNSUPPORTED_VERSION error and UnsupportedVersionException. 3. Adds broker side handling of ApiVersionsRequest. Author: Ashish Singh <asingh@cloudera.com> Reviewers: Gwen Shapira, Ismael Juma, Magnus Edenhill Closes #986 from SinghAsDev/KAFKA-3307
Author: Eno Thereska <eno.thereska@gmail.com> Reviewers: Ismael Juma, Damian Guy, Michael G. Noll, Guozhang Wang Closes #1260 from enothereska/KAFKA-3612-integration-tests
Author: Grant Henke <granthenke@gmail.com> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #1279 from granthenke/struct-fix
Author: Ashish Singh <asingh@cloudera.com> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #1275 from SinghAsDev/KAFKA-3621
ewencp granders Can you take a look? Thanks! Author: Liquan Pei <liquanpei@gmail.com> Reviewers: Ewen Cheslack-Postava <ewen@confluent.io> Closes #1259 from Ishiihara/fix-warning
Unit tests for SASL authenticator, tests for SASL/PLAIN and multiple mechanisms, authorization test for SASL/PLAIN Author: Rajini Sivaram <rajinisivaram@googlemail.com> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #1273 from rajinisivaram/KAFKA-3617
Author: Jason Gustafson <jason@confluent.io> Reviewers: Guozhang Wang <wangguoz@gmail.com>, Onur Karaman <okaraman@linkedin.com>, Ismael Juma <ismael@juma.me.uk> Closes #1284 from hachikuji/KAFKA-3636
Author: Liquan Pei <liquanpei@gmail.com> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes #1280 from Ishiihara/minor-warning-fix
Author: Ashish Singh <asingh@cloudera.com> Reviewers: Geoff Anderson <geoff@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io> Closes #1160 from SinghAsDev/KAFKA-3382
…indows This PR includes the same code as #1261 but is rebased on latest trunk. Author: Michael G. Noll <michael@confluent.io> Reviewers: Matthias J. Sax, Guozhang Wang Closes #1277 from miguno/KAFKA-3613-v2
Author: Matthias J. Sax <matthias@confluent.io> Reviewers: Michael G. Noll, Guozhang Wang Closes #1250 from mjsax/JavaDoc-publicAPI
Run a sanity test with SASL/PLAIN and a couple of replication tests with SASL/PLAIN and multiple mechanisms. Author: Rajini Sivaram <rajinisivaram@googlemail.com> Reviewers: Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io> Closes #1282 from rajinisivaram/KAFKA-2693
Author: Jason Gustafson <jason@confluent.io> Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Ewen Cheslack-Postava <ewen@confluent.io> Closes #1129 from hachikuji/KAFKA-3418
granders hachikuji Can you take a look when you have time? Appreciate your time to review. Author: Liquan Pei <liquanpei@gmail.com> Reviewers: Grant Henke <granthenke@gmail.com>, Geoff Anderson <geoff@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io> Closes #1263 from Ishiihara/classpath-no-test-jar
…down cleanly Even if a test calls stop() on console_consumer or verifiable_producer, it is still possible that producer/consumer will not shutdown cleanly, and will be killed forcefully after a timeout. It will be useful for some tests to know whether a clean shutdown happened or not. This PR adds methods to console_consumer and verifiable_producer to query whether clean shutdown happened or not. hachikuji and/or granders Please review. Author: Anna Povzner <anna@confluent.io> Reviewers: Jason Gustafson, Geoff Anderson, Gwen Shapira Closes #1278 from apovzner/kafka-3597
Author: Liquan Pei <liquanpei@gmail.com> Reviewers: Ewen Cheslack-Postava <ewen@confluent.io> Closes #1288 from Ishiihara/kip-56
Server-side implementation and tests for handling ApiVersionsRequest before SaslHandshakeRequest. Author: Rajini Sivaram <rajinisivaram@googlemail.com> Reviewers: Gwen Shapira, Ismael Juma Closes #1286 from rajinisivaram/KAFKA-3618
- add class doc for KTable, KStream, JoinWindows - add missing return tags Author: Matthias J. Sax <matthias@confluent.io> Reviewers: Guozhang Wang <wangguoz@gmail.com>, Michael G. Noll <michael@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io> Closes #1287 from mjsax/kafka-3440-JavaDoc
Author: Grant Henke <granthenke@gmail.com> Reviewers: Gwen Shapira, Ismael Juma Closes #1292 from granthenke/recordmeta-compat
…not properly clean up existing tasks hachikuji ewencp Can you take a look when you have time? Author: Liquan Pei <liquanpei@gmail.com> Reviewers: Jason Gustafson <jason@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io> Closes #1248 from Ishiihara/kafka-3459
Fixes wrong KeyValue equals logic when keys not equal but values equal. Original hotfix PR at #1293 (/cc enothereska) Please review: ewencp ijuma guozhangwang Author: Eno Thereska <eno.thereska@gmail.com> Author: Michael G. Noll <michael@confluent.io> Reviewers: Michael G. Noll <michael@confluent.io>, Ewen Cheslack-Postava <ewen@confluent.io> Closes #1294 from miguno/KeyValue-equality-hotfix
this is consistent with the Max stat implementation. Author: Zack Dever <zdever@pandora.com> Reviewers: Aditya Auradkar, Gwen Shapira Closes #1143 from zackdever/min-stat-fix
…artitions Author: Grant Henke <granthenke@gmail.com> Reviewers: Gwen Shapira Closes #1296 from granthenke/stop-boolean
…gression) bin/kafka-run-class.sh does not correctly setup the CLASSPATH in release rc2. Author: Dana Powers <dana.powers@gmail.com> Reviewers: Ewen Cheslack-Postava <ewen@confluent.io> Closes #1302 from dpkp/KAFKA-3615-fix
tozhan-zz
pushed a commit
that referenced
this pull request
Feb 2, 2018
This may be a reason why we see Jenkins jobs time out at times. I can reproduce it locally. With current trunk there is a possibility to run into this: ```sh "kafka-streams-close-thread" apache#585 daemon prio=5 os_prio=0 tid=0x00007f66d052d800 nid=0x7e02 waiting for monitor entry [0x00007f66ae2e5000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.kafka.streams.processor.internals.StreamThread.close(StreamThread.java:345) - waiting to lock <0x000000077d33c538> (a org.apache.kafka.streams.processor.internals.StreamThread) at org.apache.kafka.streams.KafkaStreams$1.run(KafkaStreams.java:474) at java.lang.Thread.run(Thread.java:745) "appId-bd262a91-5155-4a35-bc46-c6432552c2c5-StreamThread-97" apache#583 prio=5 os_prio=0 tid=0x00007f66d052f000 nid=0x7e01 waiting for monitor entry [0x00007f66ae4e6000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.kafka.streams.KafkaStreams.setState(KafkaStreams.java:219) - waiting to lock <0x000000077d335760> (a org.apache.kafka.streams.KafkaStreams) at org.apache.kafka.streams.KafkaStreams.access$100(KafkaStreams.java:117) at org.apache.kafka.streams.KafkaStreams$StreamStateListener.onChange(KafkaStreams.java:259) - locked <0x000000077d42f138> (a org.apache.kafka.streams.KafkaStreams$StreamStateListener) at org.apache.kafka.streams.processor.internals.StreamThread.setState(StreamThread.java:168) - locked <0x000000077d33c538> (a org.apache.kafka.streams.processor.internals.StreamThread) at org.apache.kafka.streams.processor.internals.StreamThread.setStateWhenNotInPendingShutdown(StreamThread.java:176) - locked <0x000000077d33c538> (a org.apache.kafka.streams.processor.internals.StreamThread) at org.apache.kafka.streams.processor.internals.StreamThread.access$1600(StreamThread.java:70) at org.apache.kafka.streams.processor.internals.StreamThread$RebalanceListener.onPartitionsRevoked(StreamThread.java:1321) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinPrepare(ConsumerCoordinator.java:406) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:349) at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:310) at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:296) at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1037) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1002) at org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:531) at org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:669) at org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:326) ``` In a nutshell: `KafkaStreams` and `StreamThread` are both waiting for each other since another intermittent `close` (eg. from a test) comes along also trying to lock on `KafkaStreams` : ```sh "main" #1 prio=5 os_prio=0 tid=0x00007f66d000c800 nid=0x78bb in Object.wait() [0x00007f66d7a15000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1249) - locked <0x000000077d45a590> (a java.lang.Thread) at org.apache.kafka.streams.KafkaStreams.close(KafkaStreams.java:503) - locked <0x000000077d335760> (a org.apache.kafka.streams.KafkaStreams) at org.apache.kafka.streams.KafkaStreams.close(KafkaStreams.java:447) at org.apache.kafka.streams.KafkaStreamsTest.testCannotStartOnceClosed(KafkaStreamsTest.java:115) ``` => causing a deadlock. Fixed this by softer locking on the state change, that guarantees atomic changes to the state but does not lock on the whole object (I at least could not find another method that would require more than atomicly-locked access except for `setState`). Also qualified the state listeners with their outer-class to make the whole code-flow around this more readable (having two interfaces with the same naming for interface and method and then using them between their two outer classes is crazy hard to read imo :)). Easy to reproduced yourself by running `org.apache.kafka.streams.KafkaStreamsTest` in a loop for a bit (save yourself some time by running 2-4 in parallel :)). Eventually it will lock on one of the tests (for me this takes less than 1 min with 4 parallel runs). Author: Armin Braun <me@obrown.io> Author: Armin <me@obrown.io> Reviewers: Eno Thereska <eno@confluent.io>, Damian Guy <damian.guy@gmail.com>, Ismael Juma <ismael@juma.me.uk> Closes apache#2791 from original-brownbear/fix-streams-deadlock
kondetibharat
pushed a commit
that referenced
this pull request
Jan 16, 2020
…pache#7305) A partition log in initialized in following steps: 1. Fetch log config from ZK 2. Call LogManager.getOrCreateLog which creates the Log object, then 3. Registers the Log object Step #3 enables Configuration update thread to deliver configuration updates to the log. But if any update arrives between step #1 and #3 then that update is missed. It breaks following use case: 1. Create a topic with default configuration, and immediately after that 2. Update the configuration of topic There is a race condition here and in random cases update made in second step will get dropped. This change fixes it by tracking updates arriving between step #1 and #3 Once a Partition is done initializing log, it checks if it has missed any update. If yes, then the configuration is read from ZK again. Added unit tests to make sure a dirty configuration is refreshed. Tested on local cluster to make sure that topic configuration and updates are handled correctly. Reviewers: Jason Gustafson <jason@confluent.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.