Conversation
…Thread (#5342) If inter.broker.protocol.version is 2.0-IV1 or newer. Also fixed ListOffsetRequest so that v2 is used, if applicable. Added a unit test which verifies that we use the latest version of the various requests by default. Included a few minor tweaks to make testing easier. Reviewers: Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Dong Lin <dolin@linkedin.com>
This includes a fix for ZOOKEEPER-2184 (Zookeeper Client should re-resolve hosts when connection attempts fail), which fixes KAFKA-4041. Updated a couple of tests as unresolvable addresses are now retried until the connection timeout. Cleaned up tests a little. Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Matthias J. Sax <matthias@confluent.io>
The default server.properties file now contains the log.dirs setting and not log.dir anymore. Co-authored-by: Mickael Maison <mickael.maison@gmail.com> Co-authored-by: Katherine Farmer <kfarme3@uk.ibm.com> Reviewers: Manikumar Reddy O <manikumar.reddy@gmail.com>, Sriharsha Chintalapani <sriharsha@apache.org>
…5371) SSL `close_notify` from broker connection close was processed as a handshake failure in clients while unwrapping the message if a handshake is in progress. Updated to handle this as a retriable IOException rather than a non-retriable SslAuthenticationException to avoid authentication exceptions in clients during rolling restart of brokers. Reviewers: Ismael Juma <ismael@juma.me.uk>
Reviewer: Guozhang Wang <guozhang@confluent.io>
#5368) Reviewers: Ismael Juma <ismael@juma.me.uk>, Guozhang Wang <wangguoz@gmail.com>
The SASL/OAUTHBEARER client response as currently implemented in OAuthBearerSaslClient sends the valid gs2-header "n,," but then sends the "auth" key and value immediately after it. This does not conform to the specification because there is no %x01 after the gs2-header, no %x01 after the auth value, and no terminating %x01. Fixed this and the parsing of the client response in OAuthBearerSaslServer, which currently allows the malformed text. Also updated to accept and ignore unknown properties as required by the spec. Reviewers: Stanislav Kozlovski <familyguyuser192@windowslive.com>, Rajini Sivaram <rajinisivaram@googlemail.com>
Reviewers: Jason Gustafson <jason@confluent.io>
SslTransportLayer currently closes the SSL engine and logs a warning if close_notify message canot be sent because the remote end closed its connection. This tends to fill up broker logs, especially when using clients which close connections immediately. Since this log entry is not very useful anyway, it would be better to log at debug level. Reviewers: Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk>
Reviewers: Guozhang Wang <guozhang@confluent.io>, Rajini Sivaram <rajini@confluent.io>
Reviewers: Ismael Juma <ismael@juma.me.uk>, Rajini Sivaram <rajinisivaram@googlemail.com>
Improve the log messages while at it and fix some code style issues. Reviewers: Ismael Juma <ismael@juma.me.uk>
…g older clients/brokers Added two additional test cases to quota_test.py, which run between brokers and clients with different throttling behaviors. More specifically, 1. clients with new throttling behavior (i.e., post-KIP-219) and brokers with old throttling behavior (i.e., pre-KIP-219) 2. clients with old throttling behavior and brokers with new throttling behavior Author: Jon Lee <jonlee@linkedin.com> Author: Dong Lin <lindong28@gmail.com> Reviewers: Dong Lin <lindong28@gmail.com> Closes #5294 from jonlee2/kafka-6944
…rs by KIP-219 Updated the 2.0 document for changed quota behaviors. Author: Jon Lee <jonlee@linkedin.com> Reviewers: Ismael Juma <ismael@juma.me.uk>, Dong Lin <lindong28@gmail.com> Closes #5384 from jonlee2/KAFKA-7177
… closed (#5027) After successful completion of KafkaProducer#close, it is possible that an application calls KafkaProducer#send. If the send is invoked for a topic for which we do not have any metadata, the producer will block until `max.block.ms` elapses - we do not expect to receive any metadata update in this case because Sender (and NetworkClient) has already exited. It is only when RecordAccumulator#append is invoked that we notice that the producer has already been closed and throw an exception. If `max.block.ms` is set to Long.MaxValue (or a sufficiently high value in general), the producer could block awaiting metadata indefinitely. This patch makes sure `Metadata#awaitUpdate` periodically checks if the network client has been closed, and if so bails out as soon as possible.
…ruction (#5411) This has always been an issue, but the recent upgrade to ZooKeeper 3.4.13 means it is also an issue when an unresolvable ZK address is used, causing some tests to leak threads. The change in behaviour in ZK 3.4.13 is that no exception is thrown from the ZooKeeper constructor in case of an unresolvable address. Instead, ZooKeeper tries to re-resolve the address hoping it becomes resolvable again. We eventually throw a `ZooKeeperClientTimeoutException`, which is similar to the case where the the address is resolvable but ZooKeeper is not reachable. Reviewers: Ismael Juma <ismael@juma.me.uk>
…ts (#5414) ZooKeeper client from version 3.4.13 doesn't handle connections to localhost very well. If ZooKeeper is started on 127.0.0.1 on a machine that has both ipv4 and ipv6 and a client is created using localhost rather than the IP address in the connection string, ZooKeeper client attempts to connect to ipv4 or ipv6 randomly with a fixed one second backoff if connection fails. Use 127.0.0.1 instead of localhost in streams tests to avoid intermittent test failures due to ZK client connection timeouts if ipv6 is chosen in consecutive address selections. Also add note to upgrade docs for 2.0.0. Reviewers: Ismael Juma <github@juma.me.uk>, Matthias J. Sax <matthias@confluent.io>
…ailure (#5417) An untimely wakeup can cause ConsumerCoordinator.onJoinComplete to throw a WakeupException before completion. On the next poll(), it will be retried, but this leads to an underflow error because the buffer containing the assignment data will already have been advanced. The solution is to duplicate the buffer passed to onJoinComplete. Reviewers: Konstantine Karantasis <konstantine@confluent.io>, Rajini Sivaram <rajinisivaram@googlemail.com>
In system tests, it is useful to have the thread dumps if a broker cannot be stopped using SIGTERM. Reviewers: Xavier Léauté <xl+github@xvrl.net>, Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
… a topic is created This patch forces metadata update for consumers with pattern subscription at the beginning of rebalance (retry.backoff.ms is respected). This is to prevent such consumers from detecting subscription changes (e.g., new topic creation) independently and triggering multiple unnecessary rebalances. KAFKA-7126 contains detailed scenarios and rationale. Author: Jon Lee <jonlee@linkedin.com> Reviewers: Jason Gustafson <jason@confluent.io>, Ted Yu <yuzhihong@gmail.com>, Dong Lin <lindong28@gmail.com> Closes #5408 from jonlee2/KAFKA-7126 (cherry picked from commit a932520) Signed-off-by: Dong Lin <lindong28@gmail.com>
Reviewer: Guozhang Wang <guozhang@confluent.io>
2.0.0-rc3
…ogSegment By waiting until server1 has joined the ISR before shutting down server2 Rerun the test method many times after the code change, and there is no flakiness any more. Author: Lucas Wang <luwang@linkedin.com> Reviewers: Mayuresh Gharat <gharatmayuresh15@gmail.com>, Dong Lin <lindong28@gmail.com> Closes #5387 from gitlw/fixing_flacky_logrecevorytest (cherry picked from commit 96bc0b8) Signed-off-by: Dong Lin <lindong28@gmail.com>
The changes are cherry-picked from commit 914ffa9 which is present in 2.1+ versions
…6968) Kafka should not NPE while loading a deleted partition dir with no log segments. This patch ensures that there will always be at least one segment after initialization. Co-authored-by: Edoardo Comar <ecomar@uk.ibm.com> Co-authored-by: Mickael Maison <mickael.maison@gmail.com> Reviewers: Ismael Juma <ismael@juma.me.uk>, Jason Gustafson <jason@confluent.io>
…fficiently sized (#6974) When the log contains out of order message formats (for example v2 message followed by v1 message) and consists of compressed batches typically greater than 1kB in size, it is possible for down-conversion to fail. With compressed batches, we estimate the size of down-converted batches using: ``` private static int estimateCompressedSizeInBytes(int size, CompressionType compressionType) { return compressionType == CompressionType.NONE ? size : Math.min(Math.max(size / 2, 1024), 1 << 16); } ``` This almost always underestimates size of down-converted records if the batch is between 1kB-64kB in size. In general, this means we may under estimate the total size required for compressed batches. Because of an implicit assumption in the code that messages with a lower message format appear before any with a higher message format, we do not grow the buffer we copy the down converted records into when we see a message <= the target message format. This assumption becomes incorrect when the log contains out of order message formats, for example because of leaders flapping while upgrading the message format. Reviewers: Jason Gustafson <jason@confluent.io>
The scoverage plugin is causing the build to fail with recent versions of gradle. I see the following error:
```
* What went wrong:
A problem occurred evaluating root project 'kafka'.
> Failed to apply plugin [id 'org.scoverage']
> Could not create an instance of type org.scoverage.ScoverageExtension.
> You can't map a property that does not exist: propertyName=testClassesDir
```
This patch disables the plugin since we are not typically checking coverage for old branches anyway.
Reviewers: Ismael Juma <ismael@juma.me.uk>
… in AclCommand - Update the AclCommandTest Author: Manikumar Reddy <manikumar.reddy@gmail.com> Reviewers: Jun Rao <junrao@gmail.com> Closes #6263 from omkreddy/aclcommand
) (#7042) This is a fix to #5226 to account for config properties that have an equal char in the value. Otherwise if there is one equal char in the value the following error occurs: dictionary update sequence element #XX has length 3; 2 is required Reviewers: Colin Patrick McCabe <colin@cmccabe.xyz>, Ismael Juma <ismael@juma.me.uk>
Minor fix of #8198 apache/kafka-site#210 Reviewers: John Roesler <john@confluent.io>, Bill Bejeck <bbejeck@gmail.com>
Correct the Flatten SMT to properly handle null key or value `Struct` instances. Author: Michal Borowiecki <michal.borowiecki@openbet.com> Reviewers: Arjun Satish <arjun@confluent.io>, Robert Yokota <rayokota@gmail.com>, Randall Hauch <rhauch@gmail.com>
Fix handling of nulls in TimestampConverter. Authors: Valeria Vasylieva <valeria.vasylieva@gmail.com>, Robert Yokota <rayokota@gmail.com> Reviewers: Arjun Satish <arjun@confluent.io>, Randall Hauch <rhauch@gmail.com>
…n size() == 0 (#7164) Reviewers: A. Sophie Blee-Goldman <sophie@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Guozhang Wang <guozhang@confluent.io>
Reviewers: Bill Bejeck <bill@confluent.io>, Matthias J. Sax <matthias@confluent.io>
Connector validation fails if an alias is used for the converter since the validation for that is done via `ConfigDef.validateAll(...)`, which in turn invokes `Class.forName(...)` on the alias. Even though the class is successfully loaded by the DelegatingClassLoader, some Java implementations will refuse to return a class from `Class.forName(...)` whose name differs from the argument provided. This commit alters `ConfigDef.parseType(...)` to first invoke `ClassLoader.loadClass(...)` on the class using our class loader in order to get a handle on the actual class object to be loaded, then invoke `Class.forName(...)` with the fully-qualified class name of the to-be-loaded class and return the result. The invocation of `Class.forName(...)` is necessary in order to allow static initialization to take place; simply calling `ClassLoader.loadClass(...)` is insufficient. Also corrected a unit test that relied upon the old behavior. Author: Chris Egerton <chrise@confluent.io> Reviewers: Robert Yokota <rayokota@gmail.com>, Randall Hauch <rhauch@gmail.com>
Corrected the AbstractHerder to correctly identify task configs that contain variables for externalized secrets. The original method incorrectly used `matcher.matches()` instead of `matcher.find()`. The former method expects the entire string to match the regex, whereas the second one can find a pattern anywhere within the input string (which fits this use case more correctly). Added unit tests to cover various cases of a config with externalized secrets, and updated system tests to cover case where config value contains additional characters besides secret that requires regex pattern to be found anywhere in the string (as opposed to complete match). Author: Arjun Satish <arjun@confluent.io> Reviewer: Randall Hauch <rhauch@gmail.com>
) When calling readLogToEnd(), the KafkaBasedLog worker thread should catch TimeoutException and log a warning, which can occur if brokers are unavailable, otherwise the worker thread terminates. Includes an enhancement to MockConsumer that allows simulating exceptions not just when polling but also when querying for offsets, which is necessary for testing the fix. Author: Paul Whalen <pgwhalen@gmail.com> Reviewers: Randall Hauch <rhauch@gmail.com>, Arjun Satish <arjun@confluent.io>, Ryanne Dolan <ryannedolan@gmail.com>
) Changed Connect's `WorkerSourceTask` to capture non-retriable exceptions from the `producer.send(...)` (e.g., authentication or authorization errors) and to fail the connector task when such an error is encountered. Modified the existing unit tests to verify this functionality. Note that most producer errors are retriable, and Connect will (by default) set up each producer with 1 max in-flight message and infinite retries. This change only affects non-retriable errors.
…CanReadFromPartitionedTopic (#7281) similar to https://issues.apache.org/jira/browse/KAFKA-8011 and https://issues.apache.org/jira/browse/KAFKA-8026 Reviewers: Matthias J. Sax <mjsax@apache.org>, Bill Bejeck <bbejeck@gmail.com>
#7223) Make offsets immutable to users of RecordCollector.offsets. Fix up an existing case where offsets could be modified in this way. Add a simple test to verify offsets cannot be changed externally. Reviewers: Bruno Cadonna <bruno@confluent.io>, Guozhang Wang <guozhang@confluent.io>, Matthias J. Sax <matthias@confluent.io>
Author: Matthias J. Sax <matthias@confluent.io> Reviewers: Bill Bejeck <bill@confluent.io>, Randall Hauch <rhauch@gmail.com>
…pps (#7346) Reviewer: Matthias J. Sax <matthias@confluent.io>
…vents (#6914) * KAFKA-8523 Avoiding raw type usage * KAFKA-8523 Gracefully handling tombstone events in InsertField SMT
Allow to cast LogicalType to string by calling the serialized (Java) object's toString(). Added tests for `BigDecimal` and `Date` as whole record and as fields. Author: Amit Sela <amitsela33@gmail.com> Reviewers: Randall Hauch <rhauch@gmail.com>, Robert Yokota <rayokota@gmail.com>, Ewen Cheslack-Postava <ewen@confluent.io> Closes #4820 from amitsela/cast-transform-bytes
Adds support for the Connect Cast transforms to cast from Connect logical types, such as DATE, TIME, TIMESTAMP, and DECIMAL. Casting to numeric types will produce the underlying numeric value represented in the desired type. For logical types represented by underlying Java Date class, this means the milliseconds since EPOCH. For Decimal, this means the underlying value. If the value does not fit in the desired target type, it may overflow. Casting to String from Date, Time, and Timestamp types will produce their ISO 8601 representation. Casting to String from Decimal will result in the value represented as a string. e.g. 1234 -> "1234". Author: Nigel Liang <nigel@nigelliang.com> Reviewer: Randall Hauch <rhauch@gmail.com>
…#7442) Trim whitespaces in topic names specified in sink connector configs before subscribing to the consumer. Topic names don't allow whitespace characters, so trimming only will eliminate potential problems and will not place additional limits on topics specified in sink connectors. Author: Magesh Nandakumar <magesh.n.kumar@gmail.com> Reviewers: Arjun Satish <arjun@confluent.io>, Randall Hauch <rhauch@gmail.com>
…list (#7491) Author: Konstantine Karantasis <konstantine@confluent.io> Reviewer: Randall Hauch <rhauch@gmail.com>
Fix bug in Connect REST extension API caused by invalid constructor parameter validation, and update integration test to play nicely with Jenkins Fix instantiation of TaskState objects by Connect framework. Author: Chris Egerton <chrise@confluent.io> Reviewers: Magesh Nandakumar <mageshn@confluent.io>, Randall Hauch <rhauch@gmail.com>
…lizing plugins (#7315) Added plugin isolation unit tests for various scenarios, with a `TestPlugins` class that compiles and builds multiple test plugins without them being on the classpath and verifies that the Plugins and DelegatingClassLoader behave properly. These initially failed for several cases, but now pass since the issues have been fixed. KAFKA-8340 and KAFKA-8819 are closely related, and this fix corrects the problems reported in both issues. Author: Greg Harris <gregh@confluent.io> Reviewers: Chris Egerton <chrise@confluent.io>, Magesh Nandakumar <mageshn@confluent.io>, Konstantine Karantasis <konstantine@confluent.io>, Randall Hauch <rhauch@gmail.com>
…ped tasks (#7532) Prematurely complete source offset read requests for stopped tasks, and added unit tests. Author: Chris Egerton <chrise@confluent.io> Reviewers: Arjun Satish <arjun@confluent.io>, Nigel Liang <nigel@nigelliang.com>, Jinxin Liu <liukrimhim@gmail.com>, Randall Hauch <rhauch@gmail.com>
Given we need to follow the Apache rule of not checking any binaries into the source code, Kafka has always had a bit of a tricky Gradle bootstrap. Using ./gradlew as users expect doesn’t work and a local and compatible version of Gradle was required to generate the wrapper first. This patch changes the behavior of the wrapper task to instead generate a gradlew script that can bootstrap the jar itself. Additionally it adds a license, removes the bat script, and handles retries. The documentation in the readme was also updated. Going forward patches that upgrade gradle should run `gradle wrapper` before checking in the change. With this change users using ./gradlew can be sure they are always building with the correct version of Gradle. Reviewers: Viktor Somogyi <viktorsomogyi@gmail.com>, Ismael Juma <ismael@juma.me.uk
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
"wu_ba-0-C-1" #18 prio=5 os_prio=0 tid=0x00007f4931e62800 nid=0x1c runnable [0x00007f48e07ce000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000003d92e2ae0> (a sun.nio.ch.Util$3)
- locked <0x00000003d92e2ad0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000003d92e2a88> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.kafka.common.network.Selector.select(Selector.java:691)
at org.apache.kafka.common.network.Selector.poll(Selector.java:411)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:510)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:271)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:242)
at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1247)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1187)
at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1154)
at org.springframework.kafka.listener.KafkaMessageListenerContainer$ListenerConsumer.run(KafkaMessageListenerContainer.java:728)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:745)
大量这样的状态导致cpu飙升，如何解决？