KAFKA-4317: Checkpoint StateStores on commit interval#3023
Closed
dguy wants to merge 179 commits intoapache:trunkfrom
Closed
KAFKA-4317: Checkpoint StateStores on commit interval#3023dguy wants to merge 179 commits intoapache:trunkfrom
dguy wants to merge 179 commits intoapache:trunkfrom
Conversation
Remove applicationId parameter as it is no longer used. Author: Damian Guy <damian.guy@gmail.com> Reviewers: Guozhang Wang <wangguoz@gmail.com> Closes apache#2385 from dguy/minor-remove-unused-param
Author: Rajini Sivaram <rajinisivaram@googlemail.com> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes apache#2316 from rajinisivaram/KAFKA-4363
Runs sanity test and one replication test using SASL/SCRAM. Author: Rajini Sivaram <rajinisivaram@googlemail.com> Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Ismael Juma <ismael@juma.me.uk> Closes apache#2355 from rajinisivaram/KAFKA-4590
Switched console_consumer, verifiable_consumer and verifiable_producer to use new sasl.jaas_config property instead of static JAAS configuration file when used with SASL_PLAINTEXT. Author: Rajini Sivaram <rajinisivaram@googlemail.com> Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Ismael Juma <ismael@juma.me.uk> Closes apache#2323 from rajinisivaram/KAFKA-4580 (cherry picked from commit 3f6c4f6) Signed-off-by: Ismael Juma <ismael@juma.me.uk>
Author: Jason Gustafson <jason@confluent.io> Reviewers: Vahid Hashemian <vahidhashemian@us.ibm.com>, Ismael Juma <ismael@juma.me.uk> Closes apache#2383 from hachikuji/minor-cleanup-kip-88
…nTest.shouldNotMakeStoreAvailableUntilAllStoresAvailable After debugging this i can see the times that it fails there is a race between when the topic is actually created/ready on the broker and when the assignment happens. When it fails `StreamPartitionAssignor.assign(..)` gets called with a `Cluster` with no topics. Hence the test hangs as no tasks get assigned. To fix this I added a `waitForTopics` method to `EmbeddedKafkaCluster`. This will wait until the topics have been created. Author: Damian Guy <damian.guy@gmail.com> Reviewers: Matthias J. Sax, Guozhang Wang Closes apache#2371 from dguy/integration-test-fix (cherry picked from commit 825f225) Signed-off-by: Guozhang Wang <wangguoz@gmail.com>
This is a follow up of apache#2166 - refactoring the store hierarchies as requested Author: Damian Guy <damian.guy@gmail.com> Reviewers: Guozhang Wang <wangguoz@gmail.com> Closes apache#2360 from dguy/state-store-refactor (cherry picked from commit 73b7ae0) Signed-off-by: Guozhang Wang <wangguoz@gmail.com>
1. Added javadoc to public classes 2. Removed `s` from config name for consistency with interface name 3. The policy interface now implements Configurable and AutoCloseable as per the KIP 4. Use `null` instead of `-1` in `RequestMetadata` 5. Perform all broker validation before invoking the policy 6. Add tests Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Jason Gustafson <jason@confluent.io> Closes apache#2388 from ijuma/create-topic-policy-docs-and-config-name-change (cherry picked from commit fd6d7bc) Signed-off-by: Ismael Juma <ismael@juma.me.uk>
In RocksDBStore, options / wOptions / fOptions are constructed in the constructor, which needs to be dismissed in the close() call; however in some tests, the generated topology is not initialized at all, and hence the corresponding state stores are supposed to not be able to be closed as well since their `init` function is not called. This could cause the above option objects to be not released. This is fixed in this patch to move the logic out of constructor and inside `init` functions, so that no RocksDB objects will be created in the constructor only. Also some minor cleanups: 1. In KStreamTestDriver.close(), we lost the logic to close the state stores but only call `flush`; it is now changed back to call both. 2. Moved the forwarding logic from KStreamTestDriver to MockProcessorContext to remove the mutual dependency: these functions should really be in ProcessorContext, not the test driver. Author: Guozhang Wang <wangguoz@gmail.com> Reviewers: Damian Guy <damian.guy@gmail.com>, Matthias J. Sax <matthias@confluent.io>, Jason Gustafson <jason@confluent.io> Closes apache#2381 from guozhangwang/K3502-pure-virtual-function-unit-tests (cherry picked from commit 1974e1b) Signed-off-by: Jason Gustafson <jason@confluent.io>
Remove workaround for testing multiple SASL mechanisms using sasl.jaas.config and the new support for multiple client modules within a JVM. Author: Rajini Sivaram <rajinisivaram@googlemail.com> Reviewers: Sriharsha Chintalapani <harsha@hortonworks.com>, Ismael Juma <ismael@juma.me.uk> Closes apache#2373 from rajinisivaram/KAFKA-4568 (cherry picked from commit 4c49297) Signed-off-by: Ismael Juma <ismael@juma.me.uk>
interface for `Processor` in comments incorrectly had `transform` rather than `process`. Author: Damian Guy <damian.guy@gmail.com> Reviewers: Michael G. Noll, Ismael Juma <ismael@juma.me.uk> Closes apache#2396 from dguy/minor-javadoc (cherry picked from commit ccb183f) Signed-off-by: Ismael Juma <ismael@juma.me.uk>
This class doesn't need to override this method as it is handled appropriately by the super class Author: Damian Guy <damian.guy@gmail.com> Reviewers: Guozhang Wang <wangguoz@gmail.com> Closes apache#2397 from dguy/hotfix-npe-state-store (cherry picked from commit 961ebca) Signed-off-by: Guozhang Wang <wangguoz@gmail.com>
Author: Matthias J. Sax <matthias@confluent.io> Reviewers: Eno Thereska, Guozhang Wang Closes apache#2392 from mjsax/minor-system-test-rework (cherry picked from commit d8a7756) Signed-off-by: Guozhang Wang <wangguoz@gmail.com>
Author: Colin P. Mccabe <cmccabe@confluent.io> Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Apurva Mehta <apurva.1618@gmail.com>, Ismael Juma <ismael@juma.me.uk> Closes apache#2184 from cmccabe/KAFKA-4457 (cherry picked from commit 4a6f2c6) Signed-off-by: Ismael Juma <ismael@juma.me.uk>
…e already exist with unexpected number of partitions Re-branched the trunk and applied the changes to the new branch to simplify commit log. Author: Hojjat Jafarpour <hojjat@Hojjat-Jafarpours-MBP.local> Reviewers: Ismael Juma, Damian Guy, Eno Thereska, Guozhang Wang Closes apache#2389 from hjafarpour/KAFKA-4060-Remove-ZkClient-dependency-in-Kafka-Streams-followup-from-trunk Address Ismael's comments upon merging (cherry picked from commit 8e2cbae) Signed-off-by: Guozhang Wang <wangguoz@gmail.com>
Author: Rajini Sivaram <rajinisivaram@googlemail.com> Reviewers: Ismael Juma <ismael@juma.me.uk>, Gwen Shapira <cshapi@gmail.com>, Sriharsha Chintalapani <harsha@hortonworks.com>, Jun Rao <junrao@gmail.com> Closes apache#2369 from rajinisivaram/KAFKA-4589 (cherry picked from commit 666abaf) Signed-off-by: Jun Rao <junrao@gmail.com>
…ause Issue: https://issues.apache.org/jira/browse/KAFKA-4614 Fixes the problem that the broker threads suffered by long GC pause. When GC thread collects mmap objects which were created for index files, it unmaps memory mapping so kernel turns to delete a file physically. This work may transparently read file's metadata from physical disk if it's not available on cache. This seems to happen typically when we're using G1GC, due to it's strategy to left a garbage for a long time if other objects in the same region are still alive. See the link for the details. Author: Yuto Kawamura <kawamuray.dadada@gmail.com> Reviewers: Apurva Mehta <apurva.1618@gmail.com>, Guozhang Wang <wangguoz@gmail.com>, Ismael Juma <ismael@juma.me.uk>, Closes apache#2352 from kawamuray/KAFKA-4614-force-munmap-for-index (cherry picked from commit 5fc530b) Signed-off-by: Jun Rao <junrao@gmail.com>
…ed offsets Author: Vahid Hashemian <vahidhashemian@us.ibm.com> Reviewers: Jason Gustafson <jason@confluent.io> Closes apache#2341 from vahidhashemian/KAFKA-4547
…roducer script. Current implementation of ProducerPerformance creates static payload. This is not very useful in testing compression or when you want to test with production/custom payloads. So, we decided to add support for providing payload file as an input to producer perf test script. We made the following changes: 1. Added support to provide a payload file which can have the list of payloads that you actually want to send. 2. Moved payload generation inside the send loop for cases when payload file is provided. Following are the changes to how the producer-performance is evoked: 1. You must provide "--record-size" or "--payload-file" but not both. This is because, record size cannot be guaranteed when you are using custom events. e.g. ./kafka-producer-perf-test.sh --topic test_topic --num-records 100000 --producer-props bootstrap.servers=127.0.0.1:9092 acks=0 buffer.memory=33554432 compression.type=gzip batch.size=10240 linger.ms=10 --throughput -1 --payload-file ./test_payloads --payload-delimiter , 2. Earlier "--record-size" was a required config, now you must provide exactly one of "--record-size" or "--payload-file". Providing both will result in an error. 3. Support for an additional parameter "--payload-delimiter" has been added which defaults to "\n" Author: Sandesh K <sandesh.karkera@flipkart.com> Reviewers: dan norwood <norwood@confluent.io>, Jun Rao <junrao@gmail.com> Closes apache#2158 from SandeshKarkera/PerfProducerChanges (cherry picked from commit a37bf5f) Signed-off-by: Jun Rao <junrao@gmail.com>
This log message tends to be extremely verbose when state stores are being restored Author: Xavier Léauté <xavier@confluent.io> Reviewers: Guozhang Wang <wangguoz@gmail.com> Closes apache#2412 from xvrl/reduce-verbosity (cherry picked from commit 50e035d) Signed-off-by: Guozhang Wang <wangguoz@gmail.com>
Renames `HoistToStruct` SMT to `HoistField`. Adds the following SMTs: `ExtractField` `MaskField` `RegexRouter` `ReplaceField` `SetSchemaMetadata` `ValueToKey` Adds HTML doc generation and updates to `connect.html`. Author: Shikhar Bhushan <shikhar@confluent.io> Reviewers: Ewen Cheslack-Postava <ewen@confluent.io> Closes apache#2374 from shikhar/more-smt
…Manager branch and fixed one copartitioning bug ZK removed reveal a bug in `StreamPartitionAssigner` but did not fix it properly. This is a follow up bug fix. Issue: - If topic metadata is missing, `StreamPartitionAssigner` should not create any affected tasks that consume topics with missing metadata. - Depending downstream tasks should not be create either. - For tasks that are not created, no store changelog topics (if any) should get created - For tasks that write output to not-yet existing internal repartitioning topics, those repartitioning topics should not get created Author: Matthias J. Sax <matthias@confluent.io> Reviewers: Damian Guy, Guozhang Wang Closes apache#2404 from mjsax/kafka-4060-zk-test-follow-up (cherry picked from commit 0b99bea) Signed-off-by: Guozhang Wang <wangguoz@gmail.com>
Variance changes introduced in KIP-100 cause compilation failures with lambda expression in Java 8. To my knowledge this only affects the following method `KStreams.transform(TransformerSupplier<...>, String...)` prior to the changes it was possible to write: `streams.transform(MyTransformer::new)` where `MyTransformer` extends `Transformer` After the changes the Java compiler is unable to infer correct return types for the lambda expressions. This change fixed this by reverting to invariant return types for transformer suppliers. please cherry-pick into 0.10.2.x Author: Xavier Léauté <xavier@confluent.io> Reviewers: Ismael Juma, Damian Guy, Guozhang Wang Closes apache#2402 from xvrl/lambdas-oh-my (cherry picked from commit 20e957c) Signed-off-by: Guozhang Wang <wangguoz@gmail.com>
This is a KIP-104/105 follow-up. Thanks to ijuma for pointing out. Author: Eno Thereska <eno.thereska@gmail.com> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes apache#2350 from enothereska/minor-broker-level-config (cherry picked from commit 1eb1e2f) Signed-off-by: Ismael Juma <ismael@juma.me.uk>
Author: Matthias J. Sax <matthias@confluent.io> Reviewers: Damian Guy, Guozhang Wang Closes apache#2418 from mjsax/kafka-4060-zk-test-follow-up (cherry picked from commit dd897bd) Signed-off-by: Guozhang Wang <wangguoz@gmail.com>
Updated the docs with changes in KAFKA-4060. Author: Hojjat Jafarpour <hojjat@Hojjat-Jafarpours-MBP.local> Author: Hojjat Jafarpour <hojjat@HojjatJpoursMBP.attlocal.net> Reviewers: Ismael Juma, Matthias J. Sax, Guozhang Wang Closes apache#2377 from hjafarpour/KAFKA-4060-docs-update (cherry picked from commit 62206de) Signed-off-by: Guozhang Wang <wangguoz@gmail.com>
1. In StreamThread, always use subscribe(Pattern, ..) function in order to avoid sending MetadataRequest with specific topic names and cause brokers to possibly auto-create subscribed topics; the pattern is generated as "topic-1|topic-2..|topic-n". 2. In ConsumerCoordinator, let the leader to refresh its metadata if the generated assignment contains some topics that is not contained in the subscribed topics; also in SubscriptionState, modified the verification for regex subscription to against the regex pattern instead of the matched topics since the returned assignment may contain some topics not yet created when joining the group but existed after the rebalance; also modified some unit tests in `KafkaConsumerTest` to accommodate the above changes. 3. Minor cleanup: changed String[] to List<String> to avoid overloaded functions. 4. Minor cleanup: enforced strong typing in SinkNodeFactory and removed unnecessary unchecked tags. 5. Minor cleanup: augmented unit test error message and fixed a potential transient failure in KafkaStreamTest. Author: Guozhang Wang <wangguoz@gmail.com> Reviewers: Damian Guy <damian.guy@gmail.com>, Matthias J. Sax <matthias@confluent.io>, Jason Gustafson <jason@confluent.io> Closes apache#2379 from guozhangwang/K4633-regex-pattern (cherry picked from commit 3400d0c) Signed-off-by: Jason Gustafson <jason@confluent.io>
Author: Matthias J. Sax <matthias@confluent.io> Reviewers: Damian Guy, Eno Thereska, Guozhang Wang Closes apache#2401 from mjsax/kafka-4671-window-retention-policy (cherry picked from commit 7998759) Signed-off-by: Guozhang Wang <wangguoz@gmail.com>
`FileChannel.read` may not fill the destination buffer even if there are enough bytes in the channel to do so. Add a couple of utility methods that ensure this and use them from all the relevant places. Author: huxi <huxi@zhenrongbao.com> Author: amethystic <huxi_2b@hotmail.com> Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Jun Rao <junrao@gmail.com>, Jason Gustafson <jason@confluent.io>, Ismael Juma <ismael@juma.me.uk> Closes apache#2304 from amethystic/kafka4576_FileChannel_read (cherry picked from commit 337f576) Signed-off-by: Ismael Juma <ismael@juma.me.uk>
Author: Colin P. Mccabe <cmccabe@confluent.io> Reviewers: Ewen Cheslack-Postava <ewen@confluent.io>, Ismael Juma <ismael@juma.me.uk> Closes apache#2424 from cmccabe/KAFKA-4688 (cherry picked from commit 5671806) Signed-off-by: Ismael Juma <ismael@juma.me.uk>
Cherry pick from apache#2685 Author: Eno Thereska <eno.thereska@gmail.com> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes apache#2817 from enothereska/KMinor-improve-log4j-0.10.2
Author: Michael G. Noll <michael@confluent.io> Reviewers: Matthias J. Sax <matthias@confluent.io>, Eno Thereska <eno@confluent.io>, Ismael Juma <ismael@juma.me.uk> Closes apache#2812 from miguno/trunk-streams-examples-docs (cherry picked from commit 6ba98f6) Signed-off-by: Ismael Juma <ismael@juma.me.uk>
Highlight that the range in `fetch` is inclusive of both `timeFrom` and `timeTo` Author: Damian Guy <damian.guy@gmail.com> Reviewers: Michael G. Noll <michael@confluent.io>, Matthias J. Sax <matthias@confluent.io>, Ismael Juma <ismael@juma.me.uk> Closes apache#2811 from dguy/minor-window-fetch-java-doc (cherry picked from commit b611cfa) Signed-off-by: Ismael Juma <ismael@juma.me.uk>
…ss name(), not canonicalName() Changing getCanonicalName() references to getName() so that docs update with "$" instead of ".". Also added a connect-plugin-discovery.sh CLI to list all of the transformations available. Author: Bruce Szalwinski <bruce.szalwinski@cdk.com> Reviewers: Gwen Shapira Closes apache#2720 from bruce-szalwinski/transforms and squashes the following commits: ec3b5b9 [Bruce Szalwinski] remove connect-plugin-discovery. will submit in a different PR eba0af7 [Bruce Szalwinski] Key / Value transformations are static nested classes and so are referenced using OuterClass$Key and OuterClass$Value. (cherry picked from commit 763ea5a)
A more conservative version of the change for the 0.10.2 branch. Trunk commit: 1659ca1. Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Jun Rao <junrao@gmail.com> Closes apache#2810 from ijuma/kafka-4901-produce-request-thread-safety-0-10-2
Author: Matthias J. Sax <matthias@confluent.io> Reviewers: Eno Thereska <eno@confluent.io>, Ismael Juma <ismael@juma.me.uk> Closes apache#2774 from mjsax/minor-fix-reset-0102
Author: Eno Thereska <eno@confluent.io> Reviewers: Ewen Cheslack-Postava <ewen@confluent.io> Closes apache#2793 from enothereska/KAFKA-4916-0.10.2
Author: Eno Thereska <eno@confluent.io> Reviewers: Damian Guy <damian.guy@gmail.com>, Matthias J. Sax <matthias@confluent.io>, Ismael Juma <ismael@juma.me.uk> Closes apache#2819 from enothereska/minor-increase-retries (cherry picked from commit e19bf3f) Signed-off-by: Ismael Juma <ismael@juma.me.uk>
Author: Rajini Sivaram <rajinisivaram@googlemail.com> Reviewers: Ismael Juma, Jun Rao Closes apache#2733 from rajinisivaram/KAFKA-4943 (cherry picked from commit 67fc2a9) Signed-off-by: Gwen Shapira <cshapi@gmail.com>
In 67fc2a9, we are using an empty collection and comparing via value equality, so if a user passes an empty collection, they will get the default ACLs instead of no ACLs. We fix that issue here. Author: Ismael Juma <ismael@juma.me.uk> Reviewers: Rajini Sivaram Closes apache#2829 from ijuma/zk-utils-default-acls-improvement and squashes the following commits: 0846172 [Ismael Juma] Add missing import 2dc84f3 [Ismael Juma] Simplify logic in `sensitivePath` 8122f27 [Ismael Juma] Use a true sentinel instead of an empty collection for `UseDefaultAcls` (cherry picked from commit c31958e) Signed-off-by: Gwen Shapira <cshapi@gmail.com>
…ession stores Author: Michael G. Noll <michael@confluent.io> Reviewers: Damian Guy <damian.guy@gmail.com>, Ismael Juma <ismael@juma.me.uk> Closes apache#2751 from miguno/trunk-streams-window-iterator-doc-fixes (cherry picked from commit 7c58351) Signed-off-by: Ismael Juma <ismael@juma.me.uk>
Author: Eno Thereska <eno@confluent.io> Reviewers: Gwen Shapira Closes apache#2824 from enothereska/minor-docs-0.10.2.1
Introduced in PR apache#2824. Already fixed in the website github. Author: Gwen Shapira <cshapi@gmail.com> Reviewers: Ismael Juma <ismael@juma.me.uk> Closes apache#2844 from gwenshap/docs-hotfix
Author: Eno Thereska <eno.thereska@gmail.com> Reviewers: Damian Guy <damian.guy@gmail.com>, Matthias J. Sax <matthias@confluent.io>, Ismael Juma <ismael@juma.me.uk> Closes apache#2841 from enothereska/KAFKA-5038
…h position has already increased Author: Dong Lin <lindong28@gmail.com> Author: Dong Lin <lindong28@users.noreply.github.com> Reviewers: Jiangjie Qin <becket.qin@gmail.com> Closes apache#2859 from lindong28/KAFKA-5075 This is a backport patch for 0.10.2 after resolving the following conflict. Conflicts: clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java
…entially unassigned partitions Author: Eno Thereska <eno.thereska@gmail.com> Reviewers: Ismael Juma <ismael@juma.me.uk>, Jiangjie Qin <becket.qin@gmail.com>, Jason Gustafson <jason@confluent.io> Closes apache#2876 from enothereska/KAFKA-4755-fetcher-only
Author: Matthias J. Sax <matthias@confluent.io> Reviewers: Damian Guy, Eno Thereska, Guozhang Wang Closes apache#2789 from mjsax/minor-improve-integration-test (cherry picked from commit 14ab3dc) Signed-off-by: Guozhang Wang <wangguoz@gmail.com>
Author: Eno Thereska <eno@confluent.io> Reviewers: Damian Guy, Guozhang Wang Closes apache#2877 from enothereska/KAFKA-5095-cacheOverheads (cherry picked from commit f18a14a) Signed-off-by: Guozhang Wang <wangguoz@gmail.com>
We got test error `org.apache.kafka.common.errors.TopicExistsException: Topic 'inputTopic' already exists.` in some builds. Can reproduce reliably at local machine. Root cause it async "topic delete" that might not be finished before topic gets re-created. Author: Matthias J. Sax <matthias@confluent.io> Reviewers: Ismael Juma, Damian Guy, Guozhang Wang Closes apache#2757 from mjsax/minor-fix-resetintegrationtest
This addresses some tests' instabilities in 0.10.2 that have been fixed in trunk Author: Eno Thereska <eno.thereska@gmail.com> Reviewers: Guozhang Wang <wangguoz@gmail.com> Closes apache#2867 from enothereska/minor-performance-cherrypick-0.10.2
0.10.2.1-rc3
…cksDB This fix needs to be backported to 0.10.2 as well. Author: Eno Thereska <eno.thereska@gmail.com> Reviewers: Damian Guy, Ismael Juma, Guozhang Wang Closes apache#2982 from enothereska/KAFKA-5174-1-core
…rvice in 0.10.2 Author: Konstantine Karantasis <konstantine@confluent.io> Reviewers: Jason Gustafson <jason@confluent.io> Closes apache#3007 from kkonstantine/HOTFIX-Align-startup-wait-time-for-ConnectDistributed-service-with-ConnectStandalone-in-kafkatests-0.10.2
|
Refer to this link for build results (access rights to CI server needed): |
|
Refer to this link for build results (access rights to CI server needed): |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a backport of: #2471 from trunk