KAFKA-14309: FK join upgrades not tested with DEV_VERSION#12760
Merged
ableegoldman merged 4 commits intoapache:trunkfrom Nov 7, 2022
Merged
KAFKA-14309: FK join upgrades not tested with DEV_VERSION#12760ableegoldman merged 4 commits intoapache:trunkfrom
ableegoldman merged 4 commits intoapache:trunkfrom
Conversation
Member
Author
|
Test failure is unrelated. A passing system test run for stream_upgrade_test.py is shown on https://jenkins.confluent.io/job/system-test-kafka-branch-builder/5176/consoleText |
The streams upgrade system inserted FK join code for every version of the the StreamsUpgradeTest except for the latest. Also, the original code never switched on the `test.run_fk_join` flag for the target version of the upgrade. The effect was that FK join upgrades were not tested at all, since no FK join code was executed after the bounce in the system test. We introduce `extra_properties` in the system tests, that can be used to pass any property to the upgrade driver, which is supposed to be reused by system tests for switching on and off flags (e.g. for the state restoration code).
Gerrrr
approved these changes
Oct 24, 2022
Contributor
Gerrrr
left a comment
Gerrrr
left a comment
There was a problem hiding this comment.
Looks good! Thank you for fixing this. I left several suggestions; they are pretty much nitpicks, so feel free to ignore them.
Member
Author
|
System test run https://jenkins.confluent.io/job/system-test-kafka-branch-builder/5183 Only failure is unrelated kafkatest.tests.core.round_trip_fault_test.RoundTripFaultTest.test_produce_consume_with_broker_pause.metadata_quorum=ZK |
Co-authored-by: Alex Sorokoumov <918393+Gerrrr@users.noreply.github.com>
ableegoldman
approved these changes
Nov 7, 2022
Member
ableegoldman
left a comment
ableegoldman
left a comment
There was a problem hiding this comment.
LGTM, test failures are unrelated/known to be flaky
Thanks for digging into this and fixing one of our most important system tests!
guozhangwang
pushed a commit
to guozhangwang/kafka
that referenced
this pull request
Jan 25, 2023
The streams upgrade system inserted FK join code for every version of the the StreamsUpgradeTest except for the latest. Also, the original code never switched on the `test.run_fk_join` flag for the target version of the upgrade. The effect was that FK join upgrades were not tested at all, since no FK join code was executed after the bounce in the system test. We introduce `extra_properties` in the system tests, that can be used to pass any property to the upgrade driver, which is supposed to be reused by system tests for switching on and off flags (e.g. for the state restoration code). Reviewers: Alex Sorokoumov <asorokoumov@confluent.io>, Anna Sophie Blee-Goldman <ableegoldman@apache.org>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This change allows foreign key joins to run in the streams upgrade system test. The FK join testing code was originally introduced in #12122 but wasn't executed,
In
streams.py, a version check was performed againstKAFKA_STREAMS_VERSIONwhich would sometimes fail becauseKAFKA_STREAMS_VERSIONis the empty string when testing the current snapshot.FK join code was only added in the
StreamsUpgradeTestof theupgrade-system-test-*grade modules, but not it the mainstreamsgrade module, which is required for testing the current snapshot.The effect was that FK join upgrades were not tested at all.
We introduce
extra_propertiesin the system tests, that can be used to pass any property to the upgrade driver, which is supposed to be reused by system tests for switching on and off flags (e.g. for the state restoration code).Committer Checklist (excluded from commit message)