Add changes for CI builds and publishing artifacts to bintray.#2
Merged
jonlee2 merged 2 commits intolinkedin:2.0-lifrom Mar 8, 2019
Merged
Add changes for CI builds and publishing artifacts to bintray.#2jonlee2 merged 2 commits intolinkedin:2.0-lifrom
jonlee2 merged 2 commits intolinkedin:2.0-lifrom
Conversation
jonlee2
force-pushed
the
bintray-publish-ci
branch
from
c26bf78 to
87005c6
Compare
Travis will kick off a build and publish artifacts to bintray upon creating a tag in the "x.y.z.w" format.
jonlee2
force-pushed
the
bintray-publish-ci
branch
from
87005c6 to
8ded9d5
Compare
hzxa21
approved these changes
Mar 8, 2019
| # Excluded integration tests for now because Travis will hit build timeout (50 minutes) when including them. | ||
| # Also excluded streams unitTest because they often fail with "pure virtual method called" error (KAFKA-3502). | ||
| script: | ||
| - ./gradlew clean compileJava compileScala compileTestJava compileTestScala checkstyleMain checkstyleTest findbugsMain unitTest -x :streams:unitTest rat --no-daemon -PxmlFindBugsReport=true -PtestLoggingEvents=started,passed,skipped,failed |
There was a problem hiding this comment.
at least add a todo to re-enable streams unit tests
radai-rosenblatt
approved these changes
Mar 8, 2019
jonlee2
pushed a commit
that referenced
this pull request
Mar 8, 2019
…he#6115) This fix is aiming for #2 issue pointed out within https://issues.apache.org/jira/browse/KAFKA-7672 In the current setup, we do offset checkpoint file write when EOS is turned on during #suspend, which introduces the potential race condition during StateManager #closeSuspend call. To mitigate the problem, we attempt to always write checkpoint file in #suspend call. Reviewers: Guozhang Wang <wangguoz@gmail.com>, Matthias J. Sax <mjsax@apache.org>, John Roesler <john@confluent.io>, Bill Bejeck <bbejeck@gmail.com>
xiowu0
pushed a commit
that referenced
this pull request
Jun 11, 2019
…he#6115) This fix is aiming for #2 issue pointed out within https://issues.apache.org/jira/browse/KAFKA-7672 In the current setup, we do offset checkpoint file write when EOS is turned on during #suspend, which introduces the potential race condition during StateManager #closeSuspend call. To mitigate the problem, we attempt to always write checkpoint file in #suspend call. Reviewers: Guozhang Wang <wangguoz@gmail.com>, Matthias J. Sax <mjsax@apache.org>, John Roesler <john@confluent.io>, Bill Bejeck <bbejeck@gmail.com>
xiowu0
pushed a commit
that referenced
this pull request
Jun 11, 2019
…eiving offset commit exception. TICKET = LI_DESCRIPTION = EXIT_CRITERIA = MANUAL ["describe exit criteria"]
xiowu0
pushed a commit
that referenced
this pull request
Jun 11, 2019
…tray. (#2) TICKET = LI_DESCRIPTION = Travis will kick off a build and publish artifacts to bintray upon creating a tag in the "x.y.z.w" format. EXIT_CRITERIA = MANUAL ["describe exit criteria"]
xiowu0
pushed a commit
that referenced
this pull request
Jul 16, 2019
…eiving offset commit exception. TICKET = LI_DESCRIPTION = EXIT_CRITERIA = MANUAL ["needed until we make a long-term fix"]
xiowu0
pushed a commit
that referenced
this pull request
Jul 17, 2019
TICKET = LI_DESCRIPTION = Add build changes to publish github LinkedIn Kafka artifacts to bintray (#1) [NOTE] This is a temporary measure to publish artifacts until CI is properly set up to do the job automatically. Users are not expected to run this themselves. Add changes for CI builds and publishing artifacts to bintray. (#2) Travis will kick off a build and publish artifacts to bintray upon creating a tag in the "x.y.z.w" format. Try different encrypted bintray-related env variables for Travis (#4) Travis couldn't access one of the initially encrypted variables for some reason. Set Bintray-related env variables via repository setting instead of in .travis.yml (#5) Use the maven repo under the LinkedIn Bintray account to publish artifacts (#25) EXIT_CRITERIA = MANUAL [""]
xiowu0
pushed a commit
that referenced
this pull request
Aug 22, 2019
…eiving offset commit exception. TICKET = LI_DESCRIPTION = EXIT_CRITERIA = MANUAL ["needed until we make a long-term fix"]
gitlw
pushed a commit
to gitlw/kafka
that referenced
this pull request
May 5, 2020
…fter receiving offset commit exception. TICKET = LI_DESCRIPTION = EXIT_CRITERIA = MANUAL ["needed until we make a long-term fix"]
gitlw
pushed a commit
that referenced
this pull request
Jun 13, 2020
…eiving offset commit exception. TICKET = LI_DESCRIPTION = EXIT_CRITERIA = MANUAL ["needed until we make a long-term fix"]
gitlw
pushed a commit
to gitlw/kafka
that referenced
this pull request
Oct 5, 2021
…fter receiving offset commit exception. TICKET = LI_DESCRIPTION = EXIT_CRITERIA = MANUAL ["needed until we make a long-term fix"]
lmr3796
referenced
this pull request
in lmr3796/kafka
Oct 22, 2021
…eiving offset commit exception. TICKET = LI_DESCRIPTION = EXIT_CRITERIA = MANUAL ["needed until we make a long-term fix"]
lmr3796
pushed a commit
that referenced
this pull request
Oct 22, 2021
…eiving offset commit exception. TICKET = LI_DESCRIPTION = EXIT_CRITERIA = MANUAL ["needed until we make a long-term fix"]
lmr3796
pushed a commit
that referenced
this pull request
Oct 27, 2021
…eiving offset commit exception. TICKET = LI_DESCRIPTION = EXIT_CRITERIA = MANUAL ["needed until we make a long-term fix"]
lmr3796
pushed a commit
that referenced
this pull request
Oct 28, 2021
…eiving offset commit exception. TICKET = LI_DESCRIPTION = EXIT_CRITERIA = MANUAL ["needed until we make a long-term fix"]
lmr3796
pushed a commit
that referenced
this pull request
Nov 19, 2021
…eiving offset commit exception. TICKET = LI_DESCRIPTION = EXIT_CRITERIA = MANUAL ["needed until we make a long-term fix"]
lmr3796
pushed a commit
that referenced
this pull request
Nov 19, 2021
TICKET = [KAFKA-8667, KAFKA-8668] LI_DESCRIPTION = - Avoid acquiring partitionMap lock in shutdownIdleFetcherThread - ~~Avoid appending to the time index during shutdown if the time index has not yet be initialized~~ This is covered in apache#8346 and apache#10960 EXIT_CRITERIA = TICKET [KAFKA-8667, KAFKA-8668] The patch is a sqaush of the 2 commits: == This is the 1st commit [155b4f8] == [LI-HOTFIX] Reduce lock retention and improve broker shutdown time: TICKET = [KAFKA-8667, KAFKA-8668] LI_DESCRIPTION = - Avoid acquiring partitionMap lock in shutdownIdleFetcherThread - Avoid appending to the time index during shutdown if the time index has not yet be initialized RB=1431408 BUG=LIKAFKA-19361 G=Kafka-Code-Reviews R=jkoshy,jonlee A=jkoshy,jonlee == This is the commit #2 [c53fffd] == [LI-HOTFIX] Update fetcher thread idle flag in addPartitions TICKET = KAFKA-8667 LI_DESCRIPTION = This patch fixes in bug introduced by “[LI-HOTFIX] Reduce lock retention and improve broker shutdown time” HOTFIX where the fetcher thread idle flag is not set in addPartitions, which can cause idle fetcher thread not shutdown in time. RB=1431408 BUG=LIKAFKA-19361 G=Kafka-Code-Reviews R=jkoshy,jonlee A=jkoshy,jonlee
lmr3796
pushed a commit
that referenced
this pull request
Jan 1, 2022
…eiving offset commit exception. TICKET = LI_DESCRIPTION = EXIT_CRITERIA = MANUAL ["needed until we make a long-term fix"]
lmr3796
pushed a commit
that referenced
this pull request
Jan 4, 2022
TICKET = [KAFKA-8667, KAFKA-8668] LI_DESCRIPTION = - Avoid acquiring partitionMap lock in shutdownIdleFetcherThread - ~~Avoid appending to the time index during shutdown if the time index has not yet be initialized~~ This is covered in apache#8346 and apache#10960 EXIT_CRITERIA = TICKET [KAFKA-8667, KAFKA-8668] The patch is a sqaush of the 2 commits: == This is the 1st commit [155b4f8] == [LI-HOTFIX] Reduce lock retention and improve broker shutdown time: TICKET = [KAFKA-8667, KAFKA-8668] LI_DESCRIPTION = - Avoid acquiring partitionMap lock in shutdownIdleFetcherThread - Avoid appending to the time index during shutdown if the time index has not yet be initialized RB=1431408 BUG=LIKAFKA-19361 G=Kafka-Code-Reviews R=jkoshy,jonlee A=jkoshy,jonlee == This is the commit #2 [c53fffd] == [LI-HOTFIX] Update fetcher thread idle flag in addPartitions TICKET = KAFKA-8667 LI_DESCRIPTION = This patch fixes in bug introduced by “[LI-HOTFIX] Reduce lock retention and improve broker shutdown time” HOTFIX where the fetcher thread idle flag is not set in addPartitions, which can cause idle fetcher thread not shutdown in time. RB=1431408 BUG=LIKAFKA-19361 G=Kafka-Code-Reviews R=jkoshy,jonlee A=jkoshy,jonlee
lmr3796
pushed a commit
that referenced
this pull request
Feb 9, 2022
TICKET = [KAFKA-8667, KAFKA-8668] LI_DESCRIPTION = - Avoid acquiring partitionMap lock in shutdownIdleFetcherThread - ~~Avoid appending to the time index during shutdown if the time index has not yet be initialized~~ This is covered in apache#8346 and apache#10960 EXIT_CRITERIA = TICKET [KAFKA-8667, KAFKA-8668] The patch is a sqaush of the 2 commits: == This is the 1st commit [155b4f8] == [LI-HOTFIX] Reduce lock retention and improve broker shutdown time: TICKET = [KAFKA-8667, KAFKA-8668] LI_DESCRIPTION = - Avoid acquiring partitionMap lock in shutdownIdleFetcherThread - Avoid appending to the time index during shutdown if the time index has not yet be initialized RB=1431408 BUG=LIKAFKA-19361 G=Kafka-Code-Reviews R=jkoshy,jonlee A=jkoshy,jonlee == This is the commit #2 [c53fffd] == [LI-HOTFIX] Update fetcher thread idle flag in addPartitions TICKET = KAFKA-8667 LI_DESCRIPTION = This patch fixes in bug introduced by “[LI-HOTFIX] Reduce lock retention and improve broker shutdown time” HOTFIX where the fetcher thread idle flag is not set in addPartitions, which can cause idle fetcher thread not shutdown in time. RB=1431408 BUG=LIKAFKA-19361 G=Kafka-Code-Reviews R=jkoshy,jonlee A=jkoshy,jonlee
lmr3796
pushed a commit
that referenced
this pull request
Mar 25, 2022
…eiving offset commit exception. TICKET = LI_DESCRIPTION = EXIT_CRITERIA = MANUAL ["needed until we make a long-term fix"]
lmr3796
pushed a commit
that referenced
this pull request
Mar 25, 2022
TICKET = [KAFKA-8667, KAFKA-8668] LI_DESCRIPTION = - Avoid acquiring partitionMap lock in shutdownIdleFetcherThread - ~~Avoid appending to the time index during shutdown if the time index has not yet be initialized~~ This is covered in apache#8346 and apache#10960 EXIT_CRITERIA = TICKET [KAFKA-8667, KAFKA-8668] The patch is a sqaush of the 2 commits: == This is the 1st commit [155b4f8] == [LI-HOTFIX] Reduce lock retention and improve broker shutdown time: TICKET = [KAFKA-8667, KAFKA-8668] LI_DESCRIPTION = - Avoid acquiring partitionMap lock in shutdownIdleFetcherThread - Avoid appending to the time index during shutdown if the time index has not yet be initialized RB=1431408 BUG=LIKAFKA-19361 G=Kafka-Code-Reviews R=jkoshy,jonlee A=jkoshy,jonlee == This is the commit #2 [c53fffd] == [LI-HOTFIX] Update fetcher thread idle flag in addPartitions TICKET = KAFKA-8667 LI_DESCRIPTION = This patch fixes in bug introduced by “[LI-HOTFIX] Reduce lock retention and improve broker shutdown time” HOTFIX where the fetcher thread idle flag is not set in addPartitions, which can cause idle fetcher thread not shutdown in time. RB=1431408 BUG=LIKAFKA-19361 G=Kafka-Code-Reviews R=jkoshy,jonlee A=jkoshy,jonlee
lmr3796
pushed a commit
that referenced
this pull request
Jun 2, 2022
…eiving offset commit exception TICKET = LI_DESCRIPTION = EXIT_CRITERIA = MANUAL ["needed until we make a long-term fix"]
lmr3796
pushed a commit
that referenced
this pull request
Jun 2, 2022
TICKET = [KAFKA-8667, KAFKA-8668] LI_DESCRIPTION = - Avoid acquiring partitionMap lock in shutdownIdleFetcherThread - ~~Avoid appending to the time index during shutdown if the time index has not yet be initialized~~ This is covered in apache#8346 and apache#10960 EXIT_CRITERIA = TICKET [KAFKA-8667, KAFKA-8668] The patch is a sqaush of the 2 commits: == This is the 1st commit [155b4f8] == [LI-HOTFIX] Reduce lock retention and improve broker shutdown time: TICKET = [KAFKA-8667, KAFKA-8668] LI_DESCRIPTION = - Avoid acquiring partitionMap lock in shutdownIdleFetcherThread - Avoid appending to the time index during shutdown if the time index has not yet be initialized RB=1431408 BUG=LIKAFKA-19361 G=Kafka-Code-Reviews R=jkoshy,jonlee A=jkoshy,jonlee == This is the commit #2 [c53fffd] == [LI-HOTFIX] Update fetcher thread idle flag in addPartitions TICKET = KAFKA-8667 LI_DESCRIPTION = This patch fixes in bug introduced by “[LI-HOTFIX] Reduce lock retention and improve broker shutdown time” HOTFIX where the fetcher thread idle flag is not set in addPartitions, which can cause idle fetcher thread not shutdown in time. RB=1431408 BUG=LIKAFKA-19361 G=Kafka-Code-Reviews R=jkoshy,jonlee A=jkoshy,jonlee
bringhurst
pushed a commit
to bringhurst/kafka
that referenced
this pull request
Mar 16, 2026
…ker provisioning (apache#21394) ## Summary Fixes bugs where `--jdk-version` and `--jdk-arch` parameters were ignored during system test worker provisioning, and refactors `vagrant/base.sh` to support flexible JDK versions without code changes. --- ## Problem The Vagrant provisioning script (`vagrant/base.sh`) had two bugs that caused JDK version parameters to be ignored: | Bug | Problem | |-----|---------| | **linkedin#1: `--jdk-version` ignored** | `JDK_FULL` was hardcoded to `17-linux-x64`, so passing `--jdk-version 25` still downloaded JDK 17 | | **linkedin#2: `--jdk-arch` ignored** | Architecture parameter was passed but never used in the S3 download URL | --- ## Solution - Validate `JDK_MAJOR` and `JDK_ARCH` input parameters with regex - Dynamically construct `JDK_FULL` from `JDK_MAJOR` and `JDK_ARCH` - Update S3 path to use `/jdk/` subdirectory - Add logging for debugging --- ## Changes ### `vagrant/base.sh` | Change | Description | |--------|-------------| | **Input validation** | Added regex validation for `JDK_MAJOR` and `JDK_ARCH` with sensible defaults | | **Dynamic construction** | `JDK_FULL` is now constructed from `JDK_MAJOR` and `JDK_ARCH` if not explicitly provided | | **Updated S3 path** | Changed URL from `/kafka-packages/jdk-{version}.tar.gz` to `/kafka-packages/jdk/jdk-{version}.tar.gz` | | **Logging** | Added debug output for JDK configuration | | **Backward compatibility** | Vagrantfile can still pass `JDK_FULL` directly; the script validates and uses it if valid | --- ## S3 Path Change ### Old Path ``` s3://kafka-packages/jdk-{version}.tar.gz ``` ### New Path ``` s3://kafka-packages/jdk/jdk-{version}.tar.gz ``` ### Available JDKs in `s3://kafka-packages/jdk/` | File | Version | Architecture | |------|---------|--------------| | `jdk-7u80-linux-x64.tar.gz` | 7u80 | x64 | | `jdk-8u144-linux-x64.tar.gz` | 8u144 | x64 | | `jdk-8u161-linux-x64.tar.gz` | 8u161 | x64 | | `jdk-8u171-linux-x64.tar.gz` | 8u171 | x64 | | `jdk-8u191-linux-x64.tar.gz` | 8u191 | x64 | | `jdk-8u202-linux-x64.tar.gz` | 8u202 | x64 | | `jdk-11.0.2-linux-x64.tar.gz` | 11.0.2 | x64 | | `jdk-17-linux-x64.tar.gz` | 17 | x64 | | `jdk-18.0.2-linux-x64.tar.gz` | 18.0.2 | x64 | | `jdk-21.0.1-linux-x64.tar.gz` | 21.0.1 | x64 | | `jdk-21.0.1-linux-aarch64.tar.gz` | 21.0.1 | aarch64 | | `jdk-25-linux-x64.tar.gz` | 25 | x64 | | `jdk-25-linux-aarch64.tar.gz` | 25 | aarch64 | | `jdk-25.0.1-linux-x64.tar.gz` | 25.0.1 | x64 | | `jdk-25.0.1-linux-aarch64.tar.gz` | 25.0.1 | aarch64 | | `jdk-25.0.2-linux-x64.tar.gz` | 25.0.2 | x64 | | `jdk-25.0.2-linux-aarch64.tar.gz` | 25.0.2 | aarch64 | --- ## Future JDK Releases > **IMPORTANT: No code changes required for future Java major/minor releases!** The validation regex supports all version formats: - **Major versions**: `17`, `25`, `26` - **Minor versions**: `25.0.1`, `25.0.2`, `26.0.1` - **Legacy format**: `8u144`, `8u202` ### Adding New JDK Versions To add support for a new JDK version (e.g., JDK 26, 25.0.3): 1. Download the JDK tarball from Oracle/Adoptium 2. Rename to follow naming convention: `jdk-{VERSION}-linux-{ARCH}.tar.gz` 3. Upload to S3: `aws s3 cp jdk-{VERSION}-linux-{ARCH}.tar.gz s3://kafka-packages/jdk/` 4. Use in tests: `--jdk-version {VERSION} --jdk-arch {ARCH}` No modifications to `base.sh` or any other scripts are needed. --- ## Benefits | Before | After | |--------|-------| | `--jdk-version` ignored | ✅ Correctly uses specified version | | `--jdk-arch` ignored | ✅ Correctly uses specified architecture | | Only major version support | ✅ Full version support (e.g., `25.0.2`) | | Code change needed for new JDK | ✅ Just upload to S3 and pass version | --- ## Testing Tested with different JDK versions to confirm the fix works correctly: | Test | JDK_MAJOR | Expected | Actual | Result | Test Report | |------|-----------|----------|--------|--------|-------------| | JDK 17 | `17` | javac 17.0.4 | javac 17.0.4 | ✅ | | JDK 25 | `25` | javac 25.0.2 | javac 25.0.2 | ✅ | --- ## Backward Compatibility - **Vagrantfile**: Continues to work as before - **Existing workflows**: Default behavior unchanged (JDK 17 on x64 architecture) - **No breaking changes**: All existing configurations continue to work --- Reviewers: Manikumar Reddy <manikumar.reddy@gmail.com>, Chia-Ping Tsai <chia7712@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Travis will kick off a build and publish artifacts to bintray upon creating a tag in the "x.y.z.w" format.
Removed system test related settings.