Skip to content

[hotfix] Add flink1 profile to deploy and publish process #5461

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
yuzelin merged 3 commits into from
Apr 14, 2025
Merged

[hotfix] Add flink1 profile to deploy and publish process #5461

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
b5c6c58
[hotfix] Add flink1 profile to deploy and publish process
yunfengzhou-hub Apr 14, 2025
bb94c13
[hotfix] Fix missing profile of other engines
yunfengzhou-hub Apr 14, 2025
c64a85f
[hotfix] initialize paimon-flink-cdc in Flink 2.x CI
yunfengzhou-hub Apr 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/workflows/e2e-tests-flink-1.x-jdk11.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ jobs:
distribution: 'temurin'

- name: Build Flink
run: mvn -T 1C -B clean install -DskipTests -Pflink1 -pl paimon-e2e-tests -am -Pflink-${{ matrix.flink_version }},java11
run: mvn -T 1C -B clean install -DskipTests -Pflink1,spark3 -pl paimon-e2e-tests -am -Pflink-${{ matrix.flink_version }},java11

- name: Test Flink
run: |
Expand All @@ -62,9 +62,9 @@ jobs:
echo "JVM timezone is set to $jvm_timezone"
profile="flink-${{ matrix.flink_version }}"
if [ "${{ matrix.flink_version }}" = "${{ matrix.flink_version[-1] }}" ]; then
mvn -T 1C -B test -Pflink1 -pl paimon-e2e-tests -Duser.timezone=$jvm_timezone -Pjava11
mvn -T 1C -B test -Pflink1,spark3 -pl paimon-e2e-tests -Duser.timezone=$jvm_timezone -Pjava11
else
mvn -T 1C -B test -Pflink1 -pl paimon-e2e-tests -Duser.timezone=$jvm_timezone -P${profile},java11
mvn -T 1C -B test -Pflink1,spark3 -pl paimon-e2e-tests -Duser.timezone=$jvm_timezone -P${profile},java11
fi
env:
MAVEN_OPTS: -Xmx4096m
6 changes: 3 additions & 3 deletions .github/workflows/e2e-tests-flink-1.x.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ jobs:
distribution: 'temurin'

- name: Build Flink
run: mvn -T 2C -B clean install -DskipTests -Pflink1 -pl paimon-e2e-tests -am -Pflink-${{ matrix.flink_version }}
run: mvn -T 2C -B clean install -DskipTests -Pflink1,spark3 -pl paimon-e2e-tests -am -Pflink-${{ matrix.flink_version }}

- name: Test Flink
run: |
Expand All @@ -62,9 +62,9 @@ jobs:
echo "JVM timezone is set to $jvm_timezone"
profile="flink-${{ matrix.flink_version }}"
if [ "${{ matrix.flink_version }}" = "${{ matrix.flink_version[-1] }}" ]; then
mvn -T 1C -B test -Pflink1 -pl paimon-e2e-tests -Duser.timezone=$jvm_timezone
mvn -T 1C -B test -Pflink1,spark3 -pl paimon-e2e-tests -Duser.timezone=$jvm_timezone
else
mvn -T 1C -B test -Pflink1 -pl paimon-e2e-tests -Duser.timezone=$jvm_timezone -P${profile}
mvn -T 1C -B test -Pflink1,spark3 -pl paimon-e2e-tests -Duser.timezone=$jvm_timezone -P${profile}
fi
env:
MAVEN_OPTS: -Xmx4096m
8 changes: 5 additions & 3 deletions .github/workflows/e2e-tests-flink-2.x-jdk11.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,9 @@ jobs:
distribution: 'temurin'

- name: Build Flink
run: mvn -T 1C -B clean install -DskipTests -Pflink2 -pl paimon-e2e-tests -am -Pflink-${{ matrix.flink_version }},java11
run: |
mvn -T 1C -B clean install -DskipTests -Pflink1,spark3 -pl paimon-e2e-tests -am -Pflink-${{ matrix.flink_version }},java11
mvn -T 1C -B clean install -DskipTests -Pflink2,spark3 -pl paimon-e2e-tests -am -Pflink-${{ matrix.flink_version }},java11

- name: Test Flink
run: |
Expand All @@ -58,9 +60,9 @@ jobs:
echo "JVM timezone is set to $jvm_timezone"
profile="flink-${{ matrix.flink_version }}"
if [ "${{ matrix.flink_version }}" = "${{ matrix.flink_version[-1] }}" ]; then
mvn -T 1C -B test -Pflink2 -pl paimon-e2e-tests -Duser.timezone=$jvm_timezone -Pjava11
mvn -T 1C -B test -Pflink2,spark3 -pl paimon-e2e-tests -Duser.timezone=$jvm_timezone -Pjava11
else
mvn -T 1C -B test -Pflink2 -pl paimon-e2e-tests -Duser.timezone=$jvm_timezone -P${profile},java11
mvn -T 1C -B test -Pflink2,spark3 -pl paimon-e2e-tests -Duser.timezone=$jvm_timezone -P${profile},java11
fi
env:
MAVEN_OPTS: -Xmx4096m
2 changes: 1 addition & 1 deletion .github/workflows/publish_snapshot.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,6 @@ jobs:
echo "<password>$ASF_PASSWORD</password>" >> $tmp_settings
echo "</server></servers></settings>" >> $tmp_settings

mvn --settings $tmp_settings clean deploy -Dgpg.skip -Drat.skip -DskipTests -Papache-release,spark3
mvn --settings $tmp_settings clean deploy -Dgpg.skip -Drat.skip -DskipTests -Papache-release,spark3,flink1

rm $tmp_settings
4 changes: 2 additions & 2 deletions .github/workflows/utitcase-flink-1.x-jdk11.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ jobs:
java-version: ${{ env.JDK_VERSION }}
distribution: 'temurin'
- name: Build Flink
run: mvn -T 1C -B clean install -DskipTests -Pflink1
run: mvn -T 1C -B clean install -DskipTests -Pflink1,spark3
- name: Test Flink
run: |
# run tests with random timezone to find out timezone related bugs
Expand All @@ -57,6 +57,6 @@ jobs:
test_modules+="org.apache.paimon:paimon-flink-${suffix},"
done
test_modules="${test_modules%,}"
mvn -T 1C -B clean install -Pflink1 -pl "${test_modules}" -Duser.timezone=$jvm_timezone
mvn -T 1C -B clean install -Pflink1,spark3 -pl "${test_modules}" -Duser.timezone=$jvm_timezone
env:
MAVEN_OPTS: -Xmx4096m
4 changes: 2 additions & 2 deletions .github/workflows/utitcase-flink-1.x.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@ jobs:
run: |
COMPILE_MODULE="org.apache.paimon:paimon-flink-${{ matrix.flink_version }}"
echo "Start compiling modules: $COMPILE_MODULE"
mvn -T 2C -B clean install -DskipTests -Pflink1 -pl "${COMPILE_MODULE}" -am
mvn -T 2C -B clean install -DskipTests -Pflink1,spark3 -pl "${COMPILE_MODULE}" -am

- name: Test Flink
run: |
Expand All @@ -63,7 +63,7 @@ jobs:
echo "JVM timezone is set to $jvm_timezone"
TEST_MODULE="org.apache.paimon:paimon-flink-${{ matrix.flink_version }}"
echo "Start testing module: $TEST_MODULE"
mvn -T 2C -B test verify -Pflink1 -pl "${TEST_MODULE}" -Duser.timezone=$jvm_timezone
mvn -T 2C -B test verify -Pflink1,spark3 -pl "${TEST_MODULE}" -Duser.timezone=$jvm_timezone
echo "All modules tested"
env:
MAVEN_OPTS: -Xmx4096m -XX:+UseG1GC -XX:CICompilerCount=2
6 changes: 4 additions & 2 deletions .github/workflows/utitcase-flink-2.x-jdk11.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,9 @@ jobs:
java-version: ${{ env.JDK_VERSION }}
distribution: 'temurin'
- name: Build Flink
run: mvn -T 1C -B clean install -DskipTests -Pflink2
run: |
mvn -T 1C -B clean install -DskipTests -Pflink1,spark3
mvn -T 1C -B clean install -DskipTests -Pflink2,spark3
- name: Test Flink
run: |
# run tests with random timezone to find out timezone related bugs
Expand All @@ -53,6 +55,6 @@ jobs:
test_modules+="org.apache.paimon:paimon-flink-${suffix},"
done
test_modules="${test_modules%,}"
mvn -T 1C -B clean install -Pflink2 -pl "${test_modules}" -Duser.timezone=$jvm_timezone
mvn -T 1C -B clean install -Pflink2,spark3 -pl "${test_modules}" -Duser.timezone=$jvm_timezone
env:
MAVEN_OPTS: -Xmx4096m
4 changes: 2 additions & 2 deletions .github/workflows/utitcase-spark-4.x.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ jobs:
java-version: ${{ env.JDK_VERSION }}
distribution: 'temurin'
- name: Build Spark
run: mvn -T 2C -B clean install -DskipTests -Pspark4
run: mvn -T 2C -B clean install -DskipTests -Pspark4,flink1
- name: Test Spark
timeout-minutes: 60
run: |
Expand All @@ -58,6 +58,6 @@ jobs:
test_modules+="org.apache.paimon:paimon-spark-${suffix},"
done
test_modules="${test_modules%,}"
mvn -T 2C -B test -pl "${test_modules}" -Duser.timezone=$jvm_timezone -Pspark4
mvn -T 2C -B test -pl "${test_modules}" -Duser.timezone=$jvm_timezone -Pspark4,flink1
env:
MAVEN_OPTS: -Xmx4096m
4 changes: 2 additions & 2 deletions paimon-e2e-tests/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,11 +32,11 @@ Currently, e2e supports docker environment only. You need to do some preparation
1. Build with Flink-versioned profiles, like flink-1.17. The default Flink main version is declared in the parent `pom.xml` by `<test.flink.main.version>` tag.

```bash
mvn clean install -DskipTests -Pflink-1.17
mvn clean install -DskipTests -Pflink1,flink-1.17,spark3
```

2. Run e2e tests

```bash
mvn test -pl paimon-e2e-tests -Pflink-1.17
mvn test -pl paimon-e2e-tests -Pflink1,flink-1.17,spark3
```
2 changes: 1 addition & 1 deletion tools/releasing/deploy_staging_jars.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,6 @@ fi
cd ${PROJECT_ROOT}

echo "Deploying to repository.apache.org"
${MVN} clean deploy -Papache-release,docs-and-source,spark3 -DskipTests -DretryFailedDeploymentCount=10 $CUSTOM_OPTIONS
${MVN} clean deploy -Papache-release,docs-and-source,spark3,flink1 -DskipTests -DretryFailedDeploymentCount=10 $CUSTOM_OPTIONS

cd ${CURR_DIR}
Loading