Gradle build - some submodules are skipped #2850

nicoloboschi · 2021-10-25T15:57:50Z

Motivation

In order to complete migration to Gradle we must build all the subprojects.

Changes

Enabled sh integration tests with gradle, located in tests/scripts/src/test/bash/gradle
Added these modules to the build
- bookkeeper-http:servlet-http-server
- metadata-drivers:etcd
- tests:backward-compat:*
- tests:shaded:*
- stream:bk-grpc-name-resolver
DL shading process is now performed (before it didn't build any jar)
Groovy tests (tests:backward-compat:*) now are triggered by the build/tests itself; with Maven, there is a "runner" project (tests/integration-tests-base-groovy); in Gradle is useless so it is skipped

Test

Both bin/bookkeper standalone and bin/bookkeper_gradle standalone work locally
Tests are passing locally

Master Issue: #2849

bin/common.sh

tests/backward-compat/build.gradle

nicoloboschi · 2021-10-26T10:22:31Z

@eolivelli @pkumar-singh @hsaputra PTAL

bin/common_gradle.sh

pkumar-singh · 2021-10-26T17:26:53Z

bin/common_gradle.sh

    echo ${BK_CLASSPATH}
  else
    add_maven_deps_to_classpath ${MODULE_PATH} >&2
-    cat ${BK_HOME}/${MODULE_PATH}/build/classpath.txt


This file is generated here. So If you need to change the name you have to change it there too.?
But I do not see any reason to change. Because we are not caching anything, with gradle. Gradle does the caching though.
https://github.com/apache/bookkeeper/blob/master/bookkeeper-server/build.gradle#L84

I did the change because there was a sh test which checks the cached_classpath.txt file.
However I agree to duplicate sh tests in this case to facilitate maven removal, so I'm going to revert this change

tests/scripts/src/test/bash/bk_test_bin_common.sh

…to faciltate migration

nicoloboschi · 2021-10-27T07:57:12Z

@pkumar-singh I created a new directory with the shell tests only for gradle. When we'll have to remove maven, we only have to drop the root directory and moves up the gradle one.

nicoloboschi · 2021-10-27T09:09:42Z

rerun failure checks

hsaputra · 2021-10-27T21:15:03Z

@nicoloboschi - I suppose you will update this PR from comments that @pkumar-singh have added?

Thanks

nicoloboschi · 2021-10-28T07:35:23Z

@hsaputra I already addressed Pkumar's comments. I updated the description of the PR. I think we are ready to merge it, if you like

hsaputra

LGTM +1

hsaputra · 2021-10-28T17:31:30Z

Will merge this EOD PST if no objection or more comments. Thanks for the contribution @nicoloboschi !

hsaputra · 2021-10-29T00:44:40Z

Will merge this one. We can iteratively add updates if anything a miss.

### Motivation In order to complete migration to Gradle we must build all the subprojects. ### Changes - Enabled `sh` integration tests with gradle, located in `tests/scripts/src/test/bash/gradle` - Added these modules to the build - `bookkeeper-http:servlet-http-server` - `metadata-drivers:etcd` - `tests:backward-compat:*` - `tests:shaded:*` - `stream:bk-grpc-name-resolver` - DL shading process is now performed (before it didn't build any jar) - Groovy tests (`tests:backward-compat:*`) now are triggered by the build/tests itself; with Maven, there is a "runner" project (`tests/integration-tests-base-groovy`); in Gradle is useless so it is skipped ### Test - Both `bin/bookkeper standalone` and `bin/bookkeper_gradle standalone` work locally - Tests are passing locally Master Issue: apache#2849 Reviewers: Henry Saputra <hsaputra@apache.org>, Prashant Kumar <None> This closes apache#2850 from nicoloboschi/fix/2849/gradle and squashes the following commits: 00b49f4 [Nicolò Boschi] Fix common_gradle.sh regex bd739fd [Nicolò Boschi] fix sh tests 43230ba [Nicolò Boschi] revert sh files. Avoid to modify maven files, create gradle versions to faciltate migration d1f95e4 [Nicolò Boschi] fix shaded deps bcab40d [Nicolò Boschi] fix build 5fd0341 [Nicolò Boschi] fix build 0082e0e [Nicolò Boschi] fix build 2c32ac1 [Nicolò Boschi] fixes 3bc0b26 [Nicolò Boschi] bookkeeper-server-shaded-tests ba89132 [Nicolò Boschi] shaded tests 6d39e33 [Nicolò Boschi] sh tests e0032bc [Nicolò Boschi] actually run arquillian groovy tests 08dcc39 [Nicolò Boschi] backwards 2361f79 [Nicolò Boschi] hierarchical-ledger-manager 8388e11 [Nicolò Boschi] current-server-old-clients 6a24344 [Nicolò Boschi] bc-non-fips 2faca01 [Nicolò Boschi] bk-grpc-name-resolver 991bc11 [Nicolò Boschi] servlet-http-server 675ef7b [Nicolò Boschi] etcd b1d5e14 [ZhangJian He] A empty implement in EtcdLedgerManagerFactory to let the project can compile (apache#2845) bd5c50b [shustsud] Add error handling to readLedgerMetadata in over-replicated ledger GC (apache#2844) 746f9f6 [Matteo Merli] Remove direct ZK access for Auditor (apache#2842) 4117200 [ZhangJian He] the compare should be >= instead of > (apache#2782) 14ef56f [Prashant Kumar] BookieId can not be cast to BookieSocketAddress (apache#2843) e10f3fe [ZhangJian He] Forget to close preAllocator log on shutdown (apache#2819) 53954ca [shustsud] Add ensemble check to over-replicated ledger GC (apache#2813) 919fdd3 [Prashant Kumar] Issue:2840 Create bookie shellscript for gradle (apache#2841) 031d168 [gaozhangmin] fix-npe-when-pulsar-ZkBookieRackAffinityMapping-getBookieAddressResolver (apache#2788) 3dd671c [Prashant Kumar] Migrate bookkeepr-server:test to gradle run unit tests excepts org.apache.bookkeeper.bookie. org.apache.bookkeeper.client org.apache.bookkeeper.replication org.apache.bookkeeper.tls. (apache#2812) f6903b8 [Jack Vanlightly] BP-44 USE metrics a4afaa4 [Matteo Merli] Eliminate direct ZK access in ScanAndCompareGarbageCollector (apache#2833) a9b576d [Yunze Xu] Release semaphore when addEntry accepts the same entries (apache#2832) 148bf22 [Yun Tang] Ensure to release cache during KeyValueStorageRocksDB#closec (apache#2821) 4dc4260 [gaozhangmin] Heap memory leak problem when ledger replication failed (apache#2794) a522fa3 [Raúl Gracia] Issue 2815: Upgrade to log4j2 to get rid of CVE-2019-17571 (apache#2816) 0465052 [Nicolò Boschi] Upgrade httpclient from 4.5.5 to 4.5.13 (apache#2793) 594a056 [Raúl Gracia] Issue 2795: Bookkeeper upgrade using Bookie ID may fail due to cookie mismatch (apache#2796) 354cf37 [Raúl Gracia] Upgraded dependencies with CVEs (apache#2792) e413c70 [Raúl Gracia] Issue 2728: Entry Log GC may get blocked when using entryLogPerLedgerEnabled option (apache#2779) 883231e [pradeepbn] Building bookkeeper with gradle on java11

nicoloboschi added 13 commits October 25, 2021 12:05

etcd

675ef7b

servlet-http-server

991bc11

bk-grpc-name-resolver

2faca01

bc-non-fips

6a24344

current-server-old-clients

8388e11

hierarchical-ledger-manager

2361f79

backwards

08dcc39

actually run arquillian groovy tests

e0032bc

sh tests

6d39e33

shaded tests

ba89132

bookkeeper-server-shaded-tests

3bc0b26

fixes

2c32ac1

fix build

0082e0e

nicoloboschi marked this pull request as draft October 25, 2021 15:57

fix build

5fd0341

pkumar-singh reviewed Oct 25, 2021

View reviewed changes

bin/common.sh Outdated Show resolved Hide resolved

pkumar-singh reviewed Oct 25, 2021

View reviewed changes

tests/backward-compat/build.gradle Show resolved Hide resolved

nicoloboschi added 2 commits October 26, 2021 10:56

fix build

bcab40d

fix shaded deps

d1f95e4

nicoloboschi marked this pull request as ready for review October 26, 2021 10:21

nicoloboschi changed the title ~~(WIP) Gradle build - some submodules are skipped~~ Gradle build - some submodules are skipped Oct 26, 2021

pkumar-singh requested changes Oct 26, 2021

View reviewed changes

bin/common_gradle.sh Outdated Show resolved Hide resolved

pkumar-singh reviewed Oct 26, 2021

View reviewed changes

bin/common_gradle.sh Outdated Show resolved Hide resolved

pkumar-singh reviewed Oct 26, 2021

View reviewed changes

tests/scripts/src/test/bash/bk_test_bin_common.sh Outdated Show resolved Hide resolved

nicoloboschi added 2 commits October 27, 2021 09:47

revert sh files. Avoid to modify maven files, create gradle versions …

43230ba

…to faciltate migration

fix sh tests

bd739fd

Fix common_gradle.sh regex

00b49f4

pkumar-singh approved these changes Oct 27, 2021

View reviewed changes

hsaputra assigned nicoloboschi Oct 27, 2021

hsaputra self-requested a review October 27, 2021 21:15

pkumar-singh approved these changes Oct 28, 2021

View reviewed changes

hsaputra approved these changes Oct 28, 2021

View reviewed changes

hsaputra added area/build type/bug labels Oct 28, 2021

hsaputra added this to the 4.15.0 milestone Oct 28, 2021

hsaputra added the release/4.15.0 label Oct 29, 2021

hsaputra merged commit 98ddf81 into apache:master Oct 29, 2021

Gradle build - some submodules are skipped #2850

Gradle build - some submodules are skipped #2850

Uh oh!

Conversation

nicoloboschi commented Oct 25, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Changes

Test

Uh oh!

Uh oh!

Uh oh!

nicoloboschi commented Oct 26, 2021

Uh oh!

Uh oh!

Uh oh!

pkumar-singh Oct 26, 2021

Choose a reason for hiding this comment

Uh oh!

nicoloboschi Oct 27, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nicoloboschi commented Oct 27, 2021

Uh oh!

nicoloboschi commented Oct 27, 2021

Uh oh!

hsaputra commented Oct 27, 2021

Uh oh!

nicoloboschi commented Oct 28, 2021

Uh oh!

hsaputra left a comment

Choose a reason for hiding this comment

Uh oh!

hsaputra commented Oct 28, 2021

Uh oh!

hsaputra commented Oct 29, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

nicoloboschi commented Oct 25, 2021 •

edited

Loading

nicoloboschi Oct 27, 2021 •

edited

Loading