Skip to content

HDDS-10123. InaccessibleObjectException in tests using ChecksumByteBufferImpl with Java 17 #5993

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
adoroszlai merged 1 commit into from
Jan 15, 2024
Merged

HDDS-10123. InaccessibleObjectException in tests using ChecksumByteBufferImpl with Java 17 #5993

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions hadoop-ozone/dev-support/checks/junit.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ if [[ ${ITERATIONS} -le 0 ]]; then
fi

export MAVEN_OPTS="-Xmx4096m $MAVEN_OPTS"
MAVEN_OPTIONS='-B -Dskip.npx -Dskip.installnpx -Dnative.lib.tmp.dir=/tmp --no-transfer-progress'
MAVEN_OPTIONS="-B -V -Dskip.npx -Dskip.installnpx -Dnative.lib.tmp.dir=/tmp --no-transfer-progress"

if [[ "${OZONE_WITH_COVERAGE}" != "true" ]]; then
MAVEN_OPTIONS="${MAVEN_OPTIONS} -Djacoco.skip"
Expand All @@ -43,6 +43,13 @@ else
MAVEN_OPTIONS="${MAVEN_OPTIONS} --fail-at-end"
fi

# apply module access args (for Java 9+)
OZONE_MODULE_ACCESS_ARGS=""
if [[ -f hadoop-ozone/dist/src/shell/ozone/ozone-functions.sh ]]; then
source hadoop-ozone/dist/src/shell/ozone/ozone-functions.sh
ozone_java_setup
fi

if [[ "${CHECK}" == "integration" ]] || [[ ${ITERATIONS} -gt 1 ]]; then
if [[ ${OZONE_REPO_CACHED} == "false" ]]; then
mvn ${MAVEN_OPTIONS} -DskipTests clean install
Expand All @@ -61,7 +68,7 @@ for i in $(seq 1 ${ITERATIONS}); do
mkdir -p "${REPORT_DIR}"
fi

mvn ${MAVEN_OPTIONS} "$@" test \
mvn ${MAVEN_OPTIONS} -Dmaven-surefire-plugin.argLineAccessArgs="${OZONE_MODULE_ACCESS_ARGS}" "$@" test \
| tee "${REPORT_DIR}/output.log"
irc=$?

Expand Down
16 changes: 0 additions & 16 deletions hadoop-ozone/dist/src/shell/ozone/ozone
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,22 +79,6 @@ function ozonecmd_case
# This parameter significantly reduces GC pressure for Datanode.
# Corresponding Ratis issue https://issues.apache.org/jira/browse/RATIS-534.
RATIS_OPTS="-Dorg.apache.ratis.thirdparty.io.netty.allocator.useCacheForAllThreads=false ${RATIS_OPTS}"
# Add JVM parameter for Java 17
OZONE_MODULE_ACCESS_ARGS=""

# Get the version string
JAVA_VERSION_STRING=$(java -version 2>&1 | head -n 1)

# Extract the major version number
JAVA_MAJOR_VERSION=$(echo "$JAVA_VERSION_STRING" | sed -E -n 's/.* version "([^.-]*).*"/\1/p' | cut -d' ' -f1)

# populate JVM args based on java version
if [[ "${JAVA_MAJOR_VERSION}" == "17" ]]; then
OZONE_MODULE_ACCESS_ARGS="--add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.management/com.sun.jmx.mbeanserver=ALL-UNNAMED --add-exports java.management/com.sun.jmx.mbeanserver=ALL-UNNAMED"
fi
if [[ "${JAVA_MAJOR_VERSION}" -ge 9 ]]; then
OZONE_MODULE_ACCESS_ARGS="${OZONE_MODULE_ACCESS_ARGS} --add-opens java.base/java.nio=ALL-UNNAMED"
fi

case ${subcmd} in
auditparser)
Expand Down
29 changes: 29 additions & 0 deletions hadoop-ozone/dist/src/shell/ozone/ozone-functions.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1406,6 +1406,35 @@ function ozone_java_setup
ozone_error "ERROR: $JAVA is not executable."
exit 1
fi

# Get the version string
JAVA_VERSION_STRING=$(${JAVA} -version 2>&1 | head -n 1)

# Extract the major version number
JAVA_MAJOR_VERSION=$(echo "$JAVA_VERSION_STRING" | sed -E -n 's/.* version "([^.-]*).*"/\1/p' | cut -d' ' -f1)

ozone_set_module_access_args
}

## @description Set OZONE_MODULE_ACCESS_ARGS based on Java version
## @audience private
## @stability evolving
## @replaceable yes
function ozone_set_module_access_args
{
if [[ -z "${JAVA_MAJOR_VERSION}" ]]; then
return
fi

# populate JVM args based on java version
if [[ "${JAVA_MAJOR_VERSION}" -ge 17 ]]; then
OZONE_MODULE_ACCESS_ARGS="--add-opens java.base/java.lang=ALL-UNNAMED"
OZONE_MODULE_ACCESS_ARGS="${OZONE_MODULE_ACCESS_ARGS} --add-opens java.management/com.sun.jmx.mbeanserver=ALL-UNNAMED"
OZONE_MODULE_ACCESS_ARGS="${OZONE_MODULE_ACCESS_ARGS} --add-exports java.management/com.sun.jmx.mbeanserver=ALL-UNNAMED"
fi
if [[ "${JAVA_MAJOR_VERSION}" -ge 9 ]]; then
Copy link
Contributor

@sodonnel sodonnel Jan 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it intended for this branch to run for Java 17 as well as Java 9?

I think it will result in two --add-opens switches in the args:

--add-opens java.base/java.lang=ALL-UNNAMED"
--add-opens java.base/java.nio=ALL-UNNAMED"

I think this is OK, but just checking.

Copy link
Contributor Author

@adoroszlai adoroszlai Jan 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. The first part (specific to Java 17) was added for HDDS-8533. Ozone Cluster components (SCM, OM, DNs) fails to start with JDK 17. The second part got added later for HDDS-7228. ChecksumByteBufferImpl.update() is expensive. I just moved these from ozone to ozone-functions.sh.

Copy link
Contributor

@sodonnel sodonnel Jan 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool - sounds good.

OZONE_MODULE_ACCESS_ARGS="${OZONE_MODULE_ACCESS_ARGS} --add-opens java.base/java.nio=ALL-UNNAMED"
fi
}

## @description Finish Java JNI paths prior to execution
Expand Down
3 changes: 2 additions & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -252,6 +252,7 @@ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xs

<!-- Plugin versions and config -->
<maven-surefire-plugin.argLine>-Xmx4096m -XX:+HeapDumpOnOutOfMemoryError</maven-surefire-plugin.argLine>
<maven-surefire-plugin.argLineAccessArgs></maven-surefire-plugin.argLineAccessArgs>
<unstable-test-groups>flaky | slow | unhealthy</unstable-test-groups>
<maven-surefire-plugin.version>3.2.2</maven-surefire-plugin.version>
<maven-surefire-report-plugin.version>${maven-surefire-plugin.version}</maven-surefire-report-plugin.version>
Expand Down Expand Up @@ -1973,7 +1974,7 @@ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xs
<reuseForks>false</reuseForks>
<forkedProcessTimeoutInSeconds>${surefire.fork.timeout}</forkedProcessTimeoutInSeconds>
<!-- @argLine is filled by jacoco maven plugin. @{} means late evaluation -->
<argLine>${maven-surefire-plugin.argLine} @{argLine}</argLine>
<argLine>${maven-surefire-plugin.argLine} ${maven-surefire-plugin.argLineAccessArgs} @{argLine}</argLine>
<environmentVariables>
<MALLOC_ARENA_MAX>4</MALLOC_ARENA_MAX>
</environmentVariables>
Expand Down