Skip to content

Fix the leadership integration test with Java 11#10846

Merged
jihoonson merged 1 commit intoapache:masterfrom
jihoonson:bigger-heap-for-custom-node
Feb 4, 2021
Merged

Fix the leadership integration test with Java 11#10846
jihoonson merged 1 commit intoapache:masterfrom
jihoonson:bigger-heap-for-custom-node

Conversation

@jihoonson
Copy link
Copy Markdown
Contributor

Description

The leadership integration test starts failing with Java 11 such as in here. It is because of the custom node failed with OOM.

java.lang.OutOfMemoryError: Java heap space
Dumping heap to /tmp/java_pid906.hprof ...
Heap dump file created [46020040 bytes in 0.143 secs]
Terminating due to java.lang.OutOfMemoryError: Java heap space

This PR increases the heap from 32m to 64m.


This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@jihoonson
Copy link
Copy Markdown
Contributor Author

jihoonson commented Feb 4, 2021

As this issue is another blocker for 0.21.0, I couldn't wait for Travis to run, so went ahead and ran the same jobs on the internal CI of Imply. All tests passed except for (Compile=openjdk8, Run=openjdk11) input source integration test, which seems like a temporary connection issue. This test uses https://druid.apache.org/data/wikipedia.json.gz. The detailed error is in below. The same test passed on my local machine.

java.lang.RuntimeException: java.net.ConnectException: Connection timed out (Connection timed out)
	at org.apache.druid.data.input.impl.InputEntityIteratingReader.lambda$read$0(InputEntityIteratingReader.java:81) ~[druid-core-0.22.0-iap1-SNAPSHOT.jar:0.22.0-iap1-SNAPSHOT]
	at org.apache.druid.java.util.common.parsers.CloseableIterator$2.findNextIteratorIfNecessary(CloseableIterator.java:84) ~[druid-core-0.22.0-iap1-SNAPSHOT.jar:0.22.0-iap1-SNAPSHOT]
	at org.apache.druid.java.util.common.parsers.CloseableIterator$2.<init>(CloseableIterator.java:69) ~[druid-core-0.22.0-iap1-SNAPSHOT.jar:0.22.0-iap1-SNAPSHOT]
	at org.apache.druid.java.util.common.parsers.CloseableIterator.flatMap(CloseableIterator.java:67) ~[druid-core-0.22.0-iap1-SNAPSHOT.jar:0.22.0-iap1-SNAPSHOT]
	at org.apache.druid.data.input.impl.InputEntityIteratingReader.createIterator(InputEntityIteratingReader.java:103) ~[druid-core-0.22.0-iap1-SNAPSHOT.jar:0.22.0-iap1-SNAPSHOT]
...

@suneet-s
Copy link
Copy Markdown
Contributor

suneet-s commented Feb 4, 2021

+1 on not waiting for travis on this change

@jihoonson
Copy link
Copy Markdown
Contributor Author

@jon-wei @clintropolis @suneet-s @maytasm thank you all!

@jihoonson jihoonson merged commit 397e745 into apache:master Feb 4, 2021
jihoonson added a commit to jihoonson/druid that referenced this pull request Feb 4, 2021
jon-wei added a commit to jon-wei/druid that referenced this pull request Feb 11, 2021
* move integration tests from ZooKeeper 3.4.x to 3.5.x (apache#10786)

* move integration tests from ZooKeeper 3.4.x to 3.5.x
* run a subset of our integration tests with ZK 3.4 for backwards compatibility testing.
* remove need to build separate docker-base image
- use multi-stage build for the base image
- use openjdk base image instead of building our own JDK base
- workaround Debian not including MySQL by using MariaDB
- download mysql connector directly instead of using distro version
* fix incorrect openssl command failing on Debian
* keep mysql connector version in sync with pom version

* K8s IT Test enhance  (apache#10785)

* do build and stop action in IT

* change base dir from druidHome to druidHome/integration-tests

* add env DRUID_HOME

* bug fix

* modify stop_sh

* ready to test

* bug fix

* modify dir

* tested on dev

* modify dir

* move DRUID_HOME env

* done

Co-authored-by: yuezhang <yuezhang@freewheel.tv>

* Update NOTICE copyright year (apache#10834)

the future is now

* Cleanup openssl fixes to keep certs

* Address CVE-2020-8570, suppress CVE-2020-8554 (apache#10826)

* Address CVE-2020-8570, suppress CVE-2020-8554

* Update licenses.yaml

* wget debug

* Suppress CVE-2020-9492 for hadoop-mapreduce-client-core (apache#10847)

* Revert "wget debug"

This reverts commit 5b81c33b4728420e2312b3c919b7de9c1b4da589.

* Add MYSQL_VERSION env variable in integration-tests-imply tests

* Increase heap to 64m for custom node (apache#10846)

* Fix CVE-2021-25646 (apache#10818)

* Add ZK_VERSION env variable

Co-authored-by: Xavier Léauté <xvrl@apache.org>
Co-authored-by: zhangyue19921010 <69956021+zhangyue19921010@users.noreply.github.com>
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
Co-authored-by: Clint Wylie <cwylie@apache.org>
Co-authored-by: Slava Mogilevsky <triggerwoods91@gmail.com>
Co-authored-by: Jihoon Son <jihoonson@apache.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants