Skip to content

K8s IT Test enhance #10785

Merged
himanshug merged 13 commits intoapache:masterfrom
zhangyue19921010:k8s-itTest-enhance
Feb 1, 2021
Merged

K8s IT Test enhance #10785
himanshug merged 13 commits intoapache:masterfrom
zhangyue19921010:k8s-itTest-enhance

Conversation

@zhangyue19921010
Copy link
Copy Markdown
Contributor

@zhangyue19921010 zhangyue19921010 commented Jan 21, 2021

Fixes #10750.

Description

do the Druid/K8s cluster creation and cleanup in integration tests instead of relying on the Travis script


This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

Key changed/added classes in this PR
  • MyFoo
  • OurBar
  • TheirBaz

@zhangyue19921010 zhangyue19921010 changed the title K8s IT Test enhance [WIP]K8s IT Test enhance Jan 21, 2021
@zhangyue19921010
Copy link
Copy Markdown
Contributor Author

Still need to deal with DIR properly

@zhangyue19921010 zhangyue19921010 changed the title [WIP]K8s IT Test enhance K8s IT Test enhance Jan 22, 2021
@zhangyue19921010
Copy link
Copy Markdown
Contributor Author

zhangyue19921010 commented Jan 22, 2021

Since Job 73 in CI is successful. I think this PR is ready for review.

Although job 66 is failed. Retry will be successful I think as it is not this pr related.

Comment thread .travis.yml Outdated
script: &run_integration_test_k8s
- ${MVN} verify -pl integration-tests -P int-tests-config-file ${IT_TEST} ${MAVEN_SKIP}
- ${MVN} verify -pl integration-tests -P int-tests-config-file ${IT_TEST} ${MAVEN_SKIP} -Dpod.name=${POD_NAME} -Dpod.namespace=${POD_NAMESPACE} -Dbuild.druid.cluster=${BUILD_DRUID_CLSUTER}
after_script: integration-tests/script/stop_k8s_cluster.sh
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we still need this after_script ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need actually. Just removed.

Comment thread .travis.yml Outdated
- integration-tests/script/setup_k8s_cluster.sh
- integration-tests/script/setup_druid_operator_on_k8s.sh
- integration-tests/script/setup_druid_on_k8s.sh
before_script: export DRUID_HOME=`pwd`
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we move this into build_run_k8s_cluster.sh so that maven based test run automatically gets it.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, done.

@zhangyue19921010
Copy link
Copy Markdown
Contributor Author

zhangyue19921010 commented Jan 26, 2021

Hi @himanshug All modifications have been completed. Thanks a lot for your review!

@himanshug himanshug merged commit 77946f9 into apache:master Feb 1, 2021
@zhangyue19921010
Copy link
Copy Markdown
Contributor Author

hi @himanshug thanks a lot for your review and merge!

@zhangyue19921010 zhangyue19921010 deleted the k8s-itTest-enhance branch February 1, 2021 23:54
jon-wei added a commit to jon-wei/druid that referenced this pull request Feb 11, 2021
* move integration tests from ZooKeeper 3.4.x to 3.5.x (apache#10786)

* move integration tests from ZooKeeper 3.4.x to 3.5.x
* run a subset of our integration tests with ZK 3.4 for backwards compatibility testing.
* remove need to build separate docker-base image
- use multi-stage build for the base image
- use openjdk base image instead of building our own JDK base
- workaround Debian not including MySQL by using MariaDB
- download mysql connector directly instead of using distro version
* fix incorrect openssl command failing on Debian
* keep mysql connector version in sync with pom version

* K8s IT Test enhance  (apache#10785)

* do build and stop action in IT

* change base dir from druidHome to druidHome/integration-tests

* add env DRUID_HOME

* bug fix

* modify stop_sh

* ready to test

* bug fix

* modify dir

* tested on dev

* modify dir

* move DRUID_HOME env

* done

Co-authored-by: yuezhang <yuezhang@freewheel.tv>

* Update NOTICE copyright year (apache#10834)

the future is now

* Cleanup openssl fixes to keep certs

* Address CVE-2020-8570, suppress CVE-2020-8554 (apache#10826)

* Address CVE-2020-8570, suppress CVE-2020-8554

* Update licenses.yaml

* wget debug

* Suppress CVE-2020-9492 for hadoop-mapreduce-client-core (apache#10847)

* Revert "wget debug"

This reverts commit 5b81c33b4728420e2312b3c919b7de9c1b4da589.

* Add MYSQL_VERSION env variable in integration-tests-imply tests

* Increase heap to 64m for custom node (apache#10846)

* Fix CVE-2021-25646 (apache#10818)

* Add ZK_VERSION env variable

Co-authored-by: Xavier Léauté <xvrl@apache.org>
Co-authored-by: zhangyue19921010 <69956021+zhangyue19921010@users.noreply.github.com>
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
Co-authored-by: Clint Wylie <cwylie@apache.org>
Co-authored-by: Slava Mogilevsky <triggerwoods91@gmail.com>
Co-authored-by: Jihoon Son <jihoonson@apache.org>
@clintropolis clintropolis added this to the 0.22.0 milestone Aug 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

k8s integration tests don't stop containers after tests

4 participants