Added local ci testing target to makefile by sflxn · Pull Request #7170 · vmware/vic

sflxn · 2018-01-25T22:06:12Z

Make testing locally as friction-free as possible by

Adding a makefile target 'local-ci-test'
Using TARGET_VCH added in VIC 1.3 to use an existing VCH
Using a custom script that doesn't utilize drone so that if
the test fails and craters, we can still access the logs

Resolves #7162

hickeng · 2018-01-27T06:22:05Z

Makefile

+	@echo To run ci tests, you must set TARGET_VCH to an already deployed VCH
+else
+	@echo running ci-tests locally...
+	infra/scripts/local-ci.sh ./test.secrets.nested


Use $(BASE_DIR)/infra/scripts/ as it allows calling from directories other than the build

hickeng · 2018-01-27T06:23:23Z

infra/scripts/local-ci.sh

+username='nloc'
+secretfile="$1"
+odir="ci-results"
+


The following allows you to get the directory of the script so that you don't need to assume the callers directory:

BASE_DIR=$(dirname $(readlink -f "$BASH_SOURCE"))

hickeng · 2018-01-27T06:24:31Z

infra/scripts/local-ci.sh

+test_password="$(grep 'TEST_PASSWORD' ${secretfile} | awk '{ print $2 }')"
+suite="$(grep -E '\s+TESTS' ${secretfile} | awk '{ print $2 }')"
+target_vch=$TARGET_VCH
+vic_dir=$(pwd)


vic_dir=$(BASE_DIR)/../../

hickeng · 2018-01-27T06:41:57Z

infra/scripts/local-ci.sh

+
+echo "Running tests"
+github_api_key="$(grep 'GITHUB_AUTOMATION_API_KEY' ${secretfile} | awk '{ print $2 }')"
+test_url="$(grep 'TEST_URL_ARRAY' ${secretfile} | awk '{ print $2 }')"


For all bar the github token you may well be able to use the output from govc env - we heavily leverage those environment variables already so likely their set.
They are being used in #7164 and the current bash-helpers.sh script sets them as part of the profiles.

hickeng · 2018-01-27T17:49:20Z

Need to add support for TLS configuration:

check the VCH config for TLS config: no-tls, no-tlsverify, or tlsverify
if no-tlsverify ensure unset DOCKER_TLS_VERIFY
if tlsverify ensure that export DOCKER_TLS_VERIFY=set_by_local_ci and that DOCKER_CERT_PATH is set

We may be able to add some better logic to vic-machine CLI to determine DOCKER_CERT_PATH during inspect and should update infra/scripts/bash-helpers.sh to allow configuring environment for a specific deployed VCH; we have vic-inspect but need to add vic-select.

Noting this because of various issues trying to use with tlsverify particularly around existing image lists:

<msg timestamp="20180127 17:41:21.350" level="INFO">Running command 'docker -H 192.168.78.127:2376 --tls images -q 2&gt;&amp;1'.</msg>
<msg timestamp="20180127 17:41:21.388" level="INFO">${rc} = 1</msg>
<msg timestamp="20180127 17:41:21.388" level="INFO">${output} = could not read CA certificate "/root/.docker/ca.pem": open /root/.docker/ca.pem: no such file or directory</msg>

mdubya66 · 2018-02-01T04:02:41Z

tests/README.md


+## Run Docker command tests via makefile target
+
+There exist a makefile target for developers to run the docker command tests locally (not in CI environment) against a pre-deployed VCH.  This is a fast way for contributors to test their potential PR submissions against the CI tests locally before pushing the commit.  There is also another benefit gained from using the makefile target.  With the way it is setup, logs from the run are written out to vic/ci-results, even if the tests fails.  The method described above to run the tests locally with drone has the weakness that a failure in the test can sometimes result in no written logs to help debug the failure.


There exists a makefile target for developers to run the docker command tests locally (not in CI environment) against a pre-deployed VCH. This is a fast way for contributors to test their potential code chages, against the CI tests locally, before pushing the commit. There is another benefit gained from using the makefile target, the way it is setup, logs from the run are written out to vic/ci-results. Even if the tests fail. The method described above, to run the tests locally with drone, has the weakness that a failure in the test can sometimes result in no written logs to help debug the failure.

mdubya66 · 2018-02-01T04:03:11Z

tests/README.md

+There are a few requirements before using this makefile target.
+
+1. A VCH must be pre-deployed before calling this makefile target
+2. The makefile target relies on a script that looks for a few more secrets variable, TEST_URL_ARRAY, TEST_USERNAME, TEST_PASSWORD, and TEST_DATASTORE


The makefile target relies on a script that looks for a few more secrets variables

mdubya66 · 2018-02-01T04:05:47Z

tests/README.md

+make local-ci-test secrets=test.secrets.nested dockertest=Group1-Docker-Commands vch=my_vch
+```
+
+Currently, only the Group1 tests are setup to use an existing VCH so this makefile target only works on the group 1 tests.  In the above, the first line runs a specific docker command test.  The second line runs the entire group 1 tests.  Notice, we define extra params to the *local-ci-test* target by specifying the param's name.  There are three parameters, *secrets*, *dockertest*, and *vch*.  The *vch* parameter must target an existing pre-deployed VCH.


Currently only the Group1 tests are setup to use an existing VCH, so this makefile target only works on the Group1 tests. In the above, the first line runs a specific docker command test. The second line runs the entire Group1 tests. Notice, we define extra params to the local-ci-test target by specifying the param's name. There are three parameters, secrets, dockertest, and vch. The vch parameter must target an existing pre-deployed VCH.

mdubya66 · 2018-02-01T04:06:16Z

tests/README.md

+
+Currently, only the Group1 tests are setup to use an existing VCH so this makefile target only works on the group 1 tests.  In the above, the first line runs a specific docker command test.  The second line runs the entire group 1 tests.  Notice, we define extra params to the *local-ci-test* target by specifying the param's name.  There are three parameters, *secrets*, *dockertest*, and *vch*.  The *vch* parameter must target an existing pre-deployed VCH.
+
+A helpful tip is to create different secrets file for different environments.  For instance, test.secrets.esx and test.secrets.vc for an ESX host and VC cluster, respectively.


A helpful tip is to create different secrets files for different environments. For instance, test.secrets.esx and test.secrets.vc for an ESX host and VC cluster, respectively.

vburenin

Can't wait to give it a try during daily routine!

hickeng · 2018-02-02T00:31:39Z

infra/scripts/local-ci.sh

+docker_test="Group1-Docker-Commands"
+target_vch=""
+odir="ci-results"
+ci_container="gcr.io/eminent-nation-87317/vic-integration-test:1.39"


@mhagen-vmware Are we tagging this with latest as well as versions? For the build/test portions of the system I'm all for explicit versioning, but for the dev scripts my preference is to use latest available versions without requiring update.

hickeng · 2018-02-02T00:40:02Z

infra/scripts/local-ci.sh

+if [[ -z "${TARGET_VCH}" ]] && [[ -z $target_vch ]] && [[ -z $secrets_target_vch ]]; then 
+  usage
+elif [[ -z $target_vch ]] ; then
+  target_vch=${TARGET_VCH}


I think this needs to be:

target_vch=${TARGET_VCH:-$secrets_target_vch}

and the whole may be easier to write as:

target_vch=${TARGET_VCH:-$secrets_target_vch} if [[ -z "${target_vch}" ]]; then usage fi

same conciseness comment applies to the rest of the tests.

hickeng · 2018-02-02T00:44:13Z

infra/scripts/local-ci.sh

+test_datastore=""
+test_username=""
+test_password=""
+vic_dir=$(pwd)


This is used to locate test sources so be explicit by basing of path of the script in the repo:

BASE_DIR=$(dirname $(readlink -f "$BASH_SOURCE")) vic_dir=${BASE_DIR}/../../

hickeng · 2018-02-02T00:46:08Z

infra/scripts/local-ci.sh

+                        -e TARGET_VCH=${target_vch}\
+                        -e DEBUG_VCH=1\
+                        ${ci_container}\
+                        bash -c "pybot -d /vic/${odir} /vic/tests/test-cases/"$docker_test";\


Looks like odir is bound to /tmp and not to /vic/odir.

hickeng · 2018-02-02T00:49:05Z

infra/scripts/local-ci.sh

+                        -e TARGET_VCH=${target_vch}\
+                        -e DEBUG_VCH=1\
+                        ${ci_container}\
+                        bash -c "pybot -d /vic/${odir} /vic/tests/test-cases/"$docker_test";\


Using double quoted string there's no need to unquote to get shell variable replacement for $docker_test

hickeng · 2018-02-02T00:49:46Z

infra/scripts/local-ci.sh

+                 mv VCH-*-vmware.log /tmp/ 2>/dev/null; \
+                 mv vic-machine.log /tmp/ 2>/dev/null; \
+                 mv index.html* /tmp/ 2>/dev/null; \
+                 mv VCH-* /tmp/ 2>/dev/null")


surprised to see these here - suggests to me that either I don't understand what the pybot -d option does, or it's not taken proper effect.

hickeng · 2018-02-02T00:51:14Z

infra/scripts/local-ci.sh

+    s)
+      secretsfile=$OPTARG
+      ;;
+    t)


I think you can provide pybot -t to run only a specific test in a suite. We may want to use a different option here so adding pass-through for that later doesn't result in option name collision.

hickeng · 2018-02-13T22:35:41Z

infra/scripts/local-ci.sh

+
+# There is a priority in the preconditions.  First, environment variable.  Second, secrets file.  Third, command line argument.
+
+if [[ -z "${SECRETS_FILE}" ]] && [[ -z $secretsfile ]] ; then


if [[ -z "${secretsfile:-$SECRETS_FILE}" ]]; then GetGovcParamsFromEnv else GetParamsFromSecrets fi

mdharamadas1

lgtm

Make testing locally as friction-free as possible by 1. Adding a makefile target 'local-ci-test' 2. Using TARGET_VCH added in VIC 1.3 to use an existing VCH 3. Using a custom script that doesn't utilize drone so that if the test fails and craters, we can still access the logs 4. Parameters can come from env vars, arguments, or secrets file Resolves vmware#7162

* Dump dmesg if container rootfs blocks or fails mount (#7260) This is to enable bridging of the guest side state with the virtual hardware if we see issues such as disks not presenting on a pvscsi controller or a mount operation hanging. * updating priority definitions to include features (#7292) * Change default fellows for gandalf (#7310) * Avoid exposing test credentials in 12-01-Delete (#7306) To avoid exposing test credentials, use the established `Run VIC Machine Delete Command` keyword, which in turn calls a secret keyword. This changes the behavior of the test slightly: - It no longer checks for the absence of "delete failed" in output. - It will wait up to 30 seconds for the deletion to succeed. - It will clean up cert files at the end of the deletion. * Bug fix in API delete test: disable volume store cleanup (#7303) * Remove volume store cleanup before re-installing VIC appliance using existing volume stores * Cleanup dangling volume stores on test teardown * Add logging for image upload (#7296) * Reduce datastore searches during non-vSAN delete operations (#6951) * Optimize portlayer volume cache rebuild on startup (#7267) This commit modifies the portlayer volume cache's rebuildCache func to only process the volumes from the volume store that is currently being added to the cache. rebuildCache is invoked for every volume store during portlayer startup. Before this change, rebuildCache would process volumes from all volume stores in the cache every time a volume store was added. This led to unneeded extra computation which could slow down portlayer startup and overwhelm NFS endpoints if NFS volume stores are being used. Fixes #6991 * Added local ci testing target to makefile (#7170) Make testing locally as friction-free as possible by 1. Adding a makefile target 'local-ci-test' 2. Using TARGET_VCH added in VIC 1.3 to use an existing VCH 3. Using a custom script that doesn't utilize drone so that if the test fails and craters, we can still access the logs 4. Parameters can come from env vars, arguments, or secrets file Resolves #7162 * Added upload progress bar tracker for ISO images. (#7320) * Added upload progress bar tracker for ISO images. Removed concurrent upload since it doesn't make any significant performance imapact. When I tried to measure performance differene with and without concurrent uppload, the results were fluctuating in a wide range so no good measurement was possible. * Document the design for the vic-machine API (#6702) This document proposes a design for a comprehensive vic-machine API, the implementation of which will be tracked by #6116. Subsets of this API (tracked by #5721, #6123, and eventually others) will be implemented incrementally, and the design will be revised as those efforts progress to reflect changes to the long-term vision. * Remove superfluous calls to Set Test VCH Name (#7304) Several tests explicitly call the `Set Test VCH Name` keyword shortly after calling `Set Test Environment Variables`. This can lead to test failures when a VCH name collision occurs; subsequent tests which re-use the VCH name fail because there may be leftover certificates from the first VCH with that name. `Set Test Environment Variables` itself calls `Set Test VCH Name` and then cleans up old certificate directories. Therefore, the explicit calls to `Set Test VCH Name` are both superfluous and problematic. * Ensure that static ip worker is on the same nimbus pod as VC otherwise network connectivity not guaranteed (#7307) * [skip ci] Add ROBO test plans (#7297) This commit adds test plans for the ROBO support features in a new directory (Group19-ROBO) under manual test cases. The existing ROBO-SKU test has been moved into this directory. The test plans include tests for the container limit feature, placement without DRS, the license/ feature checks and WAN connectivity. Fixes #7294 * Add hosts to DVS within the test bed as well (#7326) * Setup updated for Longevity Tests (#7298) * Setup updated for Longevity Tests to run on 6.5U1 * [skip ci] Terminate gracefully to gather logs (#7331) * Terminate gracefully to gather logs * Remove extra whitespace * Increase timeout to 70 minutes * Increase ELM timeout to 70 minutes * Add repo to slack message since we have multiple repos reporting now (#7334) * Not sending user credentials with every request (#6382) * Add concurrent testing tool to tests folder (#6534) Adds a minimized test case for testing our core vSphere interactions at varying degrees of concurrency. This is intended to simplify debugging issues that are suspected to be platform problems, or API usage issues that are conceptually divorced from the VIC engine product code. * Refactor Install Harbor To Test Server keyword (#7335) The secret tag on the `Install Harbor To Test Server` makes it difficult to investigate failures when they occur. Only one out of 30+ lines actually uses secret information. Refactor the keyword to extract the secret information into its own keyword, allowing the tag to be applied in a more focused way. This is similar to the pattern used by keywords in `VCH-Util`. * Add ability to cache generated dependency. (#7340) * Add ability to cache generated dependency, so not much time wasted during the build process. * Added documentation to reflect necessary steps to leverage such improvements. * Ignore not-supported result from RetrieveUserGroups in VC 6.0 (#7328) * Move build time directives from title to body of PR (#7060) * Retry the harbor setup as well (#7336) * Skip non vSphere managed datastores when granting perms (#7346) * Fix volume leak on group 23 test (#7358) * Fix github status automation filtering (#7344) Adds filtering for the event source and consolidates remote API calls. Details the specific builds and their status for quick reference. * Drone 0.8 and HaaS updates (#7364) * Add tether.debug in integration test log bundle (#7422) * Update the gcs plugin (#7421) * [skip ci] Suggest subnet/gateway to static ip worker * Ensure that static ip worker is on the same nimbus pod as VC otherwise network connectivity not guaranteed (#7307) * Refactored some proxy code to reuse with wolfpack Refactored the system, volume, container, and stream swagger code into proxy code. 1) Moved the errors.go from backends to a new folder to be accessed by all folders outside of the backends folder. 2) Refactored Container proxy and moved from engine/backends to engine/proxy 3) Refactored Volume proxy and moved from engine/backends to engine/proxy 4) Refactored System proxy and moved from engine/backends to engine/proxy 5) Refactored Stream proxy and moved from engine/backends to engine/proxy 6) Adopted some common patterns in all the proxies 7) Moved common networking util calls to engine/networking 8) Fix up unit tests 9) Changed all "not yet implemented messages" 10) Updated robot scripts More refactoring will be needed to make these proxy less dependent on docker types and portlayer swagger types. Helps resolves #7210 and #7232 * Add virtual-kubelet binary to VIC ISO (#7315) * Start virtual-kubelet inside the VCH (#7369) * Fix value of the PORTLAYER_ADDR environment variable (#7400) * Use vic kubelet provider * Made modifications for virtual kubelet * Added admin log collection and fix env variable content (#7404) * Added most-vkubelet target (#7418)

vmwclabot added the cla-not-required label Jan 25, 2018

sflxn force-pushed the make-local-ci branch 3 times, most recently from 362442d to a290d0a Compare January 25, 2018 22:22

hickeng reviewed Jan 27, 2018

View reviewed changes

sflxn force-pushed the make-local-ci branch 2 times, most recently from 97f228e to 02fc526 Compare January 31, 2018 22:34

sflxn changed the title ~~[WIP] Added local ci testing target to makefile [skip-ci]~~ Added local ci testing target to makefile Jan 31, 2018

sflxn requested review from gigawhitlocks, lcastellano, sgairo and vburenin January 31, 2018 22:39

sflxn force-pushed the make-local-ci branch 2 times, most recently from 159d654 to f62b3e7 Compare January 31, 2018 23:13

mdubya66 reviewed Feb 1, 2018

View reviewed changes

sflxn force-pushed the make-local-ci branch from f62b3e7 to d512717 Compare February 1, 2018 22:34

vburenin approved these changes Feb 2, 2018

View reviewed changes

sflxn requested a review from hickeng February 2, 2018 00:10

hickeng mentioned this pull request Feb 2, 2018

[skip-ci] Allow make push to replace and relaunch Appliance components #7233

Merged

hickeng reviewed Feb 2, 2018

View reviewed changes

sflxn force-pushed the make-local-ci branch 3 times, most recently from 695c902 to c6fdb9c Compare February 13, 2018 22:24

hickeng approved these changes Feb 13, 2018

View reviewed changes

sflxn force-pushed the make-local-ci branch from c6fdb9c to 19972b4 Compare February 13, 2018 22:49

sflxn requested a review from mhagen-vmware February 13, 2018 22:50

sflxn requested a review from mdharamadas1 February 13, 2018 22:50

mdharamadas1 approved these changes Feb 13, 2018

View reviewed changes

sflxn force-pushed the make-local-ci branch from 19972b4 to 258de1a Compare February 13, 2018 23:27

sflxn merged commit a6a4ff0 into vmware:master Feb 13, 2018


		## Run Docker command tests via makefile target

		There exist a makefile target for developers to run the docker command tests locally (not in CI environment) against a pre-deployed VCH. This is a fast way for contributors to test their potential PR submissions against the CI tests locally before pushing the commit. There is also another benefit gained from using the makefile target. With the way it is setup, logs from the run are written out to vic/ci-results, even if the tests fails. The method described above to run the tests locally with drone has the weakness that a failure in the test can sometimes result in no written logs to help debug the failure.


		Currently, only the Group1 tests are setup to use an existing VCH so this makefile target only works on the group 1 tests. In the above, the first line runs a specific docker command test. The second line runs the entire group 1 tests. Notice, we define extra params to the local-ci-test target by specifying the param's name. There are three parameters, secrets, dockertest, and vch. The vch parameter must target an existing pre-deployed VCH.

		A helpful tip is to create different secrets file for different environments. For instance, test.secrets.esx and test.secrets.vc for an ESX host and VC cluster, respectively.


		# There is a priority in the preconditions. First, environment variable. Second, secrets file. Third, command line argument.

		if [[ -z "${SECRETS_FILE}" ]] && [[ -z $secretsfile ]] ; then

Conversation

sflxn commented Jan 25, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hickeng commented Jan 27, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vburenin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mdharamadas1 left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

sflxn commented Jan 25, 2018 •

edited

Loading

hickeng commented Jan 27, 2018 •

edited

Loading