Parallelize integration tests by mhagen-vmware · Pull Request #5601 · vmware/vic

mhagen-vmware · 2017-06-27T19:39:12Z

[skip unit]
[full ci]

hickeng

Looks good in general - I think you'll be racing on the Clean Up Containers and Images however.

I'm also not sure we need to clean up existing containers for most tests, and images for even fewer.

The ps tests definitely assert number of lines in output, but many others don't care. If we support labels we may be able to label the containers with the build number/test name so that filtering can be used when needed.

hickeng · 2017-06-27T21:23:50Z

tests/resources/VCH-Util.robot

    [Return]  ${output}

+Cleanup Containers And Images
+    ${out}=  Run  docker %{VCH-PARAMS} ps -aq | xargs -n1 docker %{VCH-PARAMS} rm -f


you shouldn't need the -n1 but I don't think it'll hurt

hickeng · 2017-06-27T22:25:23Z

Can we have each test suite make a "Get VCH" call or similar. Flow would be:

is there an existing VCH?
- does the existing VCH pass basic functional check (e.g. docker info)?
  - if yes, use this VCH
deploy VCH
- is there a VCH deploy already in progress?
  - if so, wait for deploy to complete and retry from top
- start new VCH deploy
  - wait for deploy to complete and retry from top

rogeliosanchez

lgtm

caglar10ur

YES, PLEASE! 👍 💯 🥇

mdubya66 · 2017-06-29T03:00:31Z

LGTM!

mhagen-vmware · 2017-08-17T18:12:32Z

Still seeing TLS handshake timeouts with @caglar10ur latest changes.

caglar10ur · 2017-08-17T20:14:42Z

I think we are getting those cause the ESXi is getting slower and eventually timing out. Can we collect the host statistics during the run to see whether 1) This speculation is correct 2) The slowness is CPU or IO bound 3) we can do something about it?

mdharamadas1 · 2017-10-25T19:31:09Z

@mhagen-vmware This seems to have issue with cleaning up container VMs as observed in this build running on builds-16.vmware.run. There were about 87 VMs left over.

caglar10ur · 2017-10-31T17:49:53Z

Several failures occurred:

1) 'govc: 503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x31eb2938] _serverNamespace = /sdk _isRedirect = false _port = 8307)' does not contain 'OK'

2) 'govc: 503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x31eb2938] _serverNamespace = /sdk _isRedirect = false _port = 8307)' does not contain 'OK'

looks like someone restarted the host ESXi server while the tests are running

mhagen-vmware · 2017-10-31T17:56:18Z

I restarted it afterwards, because the system was in a state that we continue to see. In which everything appears up, but the system reports wrong password when trying to login or do anything via govc. A system reboot through OVH is needed and fixes it.

hickeng · 2018-02-05T22:11:07Z

.drone.local.yml


  vic-integration-test-on-pr:
-    image: ${TEST_BUILD_IMAGE=gcr.io/eminent-nation-87317/vic-integration-test:1.39}
+    image: ${TEST_BUILD_IMAGE=gcr.io/eminent-nation-87317/vic-integration-test:1.37}


should be we reverting to 1.37 from 1.39?

hickeng · 2018-02-05T22:12:08Z

tests/integration-test.sh

 if [[ $DRONE_BRANCH == "master" || $DRONE_BRANCH == "releases/"* ]] && [[ $DRONE_REPO == "vmware/vic" ]] && [[ $DRONE_BUILD_EVENT == "push" ]]; then
    echo "Running full CI for $DRONE_BUILD_EVENT on $DRONE_BRANCH"
-    pybot --removekeywords TAG:secret --exclude skip tests/test-cases
+    pabot --processes 2 --removekeywords TAG:secret --exclude skip tests/test-cases


parameterize the number of processes or do we want to be able to chose per-scenario values?

codecov-io · 2018-05-22T15:35:38Z

Codecov Report

Merging #5601 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #5601   +/-   ##
=======================================
  Coverage   26.07%   26.07%           
=======================================
  Files          37       37           
  Lines        5243     5243           
=======================================
  Hits         1367     1367           
  Misses       3769     3769           
  Partials      107      107

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 138a803...9b8365a. Read the comment docs.

(cherry picked from commit 02ba3ad)

mhagen-vmware requested review from anchal-agrawal, hickeng, rajanashok, rogeliosanchez and singhshwetaQE June 27, 2017 19:39

vmwclabot added the cla-not-required label Jun 27, 2017

hickeng reviewed Jun 27, 2017

View reviewed changes

mhagen-vmware changed the title ~~[skip ci] WIP Parallelize robot and reuse VCHs~~ WIP Parallelize robot and reuse VCHs Jun 28, 2017

mhagen-vmware changed the title ~~WIP Parallelize robot and reuse VCHs~~ [full ci]WIP Parallelize robot and reuse VCHs Jun 28, 2017

sgairo approved these changes Jun 28, 2017

View reviewed changes

rogeliosanchez approved these changes Jun 28, 2017

View reviewed changes

mhagen-vmware force-pushed the feature/parallel-robot branch 2 times, most recently from 80ae2e7 to e79962f Compare June 28, 2017 23:43

caglar10ur approved these changes Jun 29, 2017

View reviewed changes

dougm mentioned this pull request Jul 20, 2017

vSphere SDK API Endpoint becomes unresponsive - govc: Post https://builds-14.vmware.run/sdk: net/http: TLS handshake timeout #5621

Closed

mhagen-vmware force-pushed the feature/parallel-robot branch 2 times, most recently from e67f606 to 41a8ae2 Compare July 26, 2017 15:00

mhagen-vmware changed the title ~~[full ci]WIP Parallelize robot and reuse VCHs~~ [full ci] Parallelize robot and reuse VCHs Aug 17, 2017

mhagen-vmware force-pushed the feature/parallel-robot branch from 41a8ae2 to 406e6c1 Compare August 17, 2017 14:45

mdharamadas1 mentioned this pull request Sep 8, 2017

Continuous Integration v2 #6164

Closed

7 tasks

mhagen-vmware force-pushed the feature/parallel-robot branch 3 times, most recently from ad376d4 to 6250499 Compare October 24, 2017 09:17

mhagen-vmware closed this Feb 1, 2018

mhagen-vmware force-pushed the feature/parallel-robot branch from 2b1c481 to 0b314d5 Compare February 1, 2018 20:28

mhagen-vmware reopened this Feb 1, 2018

hickeng reviewed Feb 5, 2018

View reviewed changes

mhagen-vmware changed the title ~~[full ci] Parallelize robot and reuse VCHs~~ [full ci] Parallelize integration tests Feb 6, 2018

mhagen-vmware force-pushed the feature/parallel-robot branch from d23af43 to ccb1778 Compare February 6, 2018 21:10

mhagen-vmware changed the title ~~[full ci] Parallelize integration tests~~ Parallelize integration tests Apr 30, 2018

[skip ci].

5f4e2ab

mhagen-vmware force-pushed the feature/parallel-robot branch from ccb1778 to 5f4e2ab Compare April 30, 2018 15:44

mhagen-vmware added 2 commits May 2, 2018 14:41

Merge branch 'master' into feature/parallel-robot

ef79702

Merge branch 'master' into feature/parallel-robot

cc4cacb

mhagen-vmware added 6 commits May 29, 2018 11:15

Merge branch 'master' into feature/parallel-robot

2771bd3

Merge branch 'master' into feature/parallel-robot

ef123e7

Merge branch 'master' into feature/parallel-robot

b1e4e58

Merge branch 'master' into feature/parallel-robot

9b8365a

Merge branch 'master' into feature/parallel-robot

22a38b3

Merge branch 'master' into feature/parallel-robot

9c8ed20

mhagen-vmware merged commit 02ba3ad into master Jun 4, 2018

mhagen-vmware deleted the feature/parallel-robot branch June 4, 2018 14:08

zjs pushed a commit to zjs/vic that referenced this pull request Jul 27, 2018

Parallelize integration tests (vmware#5601)

da545ba

(cherry picked from commit 02ba3ad)

zjs mentioned this pull request Jul 27, 2018

Cherry-picks: Run integration tests in parallel #8162

Merged

zjs pushed a commit to zjs/vic that referenced this pull request Aug 2, 2018

Parallelize integration tests (vmware#5601) (vmware#8162)

bd40925

(cherry picked from commit 02ba3ad)

zjs pushed a commit that referenced this pull request Aug 7, 2018

Parallelize integration tests (#5601) (#8162)

89a1ca8

(cherry picked from commit 02ba3ad)

Conversation

mhagen-vmware commented Jun 27, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hickeng left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hickeng Jun 27, 2017

Choose a reason for hiding this comment

Uh oh!

hickeng commented Jun 27, 2017

Uh oh!

rogeliosanchez left a comment

Choose a reason for hiding this comment

Uh oh!

caglar10ur left a comment

Choose a reason for hiding this comment

Uh oh!

mdubya66 commented Jun 29, 2017 • edited by caglar10ur Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mhagen-vmware commented Aug 17, 2017

Uh oh!

caglar10ur commented Aug 17, 2017

Uh oh!

mdharamadas1 commented Oct 25, 2017

Uh oh!

caglar10ur commented Oct 31, 2017

Uh oh!

mhagen-vmware commented Oct 31, 2017

Uh oh!

hickeng Feb 5, 2018

Choose a reason for hiding this comment

Uh oh!

hickeng Feb 5, 2018

Choose a reason for hiding this comment

Uh oh!

codecov-io commented May 22, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

mhagen-vmware commented Jun 27, 2017 •

edited

Loading

hickeng left a comment •

edited

Loading

mdubya66 commented Jun 29, 2017 •

edited by caglar10ur

Loading

codecov-io commented May 22, 2018 •

edited

Loading