Add basic integration tests for VCH inspect API by AngieCris · Pull Request #7290 · vmware/vic

AngieCris · 2018-02-08T19:01:08Z

[specific ci=23-04-VCH-Inspect --suite 23-02-VCH-List]

some basic robot tests added for API inspect.

zjs

Also some bug fixes in API delete:

As the delete changes are logically separate from the inspect test you've written, my personal preference would be to push the delete fixes as a separate commit, which might require manually squashing the other commits and fixing up the commit messages before doing a "Rebase and merge" of the PR. Not sure if that's actually possible with our repository configuration though. ¯\_(ツ)_/¯

zjs · 2018-02-08T21:02:06Z

tests/test-cases/Group23-VIC-Machine-Service/23-04-VCH-Inspect.robot

+
+Verify VCH Inspect Output
+    # debug
+    Property Should Be Equal        .debug  1


nit: this might be slightly more readable if you insert extra spaces between .debug and 1 so that 1 is vertically aligned with the other values, below

ah, missed this one. thanks

zjs · 2018-02-08T21:03:49Z

tests/test-cases/Group23-VIC-Machine-Service/23-04-VCH-Inspect.robot

+
+    # version
+    ${version}=  Get Version String
+    Property Should Be Equal  .version  ${version}


nit: as above, I think this would be more readable if .version were vertically aligned with the other properties and ${version} were vertically aligned with the other values

zjs · 2018-02-08T21:05:28Z

tests/test-cases/Group23-VIC-Machine-Service/23-04-VCH-Inspect.robot

+
+    Property Should Contain         .storage.image_stores[0]                    %{TEST_DATASTORE}
+    Property Should Contain         .storage.volume_stores[0].datastore         %{TEST_DATASTORE}/%{VCH-NAME}-VOL
+    Property Should Be Equal        .storage.volume_stores[0].label             default


Should we also be verifying that .image_stores and .volume_stores contain only a single element? (We verify that the first element of each is what we're expecting, but the test would still pass if there were other bogus values showing up in the response.)

Solving this elegantly may require adding a new Property Should ... keyword (perhaps along the lines of Property Should Be Unary, which we could use for each array we expect to have a single element, or perhaps along the lines of Property Length Should Be, which we could use for each array we expect to have a specific length).

zjs · 2018-02-08T21:09:39Z

tests/test-cases/Group23-VIC-Machine-Service/23-04-VCH-Inspect.robot

+    Get Path Under Target Using Session    datacenter/${DC-ID}/vch/${VCH-ID}
+
+Verify VCH Inspect Output
+    # debug


Could we also verify the other top-level properties, like .name?

AngieCris · 2018-02-08T22:24:05Z

@zjs Yes you're right, it's better to put the delete bug fixes in a separate pr. Idk what's the best way to do it but I could always cherry-pick commits to a new branch, and manually revert back this branch...not sure if it will work...let me try and see

update: did git rebase -i <commit>~1 and messed up. It rolled in a bunch of other people's commits...just manually reverted the code...

hickeng · 2018-02-12T23:09:53Z

@AngieCris

Re-installing VCH with existing volume stores accidentally does a volume store clean-up step. Disabled that by passing a cleanup=${false} to VIC appliance install

I recall us testing that volumes in existing volume stores still exist after a reinstall as part of the delete tests. This behaviour should have been caught there if it's a product code issues, or we need a keyword that expects existing volume stores during VCH install.

The flip side of that is that the regular VCH install keyword should confirm that volume stores don't exist and error if they do - that way we're being explicit about our state expectations for the test.

AngieCris · 2018-02-13T19:00:15Z

@hickeng

I recall us testing that volumes in existing volume stores still exist after a reinstall as part of the delete tests.

Yes, we have a test case that re-installs VCH using existing volume stores, and then checks after installation completes that the volume stores exist. But the re-install keyword does a cleanup on test server first, which could accidentally delete the existing volume stores.

This behaviour should have been caught there if it's a product code issues, or we need a keyword that expects existing volume stores during VCH install.

The reason why this behavior hasn't been caught on CI is that because CI checks drone status and skips cleanup if the test is running (but when I ran locally, it failed, and that's how I found out this issue).
But conceptually in the test code if we expect existing volume store, we should not do any cleanup step before re-installing.

But yes passing cleanup=${false} to the vic install keyword probably isn't the most optimal way to do it...it disables all the cleanup steps including networks, volume stores, etc.
We probably need another keyword in VCH-util that expects volume stores before installation, and have the old keyword errors out if there's existing volume store.
The cleanup=${false} kind of gives similar functionality, but it's not used anywhere in the code. I wonder what's our original purpose of having this argument.

AngieCris · 2018-02-13T20:38:25Z

Kind of blocked by this ticket: #7321
Found a bug on API inspect code when working on this PR - inspect API doesn't check all allowed IPs according to the configured certificate.
Need to figure out if we want to wait on the test after the bug is fixed and checked in...

zjs · 2018-02-13T20:55:58Z

Kind of blocked by this ticket: #7321

If that issue blocks this PR, we should go ahead and fix it. (It may only be P2 on its own merits, but if it's blocking a P1 we should treat it as a P1.)

mhagen-vmware · 2018-02-15T19:26:59Z

cleanup=${false} was added for nightly tests, because some tests don't have VCH installed anymore at the end and resulted in a 5 minute hang trying to gather logs from a non-existent VCH.

hickeng

Just discovered this comment in pending from a long time back. Still seems relevent.

hickeng · 2018-02-12T23:12:02Z

tests/resources/Group23-VIC-Machine-Service-Util.robot

    Set Test Variable    ${OUTPUT}
    Set Test Variable    ${STATUS}

+Get Version String


These util methods may need to have Service in the name or similar unless robot supports some namespacing I'm unaware of @mhagen-vmware? There's also the potential for a name collision with other components in addition to the simple confusion when reading their use inline.

The original intent was to put this resource file in the group directory, as that was the closest pattern I could find to namespacing.

I was asked to move the file to the resources directory, so I put the group name in the file name instead.

Generally, I'd strongly prefer to find a pattern to allow scoping of keywords over attempting to name each keyword verbosely, for all of the reasons we would avoid defining everything at the global scope in the product code.

codecov-io · 2018-04-17T20:04:32Z

Codecov Report

Merging #7290 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #7290   +/-   ##
=======================================
  Coverage   26.34%   26.34%           
=======================================
  Files          37       37           
  Lines        5189     5189           
=======================================
  Hits         1367     1367           
  Misses       3715     3715           
  Partials      107      107

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0e43ea6...ba747ae. Read the comment docs.

zjs

Looks good!

zjs · 2018-04-19T18:12:37Z

tests/test-cases/Group23-VIC-Machine-Service/23-04-VCH-Inspect.robot

+*** Keywords ***
+Setup
+    Start VIC Machine Server
+    Install VIC Appliance To Test Server  certs=${true}  additional-args=--ops-user=%{TEST_USERNAME} --ops-password=%{TEST_PASSWORD} --debug 1


We may need to tag this keyword as secret (or move this one call into a separate "Secret Setup" keyword tagged secret) because we're using %{TEST_PASSWORD} (and don't want that logged).

mhagen-vmware · 2018-05-01T21:32:25Z

tests/test-cases/Group23-VIC-Machine-Service/23-04-VCH-Inspect.robot

+*** Keywords ***
+Secret Install VIC Appliance With Ops Credentials
+    [Tags]  secret
+    Install VIC Appliance To Test Server  certs=${true}  additional-args=--ops-user=%{TEST_USERNAME} --ops-password=%{TEST_PASSWORD} --debug 1


this is just setting the ops user to administrator@vsphere.local... not sure that really is what we want to do, also wrapping this entire thing in [tags] secret is going to be a real problem when it comes to debugging.

this is just setting the ops user to administrator@vsphere.local... not sure that really is what we want to do

in order to not duplicate with CLI's tests, here it only checks that the ops user configuration goes into VCH config (inspect output matches up with what we put in install parameter). So it doesn't really matter what ops user we set it to.

wrapping this entire thing in [tags] secret is going to be a real problem when it comes to debugging

Since we're setting ops user, I guess we don't want to expose %{TEST_USERNAME} and %{TEST_PASSWORD} in the log?

@jzt is adding an ops user to our CI already, just use that and remove the tags secret from here.

mhagen-vmware · 2018-05-04T19:24:26Z

tests/test-cases/Group23-VIC-Machine-Service/23-02-VCH-List.robot

+
+    Verify VCH List Empty
+
+    [Teardown]  NONE


I just removed this test here:
https://github.com/vmware/vic/pull/7879/files

Please do not add it back in. It inherently cannot be run in our CI, it needs to be moved to manual tests

oh sorry I didn't see it. It got automatically added by git merge. will remove it now.

just removed

mhagen-vmware

Lgtm

rogeliosanchez

lgtm

(cherry picked from commit 21f713d)

…#8162) (cherry picked from commit 21f713d)

(cherry picked from commit 21f713d)

AngieCris requested review from hickeng and zjs February 8, 2018 19:01

AngieCris self-assigned this Feb 8, 2018

vmwclabot added the cla-not-required label Feb 8, 2018

zjs reviewed Feb 8, 2018

View reviewed changes

AngieCris changed the title ~~Add integration tests for VCH inspect API~~ [specific ci=Group23-VIC-Machine-Service] Add integration tests for VCH inspect API Feb 9, 2018

AngieCris force-pushed the vic/6031 branch 2 times, most recently from 0860e88 to 24a04f8 Compare February 9, 2018 18:10

hickeng reviewed Mar 15, 2018

View reviewed changes

This was referenced Mar 16, 2018

Update VCH inspect API to output TLS friendly docker host and admin portal address #7533

Merged

Improve basic positive and negative integration testing for the VCH Inspect API #7517

Closed

AngieCris changed the title ~~[specific ci=Group23-VIC-Machine-Service] Add integration tests for VCH inspect API~~ Add integration tests for VCH inspect API Apr 17, 2018

AngieCris force-pushed the vic/6031 branch from a989ae7 to b4bb25f Compare April 17, 2018 16:13

AngieCris requested review from rogeliosanchez and singhshwetaQE April 17, 2018 16:16

AngieCris requested a review from mhagen-vmware April 19, 2018 18:11

zjs reviewed Apr 19, 2018

View reviewed changes

AngieCris changed the title ~~Add integration tests for VCH inspect API~~ Add basic integration tests for VCH inspect API Apr 19, 2018

zjs approved these changes Apr 25, 2018

View reviewed changes

mhagen-vmware reviewed May 1, 2018

View reviewed changes

AngieCris force-pushed the vic/6031 branch from 89fb3aa to fc10c0a Compare May 3, 2018 20:31

mhagen-vmware suggested changes May 4, 2018

View reviewed changes

add basic VCH inspect API tests

e0864c9

AngieCris force-pushed the vic/6031 branch from f8aab80 to e0864c9 Compare May 4, 2018 19:33

mhagen-vmware approved these changes May 4, 2018

View reviewed changes

rogeliosanchez approved these changes May 4, 2018

View reviewed changes

AngieCris added 3 commits May 4, 2018 15:05

fixup! add basic VCH inspect API tests

7cce2b1

fixup! fixup! add basic VCH inspect API tests

6daf59d

Merge branch 'master' of github.com:vmware/vic into vic/6031

ba747ae

AngieCris merged commit 21f713d into vmware:master May 4, 2018

AngieCris deleted the vic/6031 branch May 4, 2018 21:39

zjs pushed a commit to zjs/vic that referenced this pull request Jul 27, 2018

Add basic integration tests for VCH inspect API (vmware#7290)

2bd9e4f

(cherry picked from commit 21f713d)

zjs mentioned this pull request Jul 27, 2018

Cherry-picks: Run integration tests in parallel #8162

Merged

zjs pushed a commit to zjs/vic that referenced this pull request Aug 2, 2018

Add basic integration tests for VCH inspect API (vmware#7290) (vmware…

52e7661

…#8162) (cherry picked from commit 21f713d)

zjs pushed a commit that referenced this pull request Aug 7, 2018

Add basic integration tests for VCH inspect API (#7290) (#8162)

8967f3b

(cherry picked from commit 21f713d)

Conversation

AngieCris commented Feb 8, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zjs left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AngieCris Feb 8, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AngieCris commented Feb 8, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hickeng commented Feb 12, 2018

Uh oh!

AngieCris commented Feb 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AngieCris commented Feb 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

zjs commented Feb 13, 2018

Uh oh!

mhagen-vmware commented Feb 15, 2018

Uh oh!

hickeng left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov-io commented Apr 17, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

zjs left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AngieCris May 3, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mhagen-vmware left a comment

Choose a reason for hiding this comment

Uh oh!

rogeliosanchez left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

AngieCris commented Feb 8, 2018 •

edited

Loading

AngieCris Feb 8, 2018 •

edited

Loading

AngieCris commented Feb 8, 2018 •

edited

Loading

AngieCris commented Feb 13, 2018 •

edited

Loading

AngieCris commented Feb 13, 2018 •

edited

Loading

codecov-io commented Apr 17, 2018 •

edited

Loading

AngieCris May 3, 2018 •

edited

Loading