marvin: Bypass tests on Simulator

[Pearl1594]

Description

This PR attempts to fix travis failures observed post merge of #4503. Checks have been added to skip specific tests for Simulator based envs.

Exceptions noticed in travis:

2021-02-18 23:08:13,330 - CRITICAL - EXCEPTION: test_01_deployVMInSharedNetwork: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/unittest/case.py", line 329, in run\n    testMethod()\n', '  File "/home/travis/build/apache/cloudstack/test/integration/smoke/test_network.py", line 1988, in test_01_deployVMInSharedNetwork\n    host = self.get_router_host(router)\n', '  File "/home/travis/build/apache/cloudstack/test/integration/smoke/test_network.py", line 1944, in get_router_host\n    host.user, host.password = get_host_credentials(self.config, host.ipaddress)\n', '  File "/home/travis/.local/lib/python2.7/site-packages/marvin/lib/utils.py", line 231, in get_host_credentials\n    raise KeyError("Please provide the marvin configuration file with credentials to your hosts")\n', "KeyError: 'Please provide the marvin configuration file with credentials to your hosts'\n"]
2021-02-18 23:08:18,539 - CRITICAL - EXCEPTION: test_02_verifyRouterIpAfterNetworkRestart: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/unittest/case.py", line 329, in run\n    testMethod()\n', '  File "/home/travis/build/apache/cloudstack/test/integration/smoke/test_network.py", line 2016, in test_02_verifyRouterIpAfterNetworkRestart\n    host = self.get_router_host(router)\n', '  File "/home/travis/build/apache/cloudstack/test/integration/smoke/test_network.py", line 1944, in get_router_host\n    host.user, host.password = get_host_credentials(self.config, host.ipaddress)\n', '  File "/home/travis/.local/lib/python2.7/site-packages/marvin/lib/utils.py", line 231, in get_host_credentials\n    raise KeyError("Please provide the marvin configuration file with credentials to your hosts")\n', "KeyError: 'Please provide the marvin configuration file with credentials to your hosts'\n"]
2021-02-18 23:08:19,602 - CRITICAL - FAILED: test_03_destroySharedNetwork: ['Traceback (most recent call last):\n', '  File "/usr/lib/python2.7/unittest/case.py", line 329, in run\n    testMethod()\n', '  File "/home/travis/build/apache/cloudstack/test/integration/smoke/test_network.py", line 2025, in test_03_destroySharedNetwork\n    self.fail("Failed to destroy the shared network")\n', '  File "/usr/lib/python2.7/unittest/case.py", line 410, in fail\n    raise self.failureException(msg)\n', 'AssertionError: Failed to destroy the shared network\n']
2021-02-18 23:08:19,638 - CRITICAL - EXCEPTION: test_03_destroySharedNetwork: ['Traceback (most recent call last):\n', '  File "/home/travis/.local/lib/python2.7/site-packages/nose/suite.py", line 228, in run\n    self.tearDown()\n', '  File "/home/travis/.local/lib/python2.7/site-packages/nose/suite.py", line 351, in tearDown\n    self.teardownContext(ancestor)\n', '  File "/home/travis/.local/lib/python2.7/site-packages/nose/suite.py", line 367, in teardownContext\n    try_run(context, names)\n', '  File "/home/travis/.local/lib/python2.7/site-packages/nose/util.py", line 471, in try_run\n    return func()\n', '  File "/home/travis/build/apache/cloudstack/test/integration/smoke/test_network.py", line 1902, in tearDownClass\n    raise Exception("Warning: Exception during cleanup : %s" % e)\n', "Exception: Warning: Exception during cleanup : Execute cmd: deletenetworkoffering failed, due to: errorCode: 431, errorText:Can't delete network offering 22 as its used by 1 networks. To make the network offering unavaiable, disable it\n"]

Types of changes

• Breaking change (fix or feature that would cause existing functionality to change)
• New feature (non-breaking change which adds functionality)
• Bug fix (non-breaking change which fixes an issue)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

• Major
• Minor

Bug Severity

• BLOCKER
• Critical
• Major
• Minor
• Trivial

How Has This Been Tested?

Test on Simulator env:

nosetests --with-xunit --xunit-file=results.xml --with-marvin --marvin-config=setup/dev/advanced.cfg -s -a tags=xx --hypervisor=Simulator test/integration/smoke/test_network.py 

==== Marvin Init Started ====

=== Marvin Parse Config Successful ===

=== Marvin Setting TestData Successful===

==== Log Folder Path: /tmp/MarvinLogs/Feb_19_2021_16_14_35_UZ1YA0. All logs will be available here ====

=== Marvin Init Logging Successful===

==== Marvin Init Successful ====
===final results are now copied to: /tmp/MarvinLogs/test_suite_1PU3PO===

Tests on KVM:

 nosetests --with-xunit --xunit-file=results.xml --with-marvin --marvin-config=/marvin/ref-trl-2413-k-M7-david-jumani-advanced-cfg -s -a tags=xx --hypervisor=kvm test_net.py 

==== Marvin Init Started ====

=== Marvin Parse Config Successful ===

=== Marvin Setting TestData Successful===

==== Log Folder Path: /marvin/MarvinLogs/Feb_19_2021_10_03_02_TAWBVC. All logs will be available here ====

=== Marvin Init Logging Successful===

==== Marvin Init Successful ====
Executing command '{'endip': '172.16.134.21', 'acltype': 'domain', 'routerip': '172.16.134.15', 'name': 'MySharedNetwork - Test', 'networkofferingid': u'59739a6f-d682-410c-901c-aa3a62e4e551', 'physicalnetworkid': u'f8f4c4cd-ae62-4f19-b789-b4144b97aa60', 'startip': '172.16.134.2', 'vlan': 3133, 'netmask': '255.255.255.0', 'displaytext': 'MySharedNetwork', 'scope': 'all', 'gateway': '172.16.134.1'}'
Executing command 'ip addr show eth0 |grep "inet "|cut -d " " -f6 |cut -d "/" -f1 |grep -w 172.16.134.15'
====Trying SSH Connection: Host:10.10.4.136 User:root                                   Port:22 RetryCnt:60===
===SSH to Host 10.10.4.136 port : 22 SUCCESSFUL===
{Cmd: ssh -i ~/.ssh/id_rsa.cloud -ostricthostkeychecking=no -oUserKnownHostsFile=/dev/null -p 3922 169.254.133.68 ip addr show eth0 |grep "inet "|cut -d " " -f6 |cut -d "/" -f1 |grep -w 172.16.134.15 via Host: 10.10.4.136} {returns: [u'172.16.134.15']}
=== TestName: test_01_deployVMInSharedNetwork | Status : SUCCESS ===

Executing command 'ip addr show eth0 |grep "inet "|cut -d " " -f6 |cut -d "/" -f1 |grep -w 172.16.134.15'
====Trying SSH Connection: Host:10.10.3.147 User:root                                   Port:22 RetryCnt:60===
====Trying SSH Connection: Host:10.10.3.147 User:root                                   Port:22 RetryCnt:60===
===SSH to Host 10.10.3.147 port : 22 SUCCESSFUL===
===SSH to Host 10.10.3.147 port : 22 SUCCESSFUL===
{Cmd: ssh -i ~/.ssh/id_rsa.cloud -ostricthostkeychecking=no -oUserKnownHostsFile=/dev/null -p 3922 169.254.76.134 ip addr show eth0 |grep "inet "|cut -d " " -f6 |cut -d "/" -f1 |grep -w 172.16.134.15 via Host: 10.10.3.147} {returns: [u'172.16.134.15']}
{Cmd: ssh -i ~/.ssh/id_rsa.cloud -ostricthostkeychecking=no -oUserKnownHostsFile=/dev/null -p 3922 169.254.76.134 ip addr show eth0 |grep "inet "|cut -d " " -f6 |cut -d "/" -f1 |grep -w 172.16.134.15 via Host: 10.10.3.147} {returns: [u'172.16.134.15']}
=== TestName: test_02_verifyRouterIpAfterNetworkRestart | Status : SUCCESS ===

=== TestName: test_03_destroySharedNetwork | Status : SUCCESS ===

===final results are now copied to: /marvin//MarvinLogs//RNR1NY===

Tests run on VMware:

nosetests-2.7 --with-xunit --xunit-file=results.xml --with-marvin --marvin-config=/marvin/pr4577-t3581-vmware-67u3-advanced-cfg -s -a tags=xx --hypervisor=vmware test.py 

==== Marvin Init Started ====

=== Marvin Parse Config Successful ===

=== Marvin Setting TestData Successful===

==== Log Folder Path: /marvin/MarvinLogs/Feb_19_2021_10_36_43_SQ3YFY. All logs will be available here ====

=== Marvin Init Logging Successful===

==== Marvin Init Successful ====
Executing command '{'endip': '172.16.128.21', 'acltype': 'domain', 'routerip': '172.16.128.15', 'name': 'MySharedNetwork - Test', 'networkofferingid': u'a3f03129-3c17-41b0-a5a7-240c0ceb6a4c', 'physicalnetworkid': u'ee587cf1-dcce-41eb-be18-6f42e78645c3', 'startip': '172.16.128.2', 'vlan': 723, 'netmask': '255.255.255.0', 'displaytext': 'MySharedNetwork', 'scope': 'all', 'gateway': '172.16.128.1'}'
172.16.128.15
Executing command 'ip addr show eth0 |grep "inet "|cut -d " " -f6 |cut -d "/" -f1 |grep -w 172.16.128.15'
====Trying SSH Connection: Host:10.10.4.172 User:root                                   Port:22 RetryCnt:60===
===SSH to Host 10.10.4.172 port : 22 SUCCESSFUL===
{Cmd: ssh -i /var/cloudstack/management/.ssh/id_rsa -ostricthostkeychecking=no -oUserKnownHostsFile=/dev/null -p 3922 10.10.5.128 ip addr show eth0 |grep "inet "|cut -d " " -f6 |cut -d "/" -f1 |grep -w 172.16.128.15 via Host: 10.10.4.172} {returns: [u'172.16.128.15']}
=== TestName: test_01_deployVMInSharedNetwork | Status : SUCCESS ===

Executing command 'ip addr show eth0 |grep "inet "|cut -d " " -f6 |cut -d "/" -f1 |grep -w 172.16.128.15'
====Trying SSH Connection: Host:10.10.4.172 User:root                                   Port:22 RetryCnt:60===
====Trying SSH Connection: Host:10.10.4.172 User:root                                   Port:22 RetryCnt:60===
===SSH to Host 10.10.4.172 port : 22 SUCCESSFUL===
===SSH to Host 10.10.4.172 port : 22 SUCCESSFUL===
{Cmd: ssh -i /var/cloudstack/management/.ssh/id_rsa -ostricthostkeychecking=no -oUserKnownHostsFile=/dev/null -p 3922 10.10.5.126 ip addr show eth0 |grep "inet "|cut -d " " -f6 |cut -d "/" -f1 |grep -w 172.16.128.15 via Host: 10.10.4.172} {returns: [u'172.16.128.15']}
{Cmd: ssh -i /var/cloudstack/management/.ssh/id_rsa -ostricthostkeychecking=no -oUserKnownHostsFile=/dev/null -p 3922 10.10.5.126 ip addr show eth0 |grep "inet "|cut -d " " -f6 |cut -d "/" -f1 |grep -w 172.16.128.15 via Host: 10.10.4.172} {returns: [u'172.16.128.15']}
=== TestName: test_02_verifyRouterIpAfterNetworkRestart | Status : SUCCESS ===

=== TestName: test_03_destroySharedNetwork | Status : SUCCESS ===

===final results are now copied to: /marvin//MarvinLogs//O1X7IT===

[Pearl1594]

@blueorangutan package

[blueorangutan]

@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

[weizhouapache]

code lgtm

[DaanHoogland]

this should be done in the teardown method,

[Pearl1594]

The intention of removing the shared network from the cleanup list is that, if the deletion of the n/w happens as part of the test, then it need not be cleaned up during the teardown phase, else, if for any reason this test fails or is skipped, the network will be deleted during teardown

[blueorangutan]

Packaging result: ✔centos7 ✔centos8 ✔debian. JID-2768