reuse ip for non redundant VPC #2712

DaanHoogland · 2018-06-18T11:05:58Z

As rolling restart does not deallocate an IP before configuring it on a new VR,
the code must allow it to be reused on a non-redundant VPCs gateway nic.

Description

removed the condition to only look for reuse of IP on redundant VPCs

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

GitHub Issue/PRs

Screenshots (if appropriate):

How Has This Been Tested?

Checklist:

I have read the CONTRIBUTING document.
My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
Testing
I have added tests to cover my changes.
All relevant new and existing integration tests have passed.
A full integration testsuite with all test that can run on my environment has passed.

DaanHoogland · 2018-06-18T11:08:08Z

@blueorangutan package

blueorangutan · 2018-06-18T11:08:42Z

@DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2018-06-18T11:41:27Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2127

DaanHoogland · 2018-06-18T12:08:56Z

@blueorangutan test

blueorangutan · 2018-06-18T12:09:45Z

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2018-06-18T20:58:59Z

Trillian test result (tid-2762)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 29911 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2712-t2762-kvm-centos7.zip
Intermitten failure detected: /marvin/tests/smoke/test_deploy_virtio_scsi_vm.py
Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermitten failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermitten failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermitten failure detected: /marvin/tests/smoke/test_host_maintenance.py
Intermitten failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 62 look OK, 5 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestDeployVirtioSCSIVM>:setup	`Error`	0.00	test_deploy_virtio_scsi_vm.py
test_02_vpc_privategw_static_routes	`Failure`	189.01	test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanup	`Failure`	190.38	test_privategw_acl.py
test_02_not_secured_vm_migration	`Error`	5.21	test_vm_life_cycle.py
test_03_secured_to_nonsecured_vm_migration	`Error`	5.17	test_vm_life_cycle.py
test_04_nonsecured_to_secured_vm_migration	`Error`	1.09	test_vm_life_cycle.py
test_08_migrate_vm	`Error`	6.33	test_vm_life_cycle.py
test_01_cancel_host_maintenace_with_no_migration_jobs	`Failure`	0.08	test_host_maintenance.py
test_02_cancel_host_maintenace_with_migration_jobs	`Error`	1.26	test_host_maintenance.py
test_hostha_enable_ha_when_host_in_maintenance	`Error`	1.42	test_hostha_kvm.py

yadvr · 2018-06-19T06:15:04Z

@blueorangutan package

blueorangutan · 2018-06-19T06:15:42Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2018-06-19T06:41:38Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2131

PaulAngus · 2018-06-19T08:52:58Z

@DaanHoogland have you looked at the VPC private gateway test failures?

DaanHoogland · 2018-06-19T13:39:29Z

The failure is a failure to ping. This may have many causes. I think we need to manually test this functionality, to see if either the test or the private gateway (or acl/fw) functionality needs fixing, @PaulAngus

PaulAngus · 2018-06-19T15:38:44Z

@blueorangutan test

blueorangutan · 2018-06-19T15:39:43Z

@PaulAngus a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

DaanHoogland · 2018-06-19T20:26:32Z

After fixing the original problem a new bug arises. When restarting a VPC with a tier, the tier seems to come back unconnected and the VPC router and the interface for it has no IP address.

blueorangutan · 2018-06-20T00:03:21Z

Trillian test result (tid-2771)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 28506 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2712-t2771-kvm-centos7.zip
Intermitten failure detected: /marvin/tests/smoke/test_deploy_virtio_scsi_vm.py
Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermitten failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermitten failure detected: /marvin/tests/smoke/test_host_maintenance.py
Smoke tests completed. 63 look OK, 4 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestDeployVirtioSCSIVM>:setup	`Error`	0.00	test_deploy_virtio_scsi_vm.py
test_02_vpc_privategw_static_routes	`Failure`	191.27	test_privategw_acl.py
test_03_vpc_privategw_restart_vpc_cleanup	`Failure`	1205.21	test_privategw_acl.py
test_04_rvpc_privategw_static_routes	`Failure`	382.48	test_privategw_acl.py
test_01_secured_vm_migration	`Error`	1.09	test_vm_life_cycle.py
test_02_not_secured_vm_migration	`Error`	1.09	test_vm_life_cycle.py
test_03_secured_to_nonsecured_vm_migration	`Error`	1.09	test_vm_life_cycle.py
test_04_nonsecured_to_secured_vm_migration	`Error`	1.10	test_vm_life_cycle.py
test_08_migrate_vm	`Error`	7.27	test_vm_life_cycle.py
test_10_attachAndDetach_iso	`Error`	12.89	test_vm_life_cycle.py
test_01_cancel_host_maintenace_with_no_migration_jobs	`Failure`	0.09	test_host_maintenance.py
test_02_cancel_host_maintenace_with_migration_jobs	`Error`	39.94	test_host_maintenance.py

DaanHoogland · 2018-06-20T14:52:44Z

After some more testing locally I see that sometimes restart with cleanup works and sometimes it doesn't. At one occasion I tried to make a VPC redundant and ...

I undid the redundancy and after that restarts just worked. The use case in the test is to have two VPCs connected through their private gateways and static routes added towards each other. Then a VPC is restarted with cleanup and it is tested whether the VMs are still pingable.

I am now convinced that the improvement in this PR does not solve everything but neither does it cause the errors in the restart.

DaanHoogland · 2018-06-20T15:39:40Z

@blueorangutan package

blueorangutan · 2018-06-20T15:40:43Z

@DaanHoogland a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2018-06-20T16:25:04Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-2140

DaanHoogland · 2018-06-20T16:34:31Z

@blueorangutan test

blueorangutan · 2018-06-20T16:34:43Z

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2018-06-21T02:21:02Z

Trillian test result (tid-2779)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 33440 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2712-t2779-kvm-centos7.zip
Intermitten failure detected: /marvin/tests/smoke/test_deploy_virtio_scsi_vm.py
Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermitten failure detected: /marvin/tests/smoke/test_vm_life_cycle.py
Intermitten failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermitten failure detected: /marvin/tests/smoke/test_vpc_router_nics.py
Intermitten failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 63 look OK, 4 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
ContextSuite context=TestDeployVirtioSCSIVM>:setup	`Error`	0.00	test_deploy_virtio_scsi_vm.py
test_03_vpc_privategw_restart_vpc_cleanup	`Failure`	177.62	test_privategw_acl.py
test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL	`Error`	132.50	test_vpc_redundant.py
test_02_redundant_VPC_default_routes	`Failure`	280.05	test_vpc_redundant.py
test_04_rvpc_network_garbage_collector_nics	`Failure`	294.14	test_vpc_redundant.py
test_02_VPC_default_routes	`Failure`	141.28	test_vpc_router_nics.py

As rolling restart does not deallocate an IP before configuring it on a new VR, the code must allow it to be reused on a non-redundant VPCs gateway nic.

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

yadvr · 2018-06-21T06:53:29Z

Ran tests manually with increased ping thresholds: (rebased against 4.11, no changes to the original commit by Daan)

test_vpc_router_nics.py:
Create a VPC with two networks with one VM in each network and test nics after destroy ... === TestName: test_01_VPC_nics_after_destroy | Status : SUCCESS ===
ok
Create a VPC with two networks with one VM in each network and test default routes ... === TestName: test_02_VPC_default_routes | Status : SUCCESS ===
ok

----------------------------------------------------------------------
Ran 2 tests in 566.266s

OK

test_vpc_redundant.py:

Create a redundant VPC with two networks with two VMs in each network ... === TestName: test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL | Status : SUCCESS ===
ok
Create a redundant VPC with two networks with two VMs in each network and check default routes ... === TestName: test_02_redundant_VPC_default_routes | Status : SUCCESS ===
ok
Create a redundant VPC with two networks with two VMs in each network ... === TestName: test_03_create_redundant_VPC_1tier_2VMs_2IPs_2PF_ACL_reboot_routers | Status : SUCCESS ===
ok
Create a redundant VPC with 1 Tier, 1 VM, 1 ACL, 1 PF and test Network GC Nics ... === TestName: test_04_rvpc_network_garbage_collector_nics | Status : SUCCESS ===
ok
Create a redundant VPC with multiple tiers ... === TestName: test_05_rvpc_multi_tiers | Status : SUCCESS ===
ok

----------------------------------------------------------------------
Ran 5 tests in 2354.352s

OK


test_privategw_acl.py:

test_01_vpc_privategw_acl (integration.smoke.test_privategw_acl.TestPrivateGwACL) ... === TestName: test_01_vpc_privategw_acl | Status : SUCCESS ===
ok
test_02_vpc_privategw_static_routes (integration.smoke.test_privategw_acl.TestPrivateGwACL) ... === TestName: test_02_vpc_privategw_static_routes | Statu
s : SUCCESS ===
ok
test_03_vpc_privategw_restart_vpc_cleanup (integration.smoke.test_privategw_acl.TestPrivateGwACL) ... === TestName: test_03_vpc_privategw_restart_vpc_cle
anup | Status : FAILED ===
FAIL
test_04_rvpc_privategw_static_routes (integration.smoke.test_privategw_acl.TestPrivateGwACL) ... === TestName: test_04_rvpc_privategw_static_routes | Sta
tus : SUCCESS ===
ok

DaanHoogland · 2018-06-21T07:30:59Z

thanks @rhtyd so we merge? (cc @PaulAngus )

yadvr · 2018-06-21T07:31:46Z

Still working/testing @DaanHoogland I'll confirm and post once results are here.

yadvr · 2018-06-21T08:14:03Z

The test_03_vpc_privategw_restart_vpc_cleanup | Status : FAILED === test has been consistently failing for me. What I observe is that post restart with cleanup=true of the VPC VR it is unable to connect to the guest VM. I'm investigating this case now.

DaanHoogland · 2018-06-21T08:19:09Z

I know about test_03_vpc_privategw_restart_vpc_cleanup @rhtyd . The only time I got this functionality to work was on manual testing. I think there is something with the order of applying interface and addresses that makes it work or fail :(

yadvr · 2018-06-21T09:34:20Z

There was no clear solution for fixing the nic/ip sequencing/application issue, I could reproduce the error sometimes and sometimes the test would pass when the sequence applied cleanly. Given a workaround for the non-redundant restart VPC with private gateway enabled exists in case a sequence issue is hit (i.e. user should manually destroy the VPC VR and restart the VPC without cleanup selected), I'll merge this /cc @PaulAngus. I verified that the issue does not arise in case of redundant VPCs.

yadvr · 2018-06-21T09:42:41Z

Issue logged - #2719

As rolling restart does not deallocate an IP before configuring it on a new VR, the code must allow it to be reused on a non-redundant VPCs gateway nic. In crease ping counts to reduce intermittent failures in smoketests. Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

DaanHoogland added this to the 4.11.1.0 milestone Jun 18, 2018

DaanHoogland added the Severity:BLOCKER label Jun 18, 2018

DaanHoogland self-assigned this Jun 18, 2018

DaanHoogland requested review from PaulAngus, nvazquez and yadvr June 18, 2018 11:07

yadvr approved these changes Jun 19, 2018

View reviewed changes

PaulAngus added type:question component:vpc status:testing status:waiting-for-author status:waiting-for-reviewer labels Jun 19, 2018

apache deleted a comment from blueorangutan Jun 19, 2018

DaanHoogland removed type:question status:waiting-for-author labels Jun 20, 2018

DaanHoogland removed the status:waiting-for-reviewer label Jun 20, 2018

PaulAngus added status:work-in-progress and removed status:testing labels Jun 20, 2018

rafaelweingartner approved these changes Jun 20, 2018

View reviewed changes

Daan Hoogland and others added 2 commits June 21, 2018 11:11

reuse ip for non redundant VPC

f8fa446

As rolling restart does not deallocate an IP before configuring it on a new VR, the code must allow it to be reused on a non-redundant VPCs gateway nic.

smoketest: increase pind counts to reduce intermittent failures

24a5baa

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

yadvr force-pushed the privateGatewayRestartVpcFix branch from 8d45f5a to 24a5baa Compare June 21, 2018 06:18

yadvr removed the status:work-in-progress label Jun 21, 2018

yadvr merged commit 52b02de into apache:4.11 Jun 21, 2018

yadvr mentioned this pull request Jun 21, 2018

Restarting of non-redundant VPC with private gateway fails sometimes #2719

Closed

snyk-bot mentioned this pull request Jul 16, 2021

[Snyk] Security upgrade karma from 1.7.1 to 2.0.0 hixiomh/cloudstack#53

Open

snyk-bot mentioned this pull request Sep 4, 2021

[Snyk] Security upgrade karma from 1.7.1 to 2.0.0 hixiomh/cloudstack#97

Open

DaanHoogland removed their assignment Jun 13, 2023

reuse ip for non redundant VPC #2712

reuse ip for non redundant VPC #2712

Uh oh!

Conversation

DaanHoogland commented Jun 18, 2018

Description

Types of changes

GitHub Issue/PRs

Screenshots (if appropriate):

How Has This Been Tested?

Checklist:

Uh oh!

DaanHoogland commented Jun 18, 2018

Uh oh!

blueorangutan commented Jun 18, 2018

Uh oh!

blueorangutan commented Jun 18, 2018

Uh oh!

DaanHoogland commented Jun 18, 2018

Uh oh!

blueorangutan commented Jun 18, 2018

Uh oh!

blueorangutan commented Jun 18, 2018

Uh oh!

yadvr commented Jun 19, 2018

Uh oh!

blueorangutan commented Jun 19, 2018

Uh oh!

blueorangutan commented Jun 19, 2018

Uh oh!

PaulAngus commented Jun 19, 2018

Uh oh!

DaanHoogland commented Jun 19, 2018

Uh oh!

PaulAngus commented Jun 19, 2018

Uh oh!

blueorangutan commented Jun 19, 2018

Uh oh!

DaanHoogland commented Jun 19, 2018

Uh oh!

blueorangutan commented Jun 20, 2018

Uh oh!

DaanHoogland commented Jun 20, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DaanHoogland commented Jun 20, 2018

Uh oh!

blueorangutan commented Jun 20, 2018

Uh oh!

blueorangutan commented Jun 20, 2018

Uh oh!

DaanHoogland commented Jun 20, 2018

Uh oh!

blueorangutan commented Jun 20, 2018

Uh oh!

blueorangutan commented Jun 21, 2018

Uh oh!

yadvr commented Jun 21, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DaanHoogland commented Jun 21, 2018

Uh oh!

yadvr commented Jun 21, 2018

Uh oh!

yadvr commented Jun 21, 2018

Uh oh!

DaanHoogland commented Jun 21, 2018

Uh oh!

yadvr commented Jun 21, 2018

Uh oh!

yadvr commented Jun 21, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

DaanHoogland commented Jun 20, 2018 •

edited

Loading

yadvr commented Jun 21, 2018 •

edited

Loading