CLOUDSTACK-9305: Cloudstack Usage Breaks if DB HA enabled #1433

Slair1 · 2016-03-09T18:06:38Z

With DB HA enabled in db.properties, the cloudstack-usage service restarts every 10 seconds. Making the suggested change has fixed it for me. Cloudstack 4.8 on Centos7

Slair1 · 2016-03-11T22:11:05Z

@kiwiflyer Very good point, let me make a different change here so it is more generic!

Slair1 · 2016-03-11T22:19:30Z

If anyone still has issues, i also had to make the change from this PR: #1409

kishankavala · 2016-03-17T08:47:58Z

LGTM

kiwiflyer · 2016-03-17T11:55:59Z

Tested patch on a 4.7.1 lab.

Prior to patch:
INFO cloud.usage.UsageServer (logid:) UsageServer ready...
INFO context.support.ClassPathXmlApplicationContext (logid:) Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@70ea0c2a: startup date [Thu Mar 17 06:51:40 CDT 2016]; root of context hierarchy
INFO factory.xml.XmlBeanDefinitionReader (logid:) Loading XML bean definitions from class path resource [usageApplicationContext.xml]
INFO factory.annotation.AutowiredAnnotationBeanPostProcessor (logid:) JSR-330 'javax.inject.Inject' annotation found and supported for autowiring
INFO context.support.ClassPathXmlApplicationContext (logid:) Bean 'transactionContextBuilder' of type [class com.cloud.utils.db.TransactionContextBuilder] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
INFO factory.support.DefaultListableBeanFactory (logid:) Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@33037ca5: defining beans [org.springframework.context.annotation.internalConfigurationAnnotationProcessor,org.springframework.context.annotation.internalAutowiredAnnotationProcessor,org.springframework.context.annotation.internalRequiredAnnotationProcessor,org.springframework.context.annotation.internalCommonAnnotationProcessor,vmDiskUsageParser,networkOfferingUsageParser,VPNUserUsageParser,IPAddressUsageParser,VMSnapshotUsageParser,loadBalancerUsageParser,networkUsageParser,securityGroupUsageParser,storageUsageParser,portForwardingUsageParser,VMInstanceUsageParser,volumeUsageParser,usageManagerImpl,usageAlertManagerImpl,usageVmDiskDaoImpl,usageIPAddressDaoImpl,usageStorageDaoImpl,usageVolumeDaoImpl,usageDaoImpl,usagePortForwardingRuleDaoImpl,usageVPNUserDaoImpl,usageJobDaoImpl,usageLoadBalancerPolicyDaoImpl,usageNetworkDaoImpl,externalPublicIpStatisticsDaoImpl,usageSecurityGroupDaoImpl,usageVMInstanceDaoImpl,usageNetworkOfferingDaoImpl,usageVMSnapshotDaoImpl,usageEventDetailsDaoImpl,usageEventDaoImpl,eventDaoImpl,SSHKeyPairDaoImpl,userDaoImpl,vmDiskStatisticsDaoImpl,userAccountDaoImpl,accountDaoImpl,userStatsLogDaoImpl,userStatisticsDaoImpl,resourceCountDaoImpl,resourceLimitDaoImpl,alertDaoImpl,domainDaoImpl,configurationDaoImpl,quotaEmailTemplatesDaoImpl,userVmDetailsDaoImpl,quotaUsageDaoImpl,quotaBalanceDaoImpl,quotaTariffDaoImpl,quotaAccountDaoImpl,serviceOfferingDaoImpl,quotaCreditsDaoImpl,quotaAlertManagerImpl,quotaStatementImpl,quotaManagerImpl,transactionContextBuilder,instantiatePostProcessor,ComponentContext,org.springframework.context.annotation.ConfigurationClassPostProcessor.importAwareProcessor]; root of factory hierarchy
DEBUG utils.crypt.EncryptionSecretKeyChecker (logid:) Encryption Type: file
INFO db.Transaction.Transaction (logid:) Is Data Base High Availiability enabled? Ans : true
INFO db.Transaction.Transaction (logid:) The slaves configured for Cloud Data base is/are : 10.103.0.154,10.103.0.155
--then service tries to restart every few seconds--

After patch:
INFO utils.component.ComponentContext (logid:) Starting com.cloud.usage.dao.UsageVMSnapshotDaoImpl_EnhancerByCloudStack_c20890bf
INFO utils.component.ComponentContext (logid:) Starting org.apache.cloudstack.quota.QuotaStatementImpl_EnhancerByCloudStack_5d0529b4
INFO cloudstack.quota.QuotaStatementImpl (logid:) Starting Statement Manager
INFO utils.component.ComponentContext (logid:) Starting org.apache.cloudstack.quota.QuotaManagerImpl_EnhancerByCloudStack_3ffc3462
INFO cloudstack.quota.QuotaManagerImpl (logid:) Starting Quota Manager
INFO utils.component.ComponentContext (logid:) Starting com.cloud.usage.UsageManagerImpl_EnhancerByCloudStack_93e4d889
INFO cloud.usage.UsageManagerImpl (logid:) Starting Usage Manager
INFO utils.component.ComponentContext (logid:) Starting com.cloud.usage.UsageAlertManagerImpl_EnhancerByCloudStack_8fa0a2a9
INFO utils.component.ComponentContext (logid:) Starting org.apache.cloudstack.quota.QuotaAlertManagerImpl_EnhancerByCloudStack_9535c670
INFO cloudstack.quota.QuotaAlertManagerImpl (logid:) Starting Alert Manager
INFO cloud.usage.UsageServer (logid:) UsageServer ready...
--service starts correctly--

LGTM on e98f445

swill · 2016-04-07T15:10:13Z

@Slair1 can you please squash your commits. I will get this into my CI queue so we can get this merged... Thanks...

kiwiflyer · 2016-04-07T15:13:35Z

@swill Just FYI that this PR conflicts with #1409 (comment).

They fix different issues, but the patch conflicts.

swill · 2016-04-07T15:17:03Z

@kiwiflyer they conflict in what sense? They change different lines of code, so I don't expect git to have an issue with them. Can you clarify?

kiwiflyer · 2016-04-07T18:13:19Z

I applied 1433 and then tried to apply 1409, but 1433 changes prevent the later patch from matching previous lines. One or the other probably needs to be based on top of the other patch so it applies cleanly.

Slair1 · 2016-04-07T18:27:00Z

@swill , i apologize, but this is my first PR. I squashed it in Github Desktop on my local copy and did a push, but it sees like it it just added additional commits at github.com. If could you let me know how to correct it i'll do so as soon as I can.

swill · 2016-04-07T19:44:07Z

No worries. :) This should work: https://github.com/ginatrapani/todo.txt-android/wiki/Squash-All-Commits-Related-to-a-Single-Issue-into-a-Single-Commit

Also described here: https://ariejan.net/2011/07/05/git-squash-your-latests-commits-into-one/

I have used this method in the past as well: http://stackoverflow.com/questions/616556/how-do-you-squash-commits-into-one-patch-with-git-format-patch?answertab=votes#tab-top

Let me know if you have problems and I will try to help you out.

Slair1 · 2016-04-07T20:34:06Z

@swill i seem to get an access denied when pushing my change:
C:\slair\Source\Github\cloudstack [pr/1433]> git push origin pr/1433 --force
remote: Permission to apache/cloudstack.git denied to Slair1.
fatal: unable to access 'https://github.com/apache/cloudstack.git/': The requested URL returned error: 403

swill · 2016-04-07T21:45:46Z

No one is allowed to push code to apache/cloudstack, you have to push it to your fork where your PR is coming from. Does that make sense?

So you will be pushing to https://github.com/Slair1/cloudstack on branch patch-1. This will likely be setup as a different git remote It looks like your origin is pointing at the apache/cloudstack repo instead of your Slair1/cloudstack repo.

Slair1 · 2016-04-07T23:51:52Z

@swill thanks, that makes sense. I did it to my branch the first time, but for some reason when i tried to save to here it just created two more commits. i'll try again

swill · 2016-04-08T00:05:01Z

Once you get your local branch setup correctly with the squashed commit, you should be able to do a push - f to replace what is in the PR with what you have locally. Let me know if you have problems and I can do a test to validate the steps you need to take.

With DB HA enabled in db.properties, the cloudstack-usage service restarts every 10 seconds. Making the suggested change has fixed it for me. Cloudstack 4.8 on Centos7 Update cloudstack-usage.service

Slair1 · 2016-04-08T00:46:53Z

thanks @swill , it looks much better now, i needed the -f on the git push

swill · 2016-04-08T01:01:48Z

No problem. Ya it took me a couple pull requests to figure out how they work. Thanks for taking the time to do this for me. :)

swill · 2016-04-28T20:23:33Z

Running CI now. Will have results later tonight or tomorrow...

swill · 2016-04-29T02:45:49Z

CI RESULTS

Tests Run: 103
  Skipped: 0
   Failed: 3
   Errors: 2

Summary of the problem(s):

ERROR: Test Create/Delete a manual snap shot and verify
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/git/cs2/cloudstack/test/integration/component/test_project_usage.py", line 1376, in test_01_snapshot_usage
    snapshot = Snapshot.create(self.apiclient, volumes[0].id)
  File "/usr/lib/python2.7/site-packages/marvin/lib/base.py", line 1018, in create
    return Snapshot(apiclient.createSnapshot(cmd).__dict__)
  File "/usr/lib/python2.7/site-packages/marvin/cloudstackAPI/cloudstackAPIClient.py", line 797, in createSnapshot
    response = self.connection.marvinRequest(command, response_type=response, method=method)
  File "/usr/lib/python2.7/site-packages/marvin/cloudstackConnection.py", line 379, in marvinRequest
    raise e
CloudstackAPIException: Execute cmd: createsnapshot failed, due to: errorCode: 530, errorText:KVM Snapshot is not supported: 1
----------------------------------------------------------------------
Additional details in: /tmp/MarvinLogs/test_usage_A3LTTO/results.txt

ERROR: Test Create/Delete a manual snap shot and verify
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/git/cs2/cloudstack/test/integration/component/test_usage.py", line 1300, in test_01_snapshot_usage
    snapshot = Snapshot.create(self.apiclient, volumes[0].id)
  File "/usr/lib/python2.7/site-packages/marvin/lib/base.py", line 1018, in create
    return Snapshot(apiclient.createSnapshot(cmd).__dict__)
  File "/usr/lib/python2.7/site-packages/marvin/cloudstackAPI/cloudstackAPIClient.py", line 797, in createSnapshot
    response = self.connection.marvinRequest(command, response_type=response, method=method)
  File "/usr/lib/python2.7/site-packages/marvin/cloudstackConnection.py", line 379, in marvinRequest
    raise e
CloudstackAPIException: Execute cmd: createsnapshot failed, due to: errorCode: 530, errorText:KVM Snapshot is not supported: 1
----------------------------------------------------------------------
Additional details in: /tmp/MarvinLogs/test_usage_A3LTTO/results.txt

FAIL: Test redundant router internals
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/git/cs2/cloudstack/test/integration/smoke/test_routers_network_ops.py", line 290, in test_01_RVR_Network_FW_PF_SSH_default_routes_egress_true
    "Attempt to retrieve google.com index page should be successful!"
AssertionError: Attempt to retrieve google.com index page should be successful!
----------------------------------------------------------------------
Additional details in: /tmp/MarvinLogs/test_network_4ISAIJ/results.txt

FAIL: test_02_vpc_privategw_static_routes (integration.smoke.test_privategw_acl.TestPrivateGwACL)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/git/cs2/cloudstack/test/integration/smoke/test_privategw_acl.py", line 253, in test_02_vpc_privategw_static_routes
    self.performVPCTests(vpc_off)
  File "/data/git/cs2/cloudstack/test/integration/smoke/test_privategw_acl.py", line 324, in performVPCTests
    self.check_pvt_gw_connectivity(vm1, public_ip_1, vm2.nic[0].ipaddress)
  File "/data/git/cs2/cloudstack/test/integration/smoke/test_privategw_acl.py", line 559, in check_pvt_gw_connectivity
    "Ping to outside world from VM should be successful"
AssertionError: Ping to outside world from VM should be successful
----------------------------------------------------------------------
Additional details in: /tmp/MarvinLogs/test_network_4ISAIJ/results.txt

FAIL: test_03_vpc_privategw_restart_vpc_cleanup (integration.smoke.test_privategw_acl.TestPrivateGwACL)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/data/git/cs2/cloudstack/test/integration/smoke/test_privategw_acl.py", line 265, in test_03_vpc_privategw_restart_vpc_cleanup
    self.performVPCTests(vpc_off, True)
  File "/data/git/cs2/cloudstack/test/integration/smoke/test_privategw_acl.py", line 324, in performVPCTests
    self.check_pvt_gw_connectivity(vm1, public_ip_1, vm2.nic[0].ipaddress)
  File "/data/git/cs2/cloudstack/test/integration/smoke/test_privategw_acl.py", line 559, in check_pvt_gw_connectivity
    "Ping to outside world from VM should be successful"
AssertionError: Ping to outside world from VM should be successful
----------------------------------------------------------------------
Additional details in: /tmp/MarvinLogs/test_network_4ISAIJ/results.txt

Associated Uploads

/tmp/MarvinLogs/DeployDataCenter__Apr_28_2016_22_51_48_T2XTTQ:

/tmp/MarvinLogs/test_network_4ISAIJ:

/tmp/MarvinLogs/test_usage_A3LTTO:

/tmp/MarvinLogs/test_vpc_routers_LS4J5A:

Uploads will be available until 2016-06-29 02:00:00 +0200 CEST

Comment created by upr comment.

swill · 2016-04-29T02:48:15Z

None of the failures above are related to this PR, they are all issues I see periodically in my environment. All of the supported usage tests passed, so CI has passed for this PR. Sorry the output makes that hard to understand.

Slair1 · 2016-04-29T02:50:17Z

That's ok, thanks!!

kiwiflyer · 2016-04-29T18:29:48Z

2 x LGTMs and CI complete. This is Ready to Merge.

swill · 2016-04-29T18:58:06Z

Sorry, I missed the LGTMs. :) I will add this to my merge queue. Thanks @kiwiflyer for following up.

yadvr · 2016-05-02T12:41:30Z

@Slair1 if dh ha jar is missing for example there is no db-ha deb package, what happens?

Slair1 · 2016-05-02T14:03:16Z

Hi @rhtyd since it uses "ls" to get a list of packages, if the directory or .jar file does not exist it is no big deal. It will output a No such file or directory message but still loads any packages that exist without issue. the UCP variable still looks good when that occurs.

yadvr · 2016-05-02T17:40:13Z

fair enough, LGTM

tag:mergeready

CLOUDSTACK-9305: Cloudstack Usage Breaks if DB HA enabledWith DB HA enabled in db.properties, the cloudstack-usage service restarts every 10 seconds. Making the suggested change has fixed it for me. Cloudstack 4.8 on Centos7 * pr/1433: Cloudstack Usage Breaks if DB HA enabled Signed-off-by: Will Stevens <williamstevens@gmail.com>

Slair1 changed the title ~~Cloudstack Usage Breaks if DB HA enabled~~ CLOUDSTACK-9305: Cloudstack Usage Breaks if DB HA enabled Mar 9, 2016

Slair1 mentioned this pull request Mar 10, 2016

Addresses CLOUDSTACK-9300 where the MySQL HA StaticStrategy class fai… #1428

Merged

Cloudstack Usage Breaks if DB HA enabled

6089d89

With DB HA enabled in db.properties, the cloudstack-usage service restarts every 10 seconds. Making the suggested change has fixed it for me. Cloudstack 4.8 on Centos7 Update cloudstack-usage.service

Slair1 force-pushed the patch-1 branch from a19445a to 6089d89 Compare April 8, 2016 00:45

asfgit merged commit 6089d89 into apache:master May 4, 2016

Slair1 deleted the patch-1 branch May 4, 2016 15:23

CLOUDSTACK-9305: Cloudstack Usage Breaks if DB HA enabled #1433

CLOUDSTACK-9305: Cloudstack Usage Breaks if DB HA enabled #1433

Uh oh!

Conversation

Slair1 commented Mar 9, 2016

Uh oh!

Slair1 commented Mar 11, 2016

Uh oh!

Slair1 commented Mar 11, 2016

Uh oh!

kishankavala commented Mar 17, 2016

Uh oh!

kiwiflyer commented Mar 17, 2016

Uh oh!

swill commented Apr 7, 2016

Uh oh!

kiwiflyer commented Apr 7, 2016

Uh oh!

swill commented Apr 7, 2016

Uh oh!

kiwiflyer commented Apr 7, 2016

Uh oh!

Slair1 commented Apr 7, 2016

Uh oh!

swill commented Apr 7, 2016

Uh oh!

Slair1 commented Apr 7, 2016

Uh oh!

swill commented Apr 7, 2016

Uh oh!

Slair1 commented Apr 7, 2016

Uh oh!

swill commented Apr 8, 2016

Uh oh!

Slair1 commented Apr 8, 2016

Uh oh!

swill commented Apr 8, 2016

Uh oh!

swill commented Apr 28, 2016

Uh oh!

swill commented Apr 29, 2016

CI RESULTS

Uh oh!

swill commented Apr 29, 2016

Uh oh!

Slair1 commented Apr 29, 2016

Uh oh!

kiwiflyer commented Apr 29, 2016

Uh oh!

swill commented Apr 29, 2016

Uh oh!

yadvr commented May 2, 2016

Uh oh!

Slair1 commented May 2, 2016

Uh oh!

yadvr commented May 2, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants