CPU utilization metrics seems to be inconsistent

[kioie]

ISSUE TYPE

• Bug Report

COMPONENT NAME

CPU utilization

CLOUDSTACK VERSION

4.11+

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

CPU utilization metrics seems to be inconsistent with the actual CPU usage in the VM. The error seems to be generated from this getVMStat method in the LibvirtComputingResource class

 /* get cpu utilization */
            VmStats oldStats = null;

            final Calendar now = Calendar.getInstance();

            oldStats = _vmStats.get(vmName);

            long elapsedTime = 0;
            if (oldStats != null) {
                elapsedTime = now.getTimeInMillis() - oldStats._timestamp.getTimeInMillis();
                double utilization = (info.cpuTime - oldStats._usedTime) / ((double)elapsedTime * 1000000);

                final NodeInfo node = conn.nodeInfo();
                utilization = utilization / node.cpus;
                if (utilization > 0) {
                    stats.setCPUUtilization(utilization * 100);
                }
            }

STEPS TO REPRODUCE

Run stress test on the VM with 100% cpu utilization then confirm VM Metric on the statistics tab 

EXPECTED RESULTS

CPU utilization should be equally as high

ACTUAL RESULTS

CPU utilization is inconsistent i.e 8.4% (as shown)

[yadvr]

It takes upto the time set by the background thread to kick in. Refer to stats collector related timeout/interval global settings.

[kioie]

Okay...so typically, at what point would it be triggered for it to kick in? Been running some stress tests on several VMs for 3.5hrs now, seems to be returning the same metrics

[yadvr]

The vm.stats.interval is typically 60s. As long as your KVM agent is UP and able to process VM stats every 60s, you should see the stats correctly. Did you try against 4.11/latest or master?

I can confirm against master and KVM and get back to you next week. In my local lab I'm unable to see the issue against 4.11.3.0+kvm.

[kioie]

To confirm my config for vm.stats.interval is also at 60s also on KVM. Incidentally, looking at the code, I think VMWare and Xen are handling this differently, so they may possibly be returning correct results (haven't tested though)

Yes, tested on 4.11.2.0/4.12/master with ubuntu16.04/18.04/Centos7 vms

My stress test ended up running for 5hrs and the metrics never really changed.

[DaanHoogland]

@kioie, is this still an issue?

[DaanHoogland]

closing this @kioie , feel free to reopen if you encounter the issue again.

[weizhouapache]

this is fixed by #4381