cAdvisor does not return CPU metrics for 1.4

[mwringe]

cAdvisor is returning empty cpu usage for origin 1.4 (built from master, commit ffdeb1b)

  "stats": [
   {
    "timestamp": "2016-10-03T15:53:03.359772024-04:00",
    "cpu": {
     "usage": {
      "total": 0,
      "user": 0,
      "system": 0
     },

Version

commit ffdeb1b

but I suspect this also affect the v1.4.0-alpha.0

In the v1.3.0 versions, cpu metrics are collected

Steps To Reproduce

1. build origin from git master
2. access the cAdvisor endpoint for container and verify that it has a zero value for cpu.

eg:
curl -k -H "Authorization: Bearer oc whoami -t" -X POST -d '{"num_stats":1}' https://127.0.0.1:10250/stats/${PROJECT_NAME}/${POD_NAME}/${POD_ID}/${CONTAINER_NAME}

Alternatively, install origin metrics (https://docs.openshift.org/latest/install_config/cluster_metrics.html) and verify that the cpu graphs are empty.

Current Result

cpu metrics are all zero

Expected Result

cpu metrics to not be zero

[mwringe]

@spadgett the root cause is cAdvisor in the 1.4 version of OpenShift returing 0 values for the cpu usage

[mwringe]

@derekwaynecarr any idea here? This breaks the console graphing of cpu usage in 1.4

[spadgett]

/cc @jwforres

[derekwaynecarr]

1. Is openshift running as a container?
2. What version of docker?
3. What systemd version (i.e. systemctl --version)

If openshift is running as a container, this will not work until the following is merged upstream and we cherry-pick it into origin 1.4: kubernetes/kubernetes#33806

[mwringe]

docker version: 1.10.3
os: Fedora 24
systemd: 229

OpenShift is running directly on the machine, not in a docker container.

I know that cAdvisor doesn't necessarily work well with a newer systemd, was hoping that would have been resolved by now

[derekwaynecarr]

systemd 229 changed the cgroup hierarchy and put pid 1 in init.scope.
this caused a bug in runc using cgroup driver for systemd.
this bug is fixed in projectatomic/docker-1.12 but it has not yet been published for rpm install on F24.

[mwringe]

I am now running the fedora docker 1.12 rpms which should have this fix, and yet I still don't see any difference here with the cpu usage, its still zero

[derekwaynecarr]

@sjenning -- ptal

[sjenning]

i've been able to recreate. debugging now.

[sjenning]

could be related to this 9607748

[sjenning]

yes, reverting 9607748 fixes the issue