Kubernetes extended test flake: test/e2e/pre_stop.go

[rhcarvalho]

STEP: Destroying namespace "e2e-tests-prestop-jfgg5" for this suite.
• Failure [225.999 seconds]
PreStop [It] should call prestop when killing a pod 
/data/src/github.com/openshift/origin/Godeps/_workspace/src/k8s.io/kubernetes/test/e2e/pre_stop.go:149

  validating pre-stop.
  Expected error:
      <*errors.errorString | 0xc208129060>: {
          s: "timed out waiting for the condition",
      }
      timed out waiting for the condition
  not to have occurred

  /data/src/github.com/openshift/origin/Godeps/_workspace/src/k8s.io/kubernetes/test/e2e/pre_stop.go:141
------------------------------
    fail    PreStop should call prestop when killing a pod (3m45.998920062s)

https://ci.openshift.redhat.com/jenkins/job/test_pull_requests_origin/5717/console

[aveshagarwal]

Is this still an issue? kubernetes upstream e2e tests show this test is passing.

[aveshagarwal]

Currently I am having following behavior in my setup:
Downgrading to nettest:1.3 works and the test passes.

Where with nettest:1.6, the server pod is exiting with following error which is causing this e2e test to fail:

Unable to create client; error: open /var/run/secrets/kubernetes.io/serviceaccount/token: permission denied

[aveshagarwal]

It seems to be related to selinux issue, if I set selinux to permissive on F22, the test case works with nettest-1.6 (which is latest) too.

Following avcs (all seems similar to me) are causing this test to fail:

time->Wed Oct 28 12:41:13 2015
type=AVC msg=audit(1446050473.258:7210): avc: denied { read } for pid=9992 comm="webserver" name="token" dev="tmpfs" ino=499599 scontext=system_u:system_r:svirt_lxc_net_t:s0:c532,c940 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=0

time->Wed Oct 28 12:41:22 2015
type=AVC msg=audit(1446050482.074:7222): avc: denied { read } for pid=10152 comm="webserver" name="token" dev="tmpfs" ino=499599 scontext=system_u:system_r:svirt_lxc_net_t:s0:c375,c823 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=0

time->Wed Oct 28 12:41:42 2015
type=AVC msg=audit(1446050502.786:7225): avc: denied { read } for pid=10237 comm="webserver" name="token" dev="tmpfs" ino=499599 scontext=system_u:system_r:svirt_lxc_net_t:s0:c86,c999 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=0

time->Wed Oct 28 12:42:12 2015
type=AVC msg=audit(1446050532.059:7229): avc: denied { read } for pid=10349 comm="webserver" name="token" dev="tmpfs" ino=499599 scontext=system_u:system_r:svirt_lxc_net_t:s0:c734,c941 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=0

[eparis]

@pmorie

[aveshagarwal]

I have following selinux related rpms on my F22 system:

selinux-policy-targeted-3.13.1-128.16.fc22.noarch
rpm-plugin-selinux-4.12.0.1-12.fc22.x86_64
selinux-policy-3.13.1-128.16.fc22.noarch
libselinux-2.3-10.fc22.x86_64
docker-selinux-1.8.2-1.gitf1db8f2.fc22.x86_64
libselinux-utils-2.3-10.fc22.x86_64
libselinux-python-2.3-10.fc22.x86_64

[pmorie]

Interesting -- I will see if I can recreate this today once I finish some git surgery on volume security

[aveshagarwal]

@pmorie: the error seems to be happening due to failure of following line in pkg/client/unversioned/helper.go. The flow is client.NewInCluster->InClusterConfig

token, err := ioutil.ReadFile("/var/run/secrets/kubernetes.io/serviceaccount/" + api.ServiceAccountTokenKey)

[aveshagarwal]

Same issue with ServiceAccount related e2e tests:

time->Wed Oct 28 18:00:42 2015
type=AVC msg=audit(1446069642.405:8024): avc: denied { read } for pid=23946 comm="mt" name="token" dev="tmpfs" ino=554000 scontext=system_u:system_r:svirt_lxc_net_t:s0:c670,c871 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=0

time->Wed Oct 28 18:00:43 2015
type=AVC msg=audit(1446069643.266:8025): avc: denied { read } for pid=23986 comm="mt" name="ca.crt" dev="tmpfs" ino=553999 scontext=system_u:system_r:svirt_lxc_net_t:s0:c328,c496 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=0

time->Wed Oct 28 18:01:23 2015
type=AVC msg=audit(1446069683.011:8042): avc: denied { read } for pid=24175 comm="mt" name="token" dev="tmpfs" ino=555340 scontext=system_u:system_r:svirt_lxc_net_t:s0:c188,c674 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1

time->Wed Oct 28 18:01:23 2015
type=AVC msg=audit(1446069683.011:8043): avc: denied { open } for pid=24175 comm="mt" path="/var/run/secrets/kubernetes.io/serviceaccount/token" dev="tmpfs" ino=555340 scontext=system_u:system_r:svirt_lxc_net_t:s0:c188,c674 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1

time->Wed Oct 28 18:01:25 2015
type=AVC msg=audit(1446069685.258:8047): avc: denied { open } for pid=24229 comm="mt" path="/var/run/secrets/kubernetes.io/serviceaccount/ca.crt" dev="tmpfs" ino=555339 scontext=system_u:system_r:svirt_lxc_net_t:s0:c492,c824 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1

time->Wed Oct 28 18:01:25 2015
type=AVC msg=audit(1446069685.258:8046): avc: denied { read } for pid=24229 comm="mt" name="ca.crt" dev="tmpfs" ino=555339 scontext=system_u:system_r:svirt_lxc_net_t:s0:c492,c824 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1

time->Wed Oct 28 18:01:29 2015
type=AVC msg=audit(1446069689.502:8061): avc: denied { open } for pid=24361 comm="mt" path="/var/run/secrets/kubernetes.io/serviceaccount/token" dev="tmpfs" ino=556516 scontext=system_u:system_r:svirt_lxc_net_t:s0:c41,c1020 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1

time->Wed Oct 28 18:01:29 2015
type=AVC msg=audit(1446069689.502:8060): avc: denied { read } for pid=24361 comm="mt" name="token" dev="tmpfs" ino=556516 scontext=system_u:system_r:svirt_lxc_net_t:s0:c41,c1020 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1

time->Wed Oct 28 18:01:30 2015
type=AVC msg=audit(1446069690.481:8062): avc: denied { read } for pid=24405 comm="mt" name="ca.crt" dev="tmpfs" ino=556517 scontext=system_u:system_r:svirt_lxc_net_t:s0:c638,c758 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1

time->Wed Oct 28 18:01:30 2015
type=AVC msg=audit(1446069690.481:8063): avc: denied { open } for pid=24405 comm="mt" path="/var/run/secrets/kubernetes.io/serviceaccount/ca.crt" dev="tmpfs" ino=556517 scontext=system_u:system_r:svirt_lxc_net_t:s0:c638,c758 tcontext=unconfined_u:object_r:var_lib_t:s0 tclass=file permissive=1

[pmorie]

@aveshagarwal

Seems like the problem is that the secret volume isn't being relabeled. Would you make a gist of the openshift log?

[aveshagarwal]

As per discussion with @pmorie, the dir /var/lib/lkubelet was mislabled on my F22 (not sure why though) as unconfined_u:object_r:var_lib_t:s0 and changing its label to system_u:object_r:svirt_sandbox_file_t:s0 made PreStop and other e2e tests (secrets and serviceaccounts) pass.