local-storage-provisioner pod creation failed on compute node.

[whitingjr]

[provide a description of the issue]

On one compute node the creation of the local storage provisioner pod fails. This is the event

Warning FailedCreatePodContainer 2s (x6 over 1m) kubelet, benchserver2 unable to ensure pod container exists: failed to create container for /kubepods/besteffort/podb8ba5573-c097-11e8-be7f-941882782994 : dbus: connection closed by user

Version

oc version

oc v3.10.34
kubernetes v1.10.0+b81c8f8
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://benchserver7:8443
openshift v3.10.34
kubernetes v1.10.0+b81c8f8

Installed using the openshift-ansible repository release branch 3.10 in Git.

Steps To Reproduce

1. install ocp 3.10 using the git release-3.10 branch using 1 master and 2 compute nodes (named benchserver5, benchserver2 in this issue)
2. Configure local volumes
3. delete the daemonset local-volume-provisioner
4. re-create the local-storage-provisioner again using oc new-app command

Current Result

2 pods are scheduled. 1 for each compute node.
on benchserver5 the pod status is Running. On benchserver2 the problem pod status is ContainerCreating.

oc get pods

NAME READY STATUS RESTARTS AGE
local-volume-provisioner-ccb64 1/1 Running 0 35m
local-volume-provisioner-qshl4 0/1 ContainerCreating 0 35m

Expected Result

the pod container instance on both compute nodes achieve Running status.

Additional Information

console output for oc describe po local-volume-provisioner-tzqr4

console output for oc adm diagnostics

console output for oc get all -o json -n xxx

journalctl log

event log

uname -a

Linux benchserver7 3.10.0-862.3.2.el7.x86_64 #1 SMP Tue May 15 18:22:15 EDT 2018 x86_64 x86_64 x86_64 GNU/Linux

docker version

Client:
Version: 1.13.1
API version: 1.26
Package version: docker-1.13.1-68.gitdded712.el7.x86_64
Go version: go1.9.2
Git commit: dded712/1.13.1
Built: Tue Jun 12 18:30:09 2018
OS/Arch: linux/amd64

Server:
Version: 1.13.1
API version: 1.26 (minimum version 1.12)
Package version: docker-1.13.1-68.gitdded712.el7.x86_64
Go version: go1.9.2
Git commit: dded712/1.13.1
Built: Tue Jun 12 18:30:09 2018
OS/Arch: linux/amd64
Experimental: false

[whitingjr]

rebooting the compute nodes meant this issue magically went away.