"panic: runtime error: invalid memory address or nil pointer dereference" on armhfp

[mmilata]

I am unable to start openshift (9e304e5) on armhfp - it produces the following stacktrace:

# _output/local/go/bin/openshift start --loglevel=5

I0409 08:49:39.469638    2368 start_allinone.go:216] Starting an OpenShift all-in-one
(generating certificates)
I0409 08:53:38.882580    2368 start_master.go:287] Starting an OpenShift master, reachable at 0.0.0.0:8443 (etcd: [https://10.12.0.86:4001])
I0409 08:53:38.882715    2368 start_master.go:288] OpenShift master public address is https://10.12.0.86:8443
I0409 08:53:41.432693    2368 server.go:65] etcd: peerTLS: cert = openshift.local.certificates/etcd/server.crt, key = openshift.local.certificates/etcd/server.key, ca = openshift.local.certificates/ca/cert.crt
I0409 08:53:43.983565    2368 server.go:76] etcd: listening for peers on https://0.0.0.0:7001
I0409 08:53:43.983702    2368 server.go:68] etcd: clientTLS: cert = openshift.local.certificates/etcd/server.crt, key = openshift.local.certificates/etcd/server.key, ca = openshift.local.certificates/ca/cert.crt
I0409 08:53:46.531884    2368 server.go:68] etcd: listening for client requests on https://0.0.0.0:4001
2015/04/09 08:53:46 etcdserver: datadir is valid for the 2.0.1 format
2015/04/09 08:53:46 etcdserver: name = openshift.local
2015/04/09 08:53:46 etcdserver: data dir = openshift.local.etcd
2015/04/09 08:53:46 etcdserver: member dir = openshift.local.etcd/member
2015/04/09 08:53:46 etcdserver: heartbeat = 100ms
2015/04/09 08:53:46 etcdserver: election = 1000ms
2015/04/09 08:53:46 etcdserver: snapshot count = 0
2015/04/09 08:53:46 etcdserver: advertise client URLs = https://10.12.0.86:4001
2015/04/09 08:53:46 etcdserver: initial advertise peer URLs = https://10.12.0.86:7001
2015/04/09 08:53:46 etcdserver: initial cluster = openshift.local=https://10.12.0.86:7001
2015/04/09 08:53:46 etcdserver: start member eb0a4ac20fc1f6c6 in cluster 184b88513aa166b2
2015/04/09 08:53:46 raft: eb0a4ac20fc1f6c6 became follower at term 0
2015/04/09 08:53:46 raft: newRaft eb0a4ac20fc1f6c6 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
2015/04/09 08:53:46 raft: eb0a4ac20fc1f6c6 became follower at term 1
2015/04/09 08:53:46 etcdserver: set snapshot count to default 10000
I0409 08:53:46.566037    2368 etcd.go:66] Started etcd at 10.12.0.86:4001
2015/04/09 08:53:46 etcdserver: added local member eb0a4ac20fc1f6c6 [https://10.12.0.86:7001] to cluster 184b88513aa166b2
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x4 pc=0x870ec]

goroutine 32 [running]:
sync/atomic.storeUint64(0x9703f10c, 0x1, 0x0)
        /usr/lib/golang/src/sync/atomic/64bit_arm.go:20 +0x48
github.com/coreos/etcd/etcdserver.(*EtcdServer).apply(0x9703f0e0, 0x96fc1530, 0x1, 0x1, 0x96fcb060, 0x0, 0x0, 0x0)
        /root/origin/Godeps/_workspace/src/github.com/coreos/etcd/etcdserver/server.go:704 +0x210
github.com/coreos/etcd/etcdserver.(*EtcdServer).run(0x9703f0e0)
        /root/origin/Godeps/_workspace/src/github.com/coreos/etcd/etcdserver/server.go:405 +0xfac
created by github.com/coreos/etcd/etcdserver.(*EtcdServer).start
        /root/origin/Godeps/_workspace/src/github.com/coreos/etcd/etcdserver/server.go:298 +0x2cc

goroutine 1 [runnable]:
math/big.divWW_g(0xbc9c50f4, 0xc0fa6e, 0xc0d95486, 0x96fd3d70, 0x35)
        /usr/lib/golang/src/math/big/arith.go:117
math/big.nat.divLarge(0x96fd3e60, 0x20, 0x45, 0x96fd3d40, 0x41, 0x44, 0x96fd3c20, 0x40, 0x44, 0x9710bd60, ...)
        /usr/lib/golang/src/math/big/nat.go:564 +0x95c
math/big.nat.div(0x96fd3e60, 0x20, 0x45, 0x96fd3d40, 0x40, 0x44, 0x96fd3c20, 0x40, 0x44, 0x9710bd60, ...)
        /usr/lib/golang/src/math/big/nat.go:519 +0x27c
math/big.nat.expNNWindowed(0x96fd3c20, 0x40, 0x44, 0x9703f560, 0x20, 0x24, 0x9703f440, 0x20, 0x24, 0x9710bd60, ...)
        /usr/lib/golang/src/math/big/nat.go:1351 +0x788
math/big.nat.expNN(0x9703f5f0, 0x20, 0x24, 0x9703f560, 0x20, 0x24, 0x9703f440, 0x20, 0x24, 0x9710bd60, ...)
        /usr/lib/golang/src/math/big/nat.go:1260 +0x4dc
math/big.nat.probablyPrime(0x9710bd60, 0x20, 0x25, 0x14, 0x401)
        /usr/lib/golang/src/math/big/nat.go:1445 +0x6c4
math/big.(*Int).ProbablyPrime(0x970c82a0, 0x14, 0x8)
        /usr/lib/golang/src/math/big/int.go:741 +0x58
crypto/rsa.(*PrivateKey).Validate(0x96fc1980, 0x0, 0x0)
        /usr/lib/golang/src/crypto/rsa/rsa.go:110 +0xe8
crypto/x509.ParsePKCS1PrivateKey(0x96e0ea00, 0x4a7, 0x4a7, 0x96fc1980, 0x0, 0x0)
        /usr/lib/golang/src/crypto/x509/pkcs1.go:77 +0x5c4
crypto/tls.parsePrivateKey(0x96e0ea00, 0x4a7, 0x4a7, 0x0, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/crypto/tls/tls.go:259 +0x4c
crypto/tls.X509KeyPair(0x96e497ae, 0x0, 0x200, 0x96ea5c8b, 0x0, 0x200, 0x970c8240, 0x1, 0x1, 0x0, ...)
        /usr/lib/golang/src/crypto/tls/tls.go:213 +0x2ac
crypto/tls.LoadX509KeyPair(0x97013500, 0x33, 0x97013580, 0x33, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /usr/lib/golang/src/crypto/tls/tls.go:179 +0x100
github.com/coreos/go-etcd/etcd.(*Client).initHTTPSClient(0x96f893b0, 0x97013500, 0x33, 0x97013580, 0x33, 0x0, 0x0)
        /root/origin/Godeps/_workspace/src/github.com/coreos/go-etcd/etcd/client.go:188 +0x68
github.com/coreos/go-etcd/etcd.NewTLSClient(0x96efa3b8, 0x1, 0x1, 0x97013500, 0x33, 0x97013580, 0x33, 0x9705d0b0, 0x28, 0x9703f0e0, ...)
        /root/origin/Godeps/_workspace/src/github.com/coreos/go-etcd/etcd/client.go:104 +0x2c4
github.com/openshift/origin/pkg/cmd/server/etcd.GetAndTestEtcdClient(0x96efa3b8, 0x1, 0x1, 0x9705d0b0, 0x28, 0x97013500, 0x33, 0x97013580, 0x33, 0x100dfd8, ...
)
        /root/origin/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/etcd/etcd.go:83 +0xa4
github.com/openshift/origin/pkg/cmd/server/etcd.NewOpenShiftEtcdHelper(0x96efa3b8, 0x1, 0x1, 0x9705d0b0, 0x28, 0x97013500, 0x33, 0x97013580, 0x33, 0x0, ...)
        /root/origin/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/etcd/etcd.go:117 +0x58
github.com/openshift/origin/pkg/cmd/server/origin.BuildMasterConfig(0x0, 0x0, 0x0, 0x0, 0xf20a80, 0xc, 0x9705ce40, 0x2e, 0x9705cea0, 0x2e, ...)
        /root/origin/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/origin/master_config.go:95 +0x4c
github.com/openshift/origin/pkg/cmd/server/start.StartMaster(0x96eae270, 0x0, 0x0)
        /root/origin/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/start/start_master.go:300 +0x278
github.com/openshift/origin/pkg/cmd/server/start.MasterOptions.RunMaster(0x96e78b00, 0x0, 0x0, 0x0, 0x0, 0x0)
        /root/origin/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/start/start_master.go:237 +0x468
github.com/openshift/origin/pkg/cmd/server/start.AllInOneOptions.StartAllInOne(0x96e78b00, 0x96f8b900, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
        /root/origin/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/start/start_allinone.go:237 +0xf8
github.com/openshift/origin/pkg/cmd/server/start.func·001(0x96dfd4a0, 0x96e18dd8, 0x0, 0x1)
        /root/origin/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/start/start_allinone.go:84 +0x284
github.com/spf13/cobra.(*Command).execute(0x96dfd4a0, 0x96efbff0, 0x1, 0x1, 0x0, 0x0)
        /root/origin/Godeps/_workspace/src/github.com/spf13/cobra/command.go:438 +0x368
github.com/spf13/cobra.(*Command).Execute(0x96dfd400, 0x0, 0x0)
        /root/origin/Godeps/_workspace/src/github.com/spf13/cobra/command.go:503 +0x46c
main.main()
        /root/origin/_output/local/go/src/github.com/openshift/origin/cmd/openshift/openshift.go:17 +0x194

goroutine 5 [syscall, 4 minutes]:
os/signal.loop()
        /usr/lib/golang/src/os/signal/signal_unix.go:21 +0x1c
created by os/signal.init·1
        /usr/lib/golang/src/os/signal/signal_unix.go:27 +0x40

goroutine 6 [chan receive]:
github.com/golang/glog.(*loggingT).flushDaemon(0x1c09c98)
        /root/origin/Godeps/_workspace/src/github.com/golang/glog/glog.go:879 +0x78
created by github.com/golang/glog.init·1
        /root/origin/Godeps/_workspace/src/github.com/golang/glog/glog.go:410 +0x330

goroutine 17 [syscall, 4 minutes, locked to thread]:
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_arm.s:1322 +0x4

goroutine 31 [chan receive]:
github.com/coreos/etcd/pkg/osutil.func·001()
        /root/origin/Godeps/_workspace/src/github.com/coreos/etcd/pkg/osutil/interrupt_unix.go:53 +0x50
created by github.com/coreos/etcd/pkg/osutil.HandleInterrupts
        /root/origin/Godeps/_workspace/src/github.com/coreos/etcd/pkg/osutil/interrupt_unix.go:74 +0x1d4

goroutine 26 [chan receive, 4 minutes]:
github.com/GoogleCloudPlatform/kubernetes/pkg/watch.(*Broadcaster).loop(0x96ebb2e0)
        /root/origin/Godeps/_workspace/src/github.com/GoogleCloudPlatform/kubernetes/pkg/watch/mux.go:133 +0x44
created by github.com/GoogleCloudPlatform/kubernetes/pkg/watch.NewBroadcaster
        /root/origin/Godeps/_workspace/src/github.com/GoogleCloudPlatform/kubernetes/pkg/watch/mux.go:70 +0x11c

goroutine 33 [runnable]:
github.com/coreos/etcd/raft.(*node).step(0x96fc1260, 0x86cdfaf0, 0x96e93d80, 0x2, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /root/origin/Godeps/_workspace/src/github.com/coreos/etcd/raft/node.go:383 +0x194
github.com/coreos/etcd/raft.(*node).Propose(0x96fc1260, 0x86cdfaf0, 0x96e93d80, 0x96fc4960, 0x94, 0x94, 0x0, 0x0)
        /root/origin/Godeps/_workspace/src/github.com/coreos/etcd/raft/node.go:355 +0xe8
github.com/coreos/etcd/etcdserver.(*EtcdServer).Do(0x9703f0e0, 0x86cdfaf0, 0x96e93d80, 0x51ac0001, 0xc64c9e3e, 0xed8348, 0x3, 0x96fc1c50, 0x26, 0x96f89400, ...
)
        /root/origin/Godeps/_workspace/src/github.com/coreos/etcd/etcdserver/server.go:495 +0x26c
github.com/coreos/etcd/etcdserver.(*EtcdServer).publish(0x9703f0e0, 0x2a05f200, 0x1)
        /root/origin/Godeps/_workspace/src/github.com/coreos/etcd/etcdserver/server.go:658 +0x294
created by github.com/coreos/etcd/etcdserver.(*EtcdServer).Start
        /root/origin/Godeps/_workspace/src/github.com/coreos/etcd/etcdserver/server.go:279 +0x58

goroutine 34 [runnable]:
github.com/coreos/etcd/etcdserver.(*EtcdServer).purgeFile(0x9703f0e0)
        /root/origin/Godeps/_workspace/src/github.com/coreos/etcd/etcdserver/server.go:309 +0x38c
created by github.com/coreos/etcd/etcdserver.(*EtcdServer).Start
        /root/origin/Godeps/_workspace/src/github.com/coreos/etcd/etcdserver/server.go:280 +0x80

goroutine 35 [IO wait]:
net.(*pollDesc).Wait(0x96e92a38, 0x72, 0x0, 0x0)
        /usr/lib/golang/src/net/fd_poll_runtime.go:84 +0x3c
net.(*pollDesc).WaitRead(0x96e92a38, 0x0, 0x0)
        /usr/lib/golang/src/net/fd_poll_runtime.go:89 +0x38
net.(*netFD).accept(0x96e92a00, 0x0, 0x86cce0c8, 0x970c8488)
        /usr/lib/golang/src/net/fd_unix.go:419 +0x390
net.(*TCPListener).AcceptTCP(0x96efafa0, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/net/tcpsock_posix.go:234 +0x50
net.(*TCPListener).Accept(0x96efafa0, 0x0, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/net/tcpsock_posix.go:244 +0x3c
crypto/tls.(*listener).Accept(0x97019c30, 0x0, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/crypto/tls/tls.go:46 +0x6c
github.com/coreos/etcd/pkg/transport.(*rwTimeoutListener).Accept(0x96fca200, 0x0, 0x0, 0x0, 0x0)
        /root/origin/Godeps/_workspace/src/github.com/coreos/etcd/pkg/transport/timeout_listener.go:44 +0x64
net/http.(*Server).Serve(0x96e93f00, 0x86cdf068, 0x96fca200, 0x0, 0x0)
        /usr/lib/golang/src/net/http/server.go:1728 +0x98
github.com/openshift/origin/pkg/cmd/server/etcd.serveHTTP(0x86cdf068, 0x96fca200, 0x86cdf9c8, 0x96fcaf60, 0xd964b800, 0x45, 0x0, 0x0)
        /root/origin/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/etcd/server.go:202 +0x1b8
github.com/openshift/origin/pkg/cmd/server/etcd.func·004(0x86cdf068, 0x96fca200)
        /root/origin/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/etcd/server.go:137 +0x58
created by github.com/openshift/origin/pkg/cmd/server/etcd.startEtcd
        /root/origin/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/etcd/server.go:68 +0x14e4

goroutine 36 [IO wait]:
net.(*pollDesc).Wait(0x96e92d38, 0x72, 0x0, 0x0)
        /usr/lib/golang/src/net/fd_poll_runtime.go:84 +0x3c
net.(*pollDesc).WaitRead(0x96e92d38, 0x0, 0x0)
        /usr/lib/golang/src/net/fd_poll_runtime.go:89 +0x38
net.(*netFD).accept(0x96e92d00, 0x0, 0x86cce0c8, 0x970c84a0)
        /usr/lib/golang/src/net/fd_unix.go:419 +0x390
net.(*TCPListener).AcceptTCP(0x96efb0b0, 0x15b78, 0x0, 0x0)
        /usr/lib/golang/src/net/tcpsock_posix.go:234 +0x50
net.(*TCPListener).Accept(0x96efb0b0, 0x0, 0x0, 0x0, 0x0)
        /usr/lib/golang/src/net/tcpsock_posix.go:244 +0x3c
github.com/coreos/etcd/pkg/transport.(*tlsKeepaliveListener).Accept(0x97019d50, 0x0, 0x0, 0x0, 0x0)
        /root/origin/Godeps/_workspace/src/github.com/coreos/etcd/pkg/transport/keepalive_listener.go:70 +0x6c
net/http.(*Server).Serve(0x96e93fc0, 0x86cdf538, 0x97019d50, 0x0, 0x0)
        /usr/lib/golang/src/net/http/server.go:1728 +0x98
github.com/openshift/origin/pkg/cmd/server/etcd.serveHTTP(0x86cdf538, 0x97019d50, 0x86cdf9c8, 0x96fcaec0, 0x0, 0x0, 0x0, 0x0)
        /root/origin/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/etcd/server.go:202 +0x1b8
github.com/openshift/origin/pkg/cmd/server/etcd.func·005(0x86cdf538, 0x97019d50)
        /root/origin/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/etcd/server.go:145 +0x58
created by github.com/openshift/origin/pkg/cmd/server/etcd.startEtcd
        /root/origin/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/etcd/server.go:68 +0x15ec

goroutine 37 [runnable]:
github.com/openshift/origin/pkg/cmd/server/etcd.func·001()
        /root/origin/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/etcd/etcd.go:67 +0x10c
created by github.com/openshift/origin/pkg/cmd/server/etcd.RunEtcd
        /root/origin/_output/local/go/src/github.com/openshift/origin/pkg/cmd/server/etcd/etcd.go:68 +0xd08

goroutine 38 [runnable]:
github.com/coreos/etcd/pkg/fileutil.func·001()
        /root/origin/Godeps/_workspace/src/github.com/coreos/etcd/pkg/fileutil/purge.go:69 +0x5d0
created by github.com/coreos/etcd/pkg/fileutil.PurgeFile
        /root/origin/Godeps/_workspace/src/github.com/coreos/etcd/pkg/fileutil/purge.go:75 +0x26c

goroutine 39 [runnable]:
github.com/coreos/etcd/pkg/fileutil.func·001()
        /root/origin/Godeps/_workspace/src/github.com/coreos/etcd/pkg/fileutil/purge.go:69 +0x5d0
created by github.com/coreos/etcd/pkg/fileutil.PurgeFile
        /root/origin/Godeps/_workspace/src/github.com/coreos/etcd/pkg/fileutil/purge.go:75 +0x26c

Operating system is Fedora 21.

cc @vpavlin

[0xmichalis]

Can you run it again? Looks like this issue is already known upstream: etcd-io/etcd#2308

[mmilata]

Thanks for your response @Kargakis. When I run it for the second time it fails with

I0409 10:38:50.607663    2453 start_master.go:287] Starting an OpenShift master, reachable at 0.0.0.0:8443 (etcd: [https://10.12.0.86:4001])
I0409 10:38:50.607786    2453 start_master.go:288] OpenShift master public address is https://10.12.0.86:8443
I0409 10:38:53.157441    2453 server.go:65] etcd: peerTLS: cert = openshift.local.certificates/etcd/server.crt, key = openshift.local.certificates/etcd/server.key, ca = openshift.local.certificates/ca/cert.crt
I0409 10:38:55.708803    2453 server.go:76] etcd: listening for peers on https://0.0.0.0:7001
I0409 10:38:55.708930    2453 server.go:68] etcd: clientTLS: cert = openshift.local.certificates/etcd/server.crt, key = openshift.local.certificates/etcd/server.key, ca = openshift.local.certificates/ca/cert.crt
I0409 10:38:58.258395    2453 server.go:68] etcd: listening for client requests on https://0.0.0.0:4001
2015/04/09 10:38:58 etcdserver: datadir is valid for the 2.0.1 format
2015/04/09 10:38:58 etcdserver: name = openshift.local
2015/04/09 10:38:58 etcdserver: data dir = openshift.local.etcd
2015/04/09 10:38:58 etcdserver: member dir = openshift.local.etcd/member
2015/04/09 10:38:58 etcdserver: heartbeat = 100ms
2015/04/09 10:38:58 etcdserver: election = 1000ms
2015/04/09 10:38:58 etcdserver: snapshot count = 0
2015/04/09 10:38:58 etcdserver: advertise client URLs = https://10.12.0.86:4001
2015/04/09 10:38:58 etcdserver: restart member eb0a4ac20fc1f6c6 in cluster 184b88513aa166b2 at commit index 0
2015/04/09 10:38:58 raft: eb0a4ac20fc1f6c6 became follower at term 0
2015/04/09 10:38:58 raft: newRaft eb0a4ac20fc1f6c6 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 1, lastterm: 1]
2015/04/09 10:38:58 etcdserver: set snapshot count to default 10000
I0409 10:38:58.282467    2453 etcd.go:66] Started etcd at 10.12.0.86:4001
2015/04/09 10:39:03 etcdserver: publish error: etcdserver: request timed out
2015/04/09 10:39:08 etcdserver: publish error: etcdserver: request timed out
2015/04/09 10:39:13 etcdserver: publish error: etcdserver: request timed out
2015/04/09 10:39:18 etcdserver: publish error: etcdserver: request timed out
2015/04/09 10:39:23 etcdserver: publish error: etcdserver: request timed out
2015/04/09 10:39:28 etcdserver: publish error: etcdserver: request timed out
2015/04/09 10:39:33 etcdserver: publish error: etcdserver: request timed out
2015/04/09 10:39:38 etcdserver: publish error: etcdserver: request timed out
2015/04/09 10:39:43 etcdserver: publish error: etcdserver: request timed out
F0409 10:39:43.966278    2453 start_allinone.go:94] Error setting up server storage: Could not reach etcd: 501: All the given peers are not reachable (Tried to connect to each peer twice and failed) [0]

I checked and DNS seems to be configured correctly so that shouldn't be an issue. No iptables rules and selinux in permissive mode.

[0xmichalis]

@smarterclayton this and other issues should be fixed upstream. If we want to keep them open, we should probably add an upstream label?

[danmcp]

Is this still an issue?

[0xmichalis]

Is this still an issue?

Most probably yes. @mmilata can you confirm? I still need you to test my PR regarding it: #1938

ps. @danmcp this is an etcd issue not a Kubernetes one

[danmcp]

Closing without prejudice due to age.