We had an ingester running on a node which spontaneously restarted.
Kubernetes started the ingester process, but it refused to go Ready.
We have excellent debugging for this condition now:
$ curl http://10.244.202.176/ready
Not ready: this ingester owns no tokens
At the top of the ingester logs on restart was:
level=info ts=2019-04-01T10:26:31.676674615Z caller=lifecycler.go:383 msg="existing entry found in ring" state=ACTIVE tokens=0
I believe this is because that code doesn't handle the normalise-tokens case.