Skip to content

Ingester restart with -ingester.normalise-tokens on results in not-ready state #1311

@bboreham

Description

@bboreham

We had an ingester running on a node which spontaneously restarted.
Kubernetes started the ingester process, but it refused to go Ready.
We have excellent debugging for this condition now:

$ curl http://10.244.202.176/ready
Not ready: this ingester owns no tokens

At the top of the ingester logs on restart was:

level=info ts=2019-04-01T10:26:31.676674615Z caller=lifecycler.go:383 msg="existing entry found in ring" state=ACTIVE tokens=0

I believe this is because that code doesn't handle the normalise-tokens case.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions