Better error message for Series Store cardinality limit

[jtlisi]

Currently the series store returns a error that is not very descriptive.

	errCardinalityExceeded = errors.New("cardinality limit exceeded")

This makes debugging a series store cardinality error difficult. It doesn't have good logs to show that it was the series store throwing this error. Also the cardinality limit for the series store is configured in a separate package from the other limits in the project.

[bboreham]

Good point. I wasn't familiar with this code, so will add:
This happens during queries, but only if every matcher exceeds the limit.
I.e. if I query some_metric{foo="1",bar="2"} and a million series match foo="1" but only one matches bar="2" I shouldn't get the error.
How would you improve it - put the number in the error message ?

I generally turn to Jaeger for debugging; traditional logs are just too hard to make sense of in a busy server.

[jtlisi]

@bboreham Currently the code only returns an error to the user if all of the matchers yield a cardinality greater than the limit.

	// Otherwise get series which include other matchers
	incomingIDs := make(chan []string)
	incomingErrors := make(chan error)
	for _, matcher := range matchers {
		go func(matcher *labels.Matcher) {
			ids, err := c.lookupSeriesByMetricNameMatcher(ctx, from, through, metricName, matcher)
			if err != nil {
				incomingErrors <- err
				return
			}
			incomingIDs <- ids
		}(matcher)
	}
        ...
	var cardinalityExceededErrors int
	for i := 0; i < len(matchers); i++ {
		select {
                ...
		case err := <-incomingErrors:
			if err == errCardinalityExceeded {
				cardinalityExceededErrors++
			} else {
				lastErr = err
			}
		}
	}
        ...
	if cardinalityExceededErrors == len(matchers) {
		return nil, errCardinalityExceeded
	} else if lastErr != nil {
		return nil, lastErr
	}

I think adding a log to the trace with the the cardinality of the matcher would help greatly

[tomwilkie]

I believe this was fixed in #1328