Jaeger trace sampling should not be decided by every service (by default)

[leth]

If I understand correctly, the decision about whether to capture a trace should be made by one service, as close to the request source as possible.
Then that decision should be passed along with the request and honored by each service involved in the request processing.

Currently, if a service is configured to send samples to jaeger, it also defaults to a sampling rate of 10 per second. If both the auth layer, distributor and ingester were configured thusly, you would see 10-20 traces per second from the distributor, and 20-30 traces per second from the ingester. Additionally you'd see some traces which start at the distributor or the ingester.

For cortex I think whether to sample should be decided by the (exersise-for-the-reader) auth proxy, and the ruler & ingester - since the ruler & ingester originate work based on timers.

[csmarchbanks]

When reading the Jaeger docs, it looks like sampling only happens if there is not already sampling information on a parent span: https://www.jaegertracing.io/docs/sampling/

Therefore, if the auth proxy has sampling, then additional sampling won't happen in cortex. But if someone does not have sampling configured in the auth proxy then the sampling from the distributor/querier/ruler will be used instead. The ingester does not need a sampling, but it shouldn't hurt anything to have it.

Do you specifically see traces starting in the ingester? Just glancing at my logs I don't see any, but I didn't do a thorough search.

[leth]

Sorry my explanation was a bit poor, it's a bit better on the ww/common issue.

When reading the Jaeger docs, it looks like sampling only happens if there is not already sampling information on a parent span: https://www.jaegertracing.io/docs/sampling/

Therefore, if the auth proxy has sampling, then additional sampling won't happen in cortex.

Not quite, it makes the sampling decision on a per-request basis. So it'll still consider sampling any requests that the auth proxy did not choose to sample.

That's the source of the n, 2n, 3n sampling rate increase. If the parent span was not traced by the parent service, it might choose to sample it too.

Do you specifically see traces starting in the ingester? Just glancing at my logs I don't see any, but I didn't do a thorough search.

I picked the ingester because it was the more extreme example, we're not tracing it right now. We do see traces starting in the querier with the current behavior.

[leth]

I guess the thing that is easy to miss is that if the auth layer decides not to sample a request, it does not forbid other services from sampling that request, because there are situations where you might still want them to choose to sample it.

[csmarchbanks]

Huh interesting. I wonder if this is a bug in Jaeger? Or if somewhere the trace flags are not being passed correctly. In the go client it specifically states "The sampling decision is propagated to all downstream calls via the flags field of the trace context." So I would expect that if the auth layer decides to not sample a request, the "not sampling" state would be passed down to all the other services.

We do have tracing for the ingester, and the only spans starting in the ingester I am seeing are for /metrics and /ready http calls, and a CAS loop.

[leth]

The fix for this was incorporated by #870