From 166427cbe342b5ab897a299dc2012e62649ecf97 Mon Sep 17 00:00:00 2001 From: Chris Marchbanks Date: Tue, 4 Jun 2019 21:20:56 -0600 Subject: [PATCH] Add documentation for how to configure Jaeger Signed-off-by: Chris Marchbanks --- docs/tracing.md | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100644 docs/tracing.md diff --git a/docs/tracing.md b/docs/tracing.md new file mode 100644 index 00000000000..297d07fe683 --- /dev/null +++ b/docs/tracing.md @@ -0,0 +1,35 @@ +# Tracing + +Cortex uses [Jaeger](https://www.jaegertracing.io/) to implement distributed +tracing. We have found Jaeger invaluable for troubleshooting the behavior of +Cortex in production. + +## Dependencies + +In order to send traces you will need to set up a Jaeger deployment. A +deployment includes either the jaeger all-in-one binary, or else a distributed +system of agents, collectors, and queriers. If running on Kubernetes, [Jaeger +Kubernetes](https://github.com/jaegertracing/jaeger-kubernetes) is an excellent +resource. + +## Configuration + +In order to configure Cortex to send traces you must do two things: +1. Set the `JAEGER_AGENT_HOST` environment variable in all components to point + to your Jaeger agent. This defaults to `localhost`. +1. Enable sampling in the appropriate components: + * The Ingester and Ruler self-initiate traces and should have sampling + explicitly enabled. + * Sampling for the Distributor and Query Frontend can be enabled in Cortex + or in an upstream service such as your frontdoor. + +To enable sampling in Cortex components you can specify either +`JAEGER_SAMPLER_MANAGER_HOST_PORT` for remote sampling, or +`JAEGER_SAMPLER_TYPE` and `JAEGER_SAMPLER_PARAM` to manually set sampling +configuration. See the [Jaeger Client Go +documentation](https://github.com/jaegertracing/jaeger-client-go#environment-variables) +for the full list of environment variables you can configure. + +Note that you must specify one of `JAEGER_AGENT_HOST` or +`JAEGER_SAMPLER_MANAGER_HOST_PORT` in each component for Jaeger to be enabled, +even if you plan to use the default values.