Log trace IDs, correctly propagate trace IDs from frontend to querier

[tomwilkie]

(WIP, don't merge as it modifies vendored libraries - see weaveworks/common#123) merged and vendored properly.

[tomwilkie]

2018/08/30 17:47:19 http: panic serving 10.52.7.38:43394: runtime error: invalid memory address or nil pointer dereference
goroutine 78445 [running]:
net/http.(*conn).serve.func1(0xc420be4140)
	/usr/local/go/src/net/http/server.go:1726 +0xd0
panic(0xa35d80, 0x1070fd0)
	/usr/local/go/src/runtime/panic.go:502 +0x229
github.com/weaveworks/cortex/vendor/github.com/opentracing-contrib/go-stdlib/nethttp.(*Transport).RoundTrip(0xc4201dcfc0, 0xc420895000, 0xc42047e740, 0x0, 0x0)
	/go/src/github.com/weaveworks/cortex/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/client.go:128 +0x26b
github.com/weaveworks/cortex/pkg/querier/frontend.(*Frontend).ServeHTTP(0xc4201dcfa0, 0xb6c760, 0xc420525620, 0xc420895000)
	/go/src/github.com/weaveworks/cortex/pkg/querier/frontend/frontend.go:106 +0xc4
github.com/weaveworks/cortex/vendor/github.com/weaveworks/common/middleware.glob..func1.1(0xb6c760, 0xc420525620, 0xc420894e00)
	/go/src/github.com/weaveworks/cortex/vendor/github.com/weaveworks/common/middleware/http_auth.go:17 +0x107
net/http.HandlerFunc.ServeHTTP(0xc4202b0860, 0xb6c760, 0xc420525620, 0xc420894e00)
	/usr/local/go/src/net/http/server.go:1947 +0x44
github.com/weaveworks/cortex/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc4201dcd20, 0xb6c760, 0xc420525620, 0xc420894e00)
	/go/src/github.com/weaveworks/cortex/vendor/github.com/gorilla/mux/mux.go:114 +0xdc
github.com/weaveworks/cortex/vendor/github.com/weaveworks/common/middleware.Instrument.Wrap.func1(0xb6c720, 0xc42047e700, 0xc420894c00)
	/go/src/github.com/weaveworks/cortex/vendor/github.com/weaveworks/common/middleware/instrument.go:49 +0x175
net/http.HandlerFunc.ServeHTTP(0xc4202de420, 0xb6c720, 0xc42047e700, 0xc420894c00)
	/usr/local/go/src/net/http/server.go:1947 +0x44
github.com/weaveworks/cortex/vendor/github.com/weaveworks/common/middleware.Log.Wrap.func1(0xb6bfe0, 0xc420525600, 0xc420894c00)
	/go/src/github.com/weaveworks/cortex/vendor/github.com/weaveworks/common/middleware/logging.go:41 +0x1af
net/http.HandlerFunc.ServeHTTP(0xc4202de450, 0xb6bfe0, 0xc420525600, 0xc420894c00)
	/usr/local/go/src/net/http/server.go:1947 +0x44
github.com/weaveworks/cortex/vendor/github.com/opentracing-contrib/go-stdlib/nethttp.Middleware.func2(0xb6cee0, 0xc42022a380, 0xc420894b00)
	/go/src/github.com/weaveworks/cortex/vendor/github.com/opentracing-contrib/go-stdlib/nethttp/server.go:74 +0x3f3
net/http.HandlerFunc.ServeHTTP(0xc4202de480, 0xb6cee0, 0xc42022a380, 0xc420894b00)
	/usr/local/go/src/net/http/server.go:1947 +0x44
github.com/weaveworks/cortex/vendor/github.com/weaveworks/common/middleware.Tracer.Wrap.func1(0xb6cee0, 0xc42022a380, 0xc420894b00)
	/go/src/github.com/weaveworks/cortex/vendor/github.com/weaveworks/common/middleware/http_tracing.go:28 +0x8e
net/http.HandlerFunc.ServeHTTP(0xc4202de4b0, 0xb6cee0, 0xc42022a380, 0xc420894b00)
	/usr/local/go/src/net/http/server.go:1947 +0x44
net/http.serverHandler.ServeHTTP(0xc4201c1110, 0xb6cee0, 0xc42022a380, 0xc420894b00)
	/usr/local/go/src/net/http/server.go:2694 +0xbc
net/http.(*conn).serve(0xc420be4140, 0xb6d660, 0xc4206dc780)
	/usr/local/go/src/net/http/server.go:1830 +0x651
created by net/http.(*Server).Serve
	/usr/local/go/src/net/http/server.go:2795 +0x27b

[tomwilkie]

That should fix the panic. Yup, can't reproduce it.

[khaines]

LGTM. This will be handy, thank you Tom

[gouthamve]

👍 from me too :)

This works amazing. We set the sampling for our queries to 1, which means, we can take a slow query, take the trace_id, stick it in Jaeger for fun and profit :)