tracing: Zipkin support

[mattklein123]

Per an offline thread, we will do direct Zipkin support in Envoy and skip attempting to integrate with OT C++ library for now.

@RomanDzhabarov please sync with @adriancole and @rshriram to get a plan in place.

[rshriram]

cc @fabolive

[RomanDzhabarov]

https://docs.google.com/document/d/1q3F_RVf6Rozn69KbqhrKknOZP9_17WH1RK3wIc7AnuE/edit?usp=sharing

Thanks @adriancole for feedback!
Sharing with more people, this should unblock you from further work cc: @fabolive @rshriram

[codefromthecrypt]

welcome to the doc :)

[rshriram]

Thanks @adriancole . Just one question. Is it possible to push traces to remote Zipkin collectors via gRPC instead of JSON over HTTP? I am not sure if there is a gRPC api frontend to zipkin. For phase 1, JSON over http should suffice but in the long run, a gRPC interface might be helpful.

I do see that there is a Kafka/scribe adapter. Are these typically meant to run on the same host as Envoy, such that these adapters dump the traces into Kafka and the consumers pick it up on the other end? Or is it the case that Envoy would send these traces to a remote collector server with kafka backend [i.e. adding an extra hop before reaching elasticsearch/cassandra].

[codefromthecrypt]

Thanks @adriancole <https://github.com/adriancole> . Just one question. Is it possible to push traces to remote Zipkin collectors via gRPC instead of JSON over HTTP? I am not sure if there is a gRPC api frontend to zipkin. For phase 1, JSON over http should suffice but in the long run, a gRPC interface might be helpful.

We don't often implement features no-one has asked for, and no-one has asked for GRPC, yet. Care to raise an issue? There might be others interested, too. https://github.com/openzipkin/zipkin/issues/new

I do see that there is a Kafka/scribe adapter. Are these typically meant to run on the same host as Envoy, such that these adapters dump the traces into Kafka and the consumers pick it up on the other end? Or is it the case that Envoy would send these traces to a remote collector server with kafka backend [i.e. adding an extra hop before reaching elasticsearch/cassandra].

So in envoy, you'd have something similar to zipkin-reporter <https://github.com/openzipkin/zipkin-reporter-java> which sends the spans out-of-process after perhaps batching them. Here's the relationship: http -> POST list of spans in thrift or json (to a zipkin server, or better some load-balanced cluster of zipkin servers) kafka -> send message to topic "zipkin" containing a list of spans in thrift or json scribe <https://github.com/openzipkin/brave/blob/master/brave-spancollector-scribe/src/main/resources/scribe.thrift> -> log(logentry[].category="zipkin" logentry[].message=base64(thriftEncode(span)) scribe usually is store-forward from localhost, but sometimes people point scribe directly at a zipkin server listening on port 9410 * scribe is essentially effectively, but since it was the first transport, it still exists. long story short is that if envoy wants to use kafka, it would need a kafka producer library to send the span messages onto a topic. On the other side, a zipkin collector would pull those messages.

[RomanDzhabarov]

I think it would be a good start using JSON over HTTP communication.
It should not have any perf implications as we'll buffer spans and flush every 5 or so seconds.

Historically we'd been doing this with LS (before switched to gRPC) and processing 100k+ spans per second (100% sample), with zipkin lower sampling rate it should have very minimal perf concerns.