Support new OpenTracing API

[tedsuo]

A new version of the OpenTracing API for C++11 is almost ready (opentracing/opentracing-cpp#11). Once it has been merged, we will be changing the LightStep tracer so that it is OT-compatible. It would be great if we could then do the following with Envoy:

• extract the ZipKin tracer and use it as the basis for a new OT-compatible ZipKin C++ tracer
• reimplement the tracing instrumentation in Envoy to use the OpenTracing API.
• support linking to 3rd-party OT tracers beyond lightstep and zipkin.

We're happy to put the work in to make this happen. Any recommendations or objections?

[mattklein123]

This sounds good to me.

cc @adriancole @rshriram @RomanDzhabarov @fabolive for further comment.

[moderation]

I was doing some very initial testing with @RomanDzhabarov to upgrade to lightstep-tracer-cpp 0.38 (currently on 0.36). We should probably put this on hold and wait for the implementation suggested by @tedsuo?

[mattklein123]

@moderation yes I would probably just skip the upgrade at this point.

[codefromthecrypt]

There are usually mapping problems with OpenTracing efforts, surfacing as poorly encoded data in Zipkin or spans not appearing as client/server etc. These add support burden to our gitter channel and issues list. These mapping problems happen because usually OT is tested as a mock, as opposed to a micro-integration test (ex to the degree tests exist, they don't relate directly to data emitted). In other words, adding any layer of indirection is likely to undo the clean integration currently in envoy. Solving problems will become harder and not possible directly in envoy code.

TL;DR; I would highly discourage ripping out the existing Zipkin support until there are tests in permanently to show that envoy works with Zipkin via OT, not just that OT variants work.

[mattklein123]

TL;DR; I would highly discourage ripping out the existing Zipkin support until there are tests in permanently to show that envoy works with Zipkin via OT, not just that OT variants work.

Agreed. Nothing will change until we have confidence that any new solution is supported by the community and works. I'm mainly saying that I have no objection to people attempting to tackle this if they are willing to do all the work.

[tedsuo]

Yes, @adriancole this is also part of an effort to validate that the new API is correct, we want to prove that we can bind a ZipKin C++ implementation to it and have it work in real systems like Envoy and NGINX. (I'd also like to talk to you about test harnesses and other compatibility guarantees for OT, but probably a different thread). We're also fine supporting the C++ integration w/ZipKin and ensuring it is stable until the API has reached maturity.

We don't plan on making a PR until we're confident that there is solid integration that has parity with the existing instrumentation.

[codefromthecrypt]

Thanks for the offer @tedsuo, and glad you are on board with making the first version of this portable with zipkin. Having an organization of mostly volunteers, I'm more concerned about future versions. Don't read this as skepticism about you personally, as your statement sounds great.

What I'm generally concerned with is a scenario where OT is inserted then Zipkin later removed or degraded due to api decisions not made in the interest of the OpenZipkin community of end users. We've had problems with this in the past where apis crucial to Zipkin were on the chopping board and would have been removed if @basvanbeek or I didn't notice.

It integration tests are here and someone in OT makes a decision not in favor of Zipkin users (regardless of intent), the build breaks. Fixing the build is either sacrificing these users or making the OT -> Zipkin story right again.

TL;DR; if the goal includes removing the code that just went in for Zipkin support, let's make sure integration tests happen here, and are left in for the duration Envoy wants to support the OpenZipkin community. I think this "tests here" part is the only thing different than your plan.

[tedsuo]

Sounds good @adriancole, we'll include Zipkin-based integration tests when we make our PR.

[rnburn]

Hello Everyone, I extracted out the Zipkin tracing code from envoy into a stand-alone library and added a bridge to the proposed C++ OpenTracing API. I also wrote a version of LightStep’s tracer that supports the new API.

I’m starting to look into how best to integrate envoy with the OpenTracing API. @mattklein123 One thing I was wondering — Would you be willing to use the recorders that come with the Zipkin and LightStep libraries instead of having custom ones built into envoy, provided that Zipkin and LightStep support a way to get access to reporting statistics so you can continue recording things like the dropped span counts?

Doing so could simplify envoy's tracing code and make it easier to support features like Kafka collection since you could just turn on the functionality provided by Zipkin's tracing library.

[mattklein123]

@rnburn my primary requirement for the tracing libraries are that:

• We don't lose any stats (as you point out).
• We decouple framing vs. transport. Meaning, the libraries should be written in a way that we can still use Envoy per thread networking and not have any connection pools/threading/etc. inside the libraries. If the libraries implement a default transport that's fine, but they should be written in a way that allows us to use our own transport.

As long as the previous two points are handled, the more we can push into the libraries the better IMO.

[codefromthecrypt]

One thing I mentioned on another discussion about this is that diversity of interest maintaining is important. For example, the current zipkin code (which this effort aims to remove) was reviewed by folks at Lyft and IBM. The one mentioned here is a replacement, and currently single-author (no offence). In zipkin itself, there's a community provided c/c++ library https://github.com/flier/zipkin-cpp, which has a bit more history (even if short) and importantly it is used in production. If we are ripping out months of reviewed code, seems it should be the case that the replacement should have more not less ecosystem behind it. Can we please check with the folks that wrote the existing one and get buy-in that zipkin or performance of this will not become degraded by accident of being pushed under an umbrella?