Update dependency version in envoyproxy/envoy to 0.1.11

[Smeb]

Hi 👋 - hope this is the right place. I opened this issue here rather than in Envoy because it seems like @dgoffredo is a maintainer here, and is the one updating the version in Envoy. If this is not the right place, my apologies.

It seems like Envoy is still using the 0.1.8 version of this library here. This is true as of the latest release (yesterday, 1.19.4). It should probably be updated to 0.1.11. I'm not too sure how involved this is, or I would open a PR - plus I think it's probably more likely to be accepted by the maintainers here.

[dgoffredo]

Hi, Ben. A recent release of dd-trace-cpp contains changes that would break the Datadog-related unit tests within Envoy.

My teammate is working on a PR within Envoy that will make the tests compatible with more recent versions of dd-trace-cpp.

Then, we will update the version of dd-trace-cpp used within Envoy.

Is there a feature or fix in more recent versions of dd-trace-cpp that you're looking for in Envoy?

[Smeb]

Great to hear.

The issue we're running into is that with Envoy 1.27 the span names are different than with 1.26. In 1.26 the span names are envoy.proxy, while in 1.27 they show up as (what looks to be) the name of the service. This in turn leads some metrics to go missing (specifically trace.envoy.proxy.hits). At least - I think that's what's happening. I haven't got too much hands on experience with tracing / cpp though.

That said, when I dug a bit into the tracing code and old PRs, I saw that the changes here (specifically the change to setOperation) don't exist in release/v1.27 of envoy. I think the backport here was probably missing some things.

It's a little late for me, but would be happy to try updating the backport tomorrow if this sounds like it's the right direction.

[dgoffredo]

Yes, the "operation name is wrong in v1.27 and v1.28" was fixed and backported onto those release branches. You can pick up those changes whenever Envoy produces another release in either series. That will fix that problem.

The "setting the 'resource.name' tag doesn't work anymore" issue was fixed on the main branch, but the process of proposing backports has stalled. Perhaps I can propose the backport PRs myself.

[Smeb]

I haven't worked on Envoy much, but I found these docs if it's helpful. Specifically, they say that if you add backport/review as a label, that starts the process (apparently).

[Smeb]

Just to follow up, I opened backport PRs against 1.27, 1.28, hope that's ok:

• [bp/1.27] Datadog: restore "resource.name" tag (#30503) envoyproxy/envoy#30892
• [bp/1.28] Datadog: restore "resource.name" tag (#30503) envoyproxy/envoy#30893

Interesting experience getting everything running locally, but tests were passing locally for both.

Small update - 1.28 seems to break for the example. Will look into why that might be.

[njegosrailic]

Hello everyone,

I work for Wayfair, and we had a support case with DD number 1339031. I've been following multiple issues in the past two months, and it seems that the problem is finally resolved. If I understood correctly, we would still need new builds for Envoy 1.27.x and 1.28.x versions, and they will be included in the next Istio patch versions. Is that correct?

Are you planning to upgrade the official documentation once this is done? It appears that the latest supported version is v.1.17, and that was what the support was referring to when we were complaining about traces being broken in Istio v.1.19.

Could you also describe any changes related to the DD tracing library that we can expect in Istio versions newer than v1.17, such as the one related to the resource name not being taken from the tag resource.name? These changes may require some of additional adjustments on our side, and we need to communicate this to developers.
Including these details in the official documentation would help to clarify that it's related to DD Istio integration.

[dgoffredo]

Hi, Njegos. I don't know which case number 1339031 refers to. There is Zendesk, which I believe Datadog uses to communicate with customers about Support issues, and then we use another system internally when Support interacts with Engineering. What is the issue that you're referring to with number 1339031?

The table in the environment variables section of the documentation you linked is indeed not up to date. I wasn't aware that it exists.

The changes to Envoy that have caused these issues (the unexpected change in operation name, the inability to set the resource name via the "resource.name" tag, and an unintended sampling override) were made to move Envoy from an older OpenTracing-based Datadog tracing library to this newer tracing library (dd-trace-cpp). So, the version numbers tabulated will not be comparable for present and future versions of Istio that use the newer Envoys. On the other hand, dd-trace-cpp has major version zero (0.x.y), so there might not be any confusion.

I'll see to it that the documentation is updated. We have a work item to revisit our Istio documentation more extensively, as methods of deployment have changed since we initially developed the integration.

None of these problematic changes were intentional, and so I can't anticipate what others, if any, might be found. Here's a list of recent Envoy Datadog tracing issues (i.e. issues starting with Envoy v1.27) that I know about:

1. The default Datadog span name changed for spans created by Envoy: tracing: fix Datadog span name envoyproxy/envoy#29932
2. The ability to set the Datadog resource name by setting the "resource.name" tag stopped working: Datadog: restore "resource.name" tag envoyproxy/envoy#30503
3. When Envoy's sampling rate (not to be confused with Datadog's sampling rate) is less then 100%, then drop decisions made by Envoy override any decision extracted from incoming request headers (this can break traces): datadog: honor extracted sampling decisions envoyproxy/envoy#30577

Backports onto Envoy v1.27 and v1.28 are merged for all of these issues ((2) thanks to @Smeb 🎉)

The relationship between Istio and the Datadog tracing library is the following:

Istio has a version of Envoy that it calls istio-proxy. istio-proxy is based on a release of Envoy. The correspondence between Istio releases and the Envoy releases on which istio-proxy is based are tabulated here.

Envoy builds dd-trace-cpp from source and combines the library with other Datadog-specific code within Envoy (this is where I put the bugs).

The version of dd-trace-cpp on which Envoy depends is determined by envoy/bazel/repository_locations.bzl within the Envoy source tree.

In summary:

• A release of Istio tracks a particular release of Envoy.
• A release of Envoy contains both Datadog-specific code and a particular version of dd-trace-cpp, determined by the build.
• dd-trace-cpp contains the core Datadog tracing code, and is released as a source tarball on GitHub.

[njegosrailic]

Hi David,

Thank you for the detailed updates and explanations. To be honest, I don't personally use DataDog, but our developers heavily rely on it. In early October, we initiated Istio upgrades, but had to revert in most of the clusters due to the "issue" with tracing.

Starting from Istio v1.18, we observed significant visual differences in Datadog tracing. Some of the main issues include, but are not limited to:

• the function changing from envoy.proxy to <service-name>_<url for cluster>.
• the istio.operation tag seems to be dropped or turned off.
• runtime-id and process_id are missing from traces.

More details are available in the linked issue. When we approached support, we were informed that this is related to the DataDog tracing library that Envoy uses and that DataDog doesn't support Istio versions beyond v1.17, according to their documentation.

I plan to open a new support case with additional details that I cannot share here. I will mention this issue and then discuss it with our account manager.

Our primary concern is understanding when we can expect the same tracing functionality we had in previous versions. If not, could this be documented?

[dgoffredo]

@njegosrailic please share the support case number with me once it is created, so I can inquire about it internally.

Your first bullet point is the "operation name" issue discussed previously. The other two changes of behavior that you mention are news to me. istio.operation I haven't heard of before. The runtime-id and process_id tags are added by the underlying tracing library, so it is a surprise that they are missing.

[njegosrailic]

Hi @dgoffredo,

I have created a new support case https://help.datadoghq.com/hc/en-us/requests/1453323 and I'm providing the old one https://help.datadoghq.com/hc/en-us/requests/1339031 for reference.

Thank you for the help.

[dgoffredo]

This will be resolved when envoyproxy/envoy#31526 is merged.

[dgoffredo]

envoyproxy/envoy now uses dd-trace-cpp v0.1.12.