Ambient Multicluster Telemetry by keithmattix · Pull Request #6793 · istio/proxy

keithmattix · 2026-01-27T02:38:23Z

The bulk of the changes for ambient multicluster telemetry. Here's a high level summary (please read the design doc for more details).

Create a network filter for peer_metadata (alongside the existing http filter) that can propagate data from response headers (received by the connect_originate internal listener) back to the main_internal listener filter_state so it can be used for stats (see Introduce new filters discovering peer metadata from baggage header #6771 for more details)
Propagate baggage to downstream in response headers (Add Baggage metadata propagation #6776)
Use baggage in request headers for downstream peer discovery.

One thing I'll call out here is that this implementation of baggage is only meant to be used in the HBONE connect tunnel; we DO NOT modify user requests with this approach so it is incompatible with (non-HBONE) sidecar mode.

…ent-multicluster-telemetry branch (#6772) * Include myself, Steven and Gustavo as owners of the experimental-ambient-multicluster-telemetry branch Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> * Use single match - creating multiple matches means that the later overrides the earlier Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> --------- Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

* Add Baggage metadata propagation Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> * clang-tidy Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> * Go back to old baggage impl Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> * Fix baggage format Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> * Actually use new baggage approach Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> --------- Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

…6771) * Introduce new filters discovering peer metadata from baggage header This a combination of two filters that have to be used together: - regular network filter (expected to be configured in connect_originate or inner_connect_originate listeners before TCP Proxy filter) - upstream network filter (expected to be configuration in all clusters that use HBONE or double-HBONE for endpoints) Those two filters together basically create a tunnel. The tunnel protocol just prepends a fixed size header to data stream coming from regular network filter to the upstream network filter, followed by the peer metadatra encoded as protobuf Any containing a protobuf Struct inside (I'm just re-using existing code from Istio proxy, that's why encoding is such as it is). The regular network filter only triggers when there is some data coming from upstream connection in response. It's not correct in general, but in waypoints we do know that we proxy an L7 protocol (http or gRPC), so we do expect a some data in reply. The regular network filter relies on TCP Proxy filter extracting response headers and saving them in the filter state. It then extracts and parses the baggage header from the saved headers. In all cases I explicitly communicate when no peer metadata has been discovered by sending some data downstream. This ensures that upstream network filter running downstream can always remove the prefix from the data stream and does not really need to guess if it's there or not. NOTE: We still do some checks to confirm that the prefix is there, but we cannot really rely on those checks for correctness in all the cases. The upstream network filter, as pointed out above, extracts the data sent by the regular network filter from the data stream, it parses the data and populates filter state based on that. Unlike the HTTP peer metadata filter, this one runs in the context of the upstream connection, so it populates the upstream filter state and not the regular one. I plan to add support to the HTTP peer metadata filter option for new upstream metadata discovery via upstream filter metadata, thus propagating it all the way to the istio stats filter. NOTE: None of those filters are yet generated by pilot and there are certainly some additional options to configure (e.g., maybe we can come up with a good way to transfer metadata via Envoy TLS instead of injecting it into the data stream directly - this way, in principle, we could avoid creating a custom upstream filter all together, if http peer metadata filter could get the peer metadata directly from connect_originate listener). All-in-all, it's not the final implementation. Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> * Fix BUILD formatting Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> * Fix formatting of C++ code Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> * Update HTTP peer_metadata filter to consume filter state set by upstream peer_metadata filter This basically taps the upstream peer metadata into the regular filter state consumed by the istio stats filter. http peer metadata filter also takes care of priorities between different discovery methods - we just need to put different discovery methods in the right order in the configuration. Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> * Populate peer principal in the upstream workload metadata as well Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> * Support propagating baggage header to upstream and additional safety checks for upstream network filter Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> * Only register UpstreamFilterState peer metadata discovery method for upstream peer discovery Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> * Move peer_metadata filter proto config in the same directory Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> * Fix typo Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> --------- Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

* Add Baggage metadata propagation Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> * clang-tidy Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> * basics for baggage discovery downstream * removing unnecessary tests * reverting crazy claude changes in release-binary.sh * fixing tests, fixing baggage key tokens * removing comment * make lint * fixing unit tests for metadata_object * make lint * suggestions from PR * clarifying use of mappings for baggage and field access * make lint --------- Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> Co-authored-by: Keith Mattix II <keithmattix@microsoft.com>

* Add locality to proxy metadata Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> * Clang-tidy Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> * Buildifier format Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> * Rebase and fix some bugs Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> --------- Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

* Drop app labels from baggage and propagate principal I think I confused folks a bit when I mentioned that app field is missing from the baggage - it wasn't. In fact, canonical name of the workload and app in ambient are the same thing, that's why baggage does not actually need an app label - it already has service.name that encodes what we need. I updated the design document, but it happened after I mentioned here and there that we need to add a missing field to the baggage. This change corrects implementation and that makes istio stats populate the app label correctly. The other field that has not been populated is principal. WorkloadMetadataObject contained that identity field that contained principle in principle, but the methods used to conver WorkloadMetadataObject to a protobuf Struct and back ignored that field and never populated it, so it got lost and istio stats never used it. We haven't noticed that before because in ambient we used xDS-based peer metadata discovery by default and it triggers a different code path that does not rely on the methods that convert protobuf Struct to WorkloadMetadataObject, and the code path used there didn't have the same issue. Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> * Keep backwards compatibility for app.service and app.version baggage fields Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> --------- Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> Co-authored-by: Keith Mattix II <keithmattix@microsoft.com>

keithmattix · 2026-01-27T02:56:38Z

/retest

keithmattix · 2026-01-27T03:14:33Z

/retest

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

* Automator: update envoy@ in istio/proxy@master (#6777) * Automator: update envoy@ in istio/proxy@master (#6778) * Don't do workload discovery for cross-network traffic (#6767) * Get the implementation compiling * Add tests for cross-network peer metadata Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> * clang-tidy Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> * One more tidy Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> * Switch to debug for logging Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> --------- Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> * Automator: update envoy@ in istio/proxy@master (#6782) * Automator: update envoy@ in istio/proxy@master (#6784) * Automator: update go-control-plane in istio/proxy@master (#6786) * Automator: update envoy@ in istio/proxy@master (#6787) * Automator: update envoy@ in istio/proxy@master (#6788) * update x-network header key (#6790) Signed-off-by: Ian Rudie <ian.rudie@solo.io> * Automator: update envoy@ in istio/proxy@master (#6794) * Merge upstream/master and resolve merge conflicts Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> * Missed one Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> * Fixed a wrong one Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> --------- Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> Signed-off-by: Ian Rudie <ian.rudie@solo.io> Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> Co-authored-by: Istio Automation <istio-testing-bot@google.com> Co-authored-by: Keith Mattix II <keithmattix@microsoft.com> Co-authored-by: Ian Rudie <ilrudie@gmail.com>

keithmattix · 2026-01-27T14:02:21Z

/retest-required

keithmattix · 2026-01-27T14:08:13Z

/test release-test-arm64

keithmattix · 2026-01-27T14:51:44Z

Hitting capacity limits again, but these exp tests aren't required (it's just happening because HWAD is an experimental branch)

/retest

keithmattix · 2026-01-27T15:51:45Z

/test release_proxy_postsubmit_exp
/test release-arm64_proxy_postsubmit_exp

EDIT: this won't work because the exp tests are only because this was an experimental branch

istio-testing · 2026-01-27T15:51:49Z

@keithmattix: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test release-test

/test release-test-arm64

/test test

/test test-arm-arm64

/test test-asan

Use /test all to run all jobs.

Details

In response to this:

/test release_proxy_postsubmit_exp
/test release-arm64_proxy_postsubmit_exp

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

zirain · 2026-01-28T00:49:34Z

LGTM, one small comment.
IIUC, #6780 could be used for Sidecar aslo. should we split out?

keithmattix · 2026-01-28T01:20:04Z

@zirain I'll have to think through the upgrade logic for sidecar carefully; we base64 encode a struct and we'll want to make sure that adding 2 new fields won't break. I'll do some reasearch and add the sidecar implementation in a follow-up 👍🏾

istio-testing · 2026-01-28T02:34:19Z

In response to a cherrypick label: #6793 failed to apply on top of branch "release-1.29":

Applying: Include myself, Steven and Gustavo as owners of the experimental-ambient-multicluster-telemetry branch (#6772)
Using index info to reconstruct a base tree...
M	CODEOWNERS
Falling back to patching base and 3-way merge...
Auto-merging CODEOWNERS
CONFLICT (content): Merge conflict in CODEOWNERS
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
Patch failed at 0001 Include myself, Steven and Gustavo as owners of the experimental-ambient-multicluster-telemetry branch (#6772)

istio-testing · 2026-01-28T02:34:20Z

In response to a cherrypick label: new issue could not be created for failed cherrypick: status code 410 not one of [201], body: {"message":"Issues has been disabled in this repository.","documentation_url":"https://docs.github.com/rest/issues/issues#create-an-issue","status":"410"}

* Include myself, Steven and Gustavo as owners of the experimental-ambient-multicluster-telemetry branch (istio#6772) * Include myself, Steven and Gustavo as owners of the experimental-ambient-multicluster-telemetry branch Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> * Use single match - creating multiple matches means that the later overrides the earlier Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> --------- Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> * Add Baggage metadata propagation (istio#6776) * Add Baggage metadata propagation Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> * clang-tidy Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> * Go back to old baggage impl Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> * Fix baggage format Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> * Actually use new baggage approach Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> --------- Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> * Introduce new filters discovering peer metadata from baggage header (istio#6771) * Introduce new filters discovering peer metadata from baggage header This a combination of two filters that have to be used together: - regular network filter (expected to be configured in connect_originate or inner_connect_originate listeners before TCP Proxy filter) - upstream network filter (expected to be configuration in all clusters that use HBONE or double-HBONE for endpoints) Those two filters together basically create a tunnel. The tunnel protocol just prepends a fixed size header to data stream coming from regular network filter to the upstream network filter, followed by the peer metadatra encoded as protobuf Any containing a protobuf Struct inside (I'm just re-using existing code from Istio proxy, that's why encoding is such as it is). The regular network filter only triggers when there is some data coming from upstream connection in response. It's not correct in general, but in waypoints we do know that we proxy an L7 protocol (http or gRPC), so we do expect a some data in reply. The regular network filter relies on TCP Proxy filter extracting response headers and saving them in the filter state. It then extracts and parses the baggage header from the saved headers. In all cases I explicitly communicate when no peer metadata has been discovered by sending some data downstream. This ensures that upstream network filter running downstream can always remove the prefix from the data stream and does not really need to guess if it's there or not. NOTE: We still do some checks to confirm that the prefix is there, but we cannot really rely on those checks for correctness in all the cases. The upstream network filter, as pointed out above, extracts the data sent by the regular network filter from the data stream, it parses the data and populates filter state based on that. Unlike the HTTP peer metadata filter, this one runs in the context of the upstream connection, so it populates the upstream filter state and not the regular one. I plan to add support to the HTTP peer metadata filter option for new upstream metadata discovery via upstream filter metadata, thus propagating it all the way to the istio stats filter. NOTE: None of those filters are yet generated by pilot and there are certainly some additional options to configure (e.g., maybe we can come up with a good way to transfer metadata via Envoy TLS instead of injecting it into the data stream directly - this way, in principle, we could avoid creating a custom upstream filter all together, if http peer metadata filter could get the peer metadata directly from connect_originate listener). All-in-all, it's not the final implementation. Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> * Fix BUILD formatting Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> * Fix formatting of C++ code Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> * Update HTTP peer_metadata filter to consume filter state set by upstream peer_metadata filter This basically taps the upstream peer metadata into the regular filter state consumed by the istio stats filter. http peer metadata filter also takes care of priorities between different discovery methods - we just need to put different discovery methods in the right order in the configuration. Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> * Populate peer principal in the upstream workload metadata as well Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> * Support propagating baggage header to upstream and additional safety checks for upstream network filter Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> * Only register UpstreamFilterState peer metadata discovery method for upstream peer discovery Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> * Move peer_metadata filter proto config in the same directory Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> * Fix typo Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> --------- Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> * Baggage discovery (istio#6779) * Add Baggage metadata propagation Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> * clang-tidy Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> * basics for baggage discovery downstream * removing unnecessary tests * reverting crazy claude changes in release-binary.sh * fixing tests, fixing baggage key tokens * removing comment * make lint * fixing unit tests for metadata_object * make lint * suggestions from PR * clarifying use of mappings for baggage and field access * make lint --------- Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> Co-authored-by: Keith Mattix II <keithmattix@microsoft.com> * Add locality to proxy metadata (istio#6780) * Add locality to proxy metadata Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> * Clang-tidy Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> * Buildifier format Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> * Rebase and fix some bugs Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> --------- Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> * Drop app labels from baggage and propagate principal (istio#6791) * Drop app labels from baggage and propagate principal I think I confused folks a bit when I mentioned that app field is missing from the baggage - it wasn't. In fact, canonical name of the workload and app in ambient are the same thing, that's why baggage does not actually need an app label - it already has service.name that encodes what we need. I updated the design document, but it happened after I mentioned here and there that we need to add a missing field to the baggage. This change corrects implementation and that makes istio stats populate the app label correctly. The other field that has not been populated is principal. WorkloadMetadataObject contained that identity field that contained principle in principle, but the methods used to conver WorkloadMetadataObject to a protobuf Struct and back ignored that field and never populated it, so it got lost and istio stats never used it. We haven't noticed that before because in ambient we used xDS-based peer metadata discovery by default and it triggers a different code path that does not rely on the methods that convert protobuf Struct to WorkloadMetadataObject, and the code path used there didn't have the same issue. Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> * Keep backwards compatibility for app.service and app.version baggage fields Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> --------- Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> Co-authored-by: Keith Mattix II <keithmattix@microsoft.com> * Fix some test compilation errors Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> * Merge master branch and resolve merge conflicts properly (istio#6795) * Automator: update envoy@ in istio/proxy@master (istio#6777) * Automator: update envoy@ in istio/proxy@master (istio#6778) * Don't do workload discovery for cross-network traffic (istio#6767) * Get the implementation compiling * Add tests for cross-network peer metadata Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> * clang-tidy Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> * One more tidy Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> * Switch to debug for logging Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> --------- Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> * Automator: update envoy@ in istio/proxy@master (istio#6782) * Automator: update envoy@ in istio/proxy@master (istio#6784) * Automator: update go-control-plane in istio/proxy@master (istio#6786) * Automator: update envoy@ in istio/proxy@master (istio#6787) * Automator: update envoy@ in istio/proxy@master (istio#6788) * update x-network header key (istio#6790) Signed-off-by: Ian Rudie <ian.rudie@solo.io> * Automator: update envoy@ in istio/proxy@master (istio#6794) * Merge upstream/master and resolve merge conflicts Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> * Missed one Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> * Fixed a wrong one Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> --------- Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> Signed-off-by: Ian Rudie <ian.rudie@solo.io> Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> Co-authored-by: Istio Automation <istio-testing-bot@google.com> Co-authored-by: Keith Mattix II <keithmattix@microsoft.com> Co-authored-by: Ian Rudie <ilrudie@gmail.com> --------- Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> Signed-off-by: Ian Rudie <ian.rudie@solo.io> Co-authored-by: Krinkin, Mike <mkrinkin@microsoft.com> Co-authored-by: Gustavo Meira <grnmeira@users.noreply.github.com> Co-authored-by: Istio Automation <istio-testing-bot@google.com> Co-authored-by: Ian Rudie <ilrudie@gmail.com>

stevenctl · 2026-01-28T08:00:21Z

+      } else {
+        ENVOY_LOG(warn, "BaggageDiscovery peer metadata discovery option is only available for "
+                        "downstream peer discovery");
+      }


the fall-through is here is intentional? is that worth a comment? seems subtle and easy to miss

@stevenctl Yes, this is intentional. We want to keep things consistent with the upstream filter here.

I'm an Envoy maintainer and have made contributions in proxy repository and telemetry related changes in other repos: * istio/proxy#6726 - fix proxy build after Envoy move to hermetic toolchain (including debugging the OOM issue in our CI that broke tests) * istio/proxy#6793 (plus there are a few in flight PRs) * designed telemetry changes for ambient multi-network (see https://docs.google.com/document/d/1gyDsViXTHlozTbHFuggNmYFrL6P3om4ZvTMdy_-Ul74/edit?usp=sharing) There are a few other contributions in Istio that I made (particularly in ambient multi-cluster), but they are not directly related to telemetry. Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

* Include myself, Steven and Gustavo as owners of the experimental-ambient-multicluster-telemetry branch (#6772) * Include myself, Steven and Gustavo as owners of the experimental-ambient-multicluster-telemetry branch * Use single match - creating multiple matches means that the later overrides the earlier --------- * Add Baggage metadata propagation (#6776) * Add Baggage metadata propagation * clang-tidy * Go back to old baggage impl * Fix baggage format * Actually use new baggage approach --------- * Introduce new filters discovering peer metadata from baggage header (#6771) * Introduce new filters discovering peer metadata from baggage header This a combination of two filters that have to be used together: - regular network filter (expected to be configured in connect_originate or inner_connect_originate listeners before TCP Proxy filter) - upstream network filter (expected to be configuration in all clusters that use HBONE or double-HBONE for endpoints) Those two filters together basically create a tunnel. The tunnel protocol just prepends a fixed size header to data stream coming from regular network filter to the upstream network filter, followed by the peer metadatra encoded as protobuf Any containing a protobuf Struct inside (I'm just re-using existing code from Istio proxy, that's why encoding is such as it is). The regular network filter only triggers when there is some data coming from upstream connection in response. It's not correct in general, but in waypoints we do know that we proxy an L7 protocol (http or gRPC), so we do expect a some data in reply. The regular network filter relies on TCP Proxy filter extracting response headers and saving them in the filter state. It then extracts and parses the baggage header from the saved headers. In all cases I explicitly communicate when no peer metadata has been discovered by sending some data downstream. This ensures that upstream network filter running downstream can always remove the prefix from the data stream and does not really need to guess if it's there or not. NOTE: We still do some checks to confirm that the prefix is there, but we cannot really rely on those checks for correctness in all the cases. The upstream network filter, as pointed out above, extracts the data sent by the regular network filter from the data stream, it parses the data and populates filter state based on that. Unlike the HTTP peer metadata filter, this one runs in the context of the upstream connection, so it populates the upstream filter state and not the regular one. I plan to add support to the HTTP peer metadata filter option for new upstream metadata discovery via upstream filter metadata, thus propagating it all the way to the istio stats filter. NOTE: None of those filters are yet generated by pilot and there are certainly some additional options to configure (e.g., maybe we can come up with a good way to transfer metadata via Envoy TLS instead of injecting it into the data stream directly - this way, in principle, we could avoid creating a custom upstream filter all together, if http peer metadata filter could get the peer metadata directly from connect_originate listener). All-in-all, it's not the final implementation. * Fix BUILD formatting * Fix formatting of C++ code * Update HTTP peer_metadata filter to consume filter state set by upstream peer_metadata filter This basically taps the upstream peer metadata into the regular filter state consumed by the istio stats filter. http peer metadata filter also takes care of priorities between different discovery methods - we just need to put different discovery methods in the right order in the configuration. * Populate peer principal in the upstream workload metadata as well * Support propagating baggage header to upstream and additional safety checks for upstream network filter * Only register UpstreamFilterState peer metadata discovery method for upstream peer discovery * Move peer_metadata filter proto config in the same directory * Fix typo --------- * Baggage discovery (#6779) * Add Baggage metadata propagation * clang-tidy * basics for baggage discovery downstream * removing unnecessary tests * reverting crazy claude changes in release-binary.sh * fixing tests, fixing baggage key tokens * removing comment * make lint * fixing unit tests for metadata_object * make lint * suggestions from PR * clarifying use of mappings for baggage and field access * make lint --------- * Add locality to proxy metadata (#6780) * Add locality to proxy metadata * Clang-tidy * Buildifier format * Rebase and fix some bugs --------- * Drop app labels from baggage and propagate principal (#6791) * Drop app labels from baggage and propagate principal I think I confused folks a bit when I mentioned that app field is missing from the baggage - it wasn't. In fact, canonical name of the workload and app in ambient are the same thing, that's why baggage does not actually need an app label - it already has service.name that encodes what we need. I updated the design document, but it happened after I mentioned here and there that we need to add a missing field to the baggage. This change corrects implementation and that makes istio stats populate the app label correctly. The other field that has not been populated is principal. WorkloadMetadataObject contained that identity field that contained principle in principle, but the methods used to conver WorkloadMetadataObject to a protobuf Struct and back ignored that field and never populated it, so it got lost and istio stats never used it. We haven't noticed that before because in ambient we used xDS-based peer metadata discovery by default and it triggers a different code path that does not rely on the methods that convert protobuf Struct to WorkloadMetadataObject, and the code path used there didn't have the same issue. * Keep backwards compatibility for app.service and app.version baggage fields --------- * Fix some test compilation errors * Merge master branch and resolve merge conflicts properly (#6795) * Automator: update envoy@ in istio/proxy@master (#6777) * Automator: update envoy@ in istio/proxy@master (#6778) * Don't do workload discovery for cross-network traffic (#6767) * Get the implementation compiling * Add tests for cross-network peer metadata * clang-tidy * One more tidy * Switch to debug for logging --------- * Automator: update envoy@ in istio/proxy@master (#6782) * Automator: update envoy@ in istio/proxy@master (#6784) * Automator: update go-control-plane in istio/proxy@master (#6786) * Automator: update envoy@ in istio/proxy@master (#6787) * Automator: update envoy@ in istio/proxy@master (#6788) * update x-network header key (#6790) * Automator: update envoy@ in istio/proxy@master (#6794) * Merge upstream/master and resolve merge conflicts * Missed one * Fixed a wrong one --------- --------- Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> Signed-off-by: Keith Mattix II <keithmattix@microsoft.com> Signed-off-by: Ian Rudie <ian.rudie@solo.io> Co-authored-by: Krinkin, Mike <mkrinkin@microsoft.com> Co-authored-by: Gustavo Meira <grnmeira@users.noreply.github.com> Co-authored-by: Istio Automation <istio-testing-bot@google.com> Co-authored-by: Ian Rudie <ilrudie@gmail.com>

* Add krinkinmu as a telemetry maintainer I'm an Envoy maintainer and have made contributions in proxy repository and telemetry related changes in other repos: * istio/proxy#6726 - fix proxy build after Envoy move to hermetic toolchain (including debugging the OOM issue in our CI that broke tests) * istio/proxy#6793 (plus there are a few in flight PRs) * designed telemetry changes for ambient multi-network (see https://docs.google.com/document/d/1gyDsViXTHlozTbHFuggNmYFrL6P3om4ZvTMdy_-Ul74/edit?usp=sharing) There are a few other contributions in Istio that I made (particularly in ambient multi-cluster), but they are not directly related to telemetry. Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> * Fix test Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com> --------- Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

keithmattix requested a review from a team as a code owner January 27, 2026 02:38

istio-testing added needs-rebase Indicates a PR needs to be rebased before being merged size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jan 27, 2026

krinkinmu and others added 6 commits January 27, 2026 02:38

keithmattix force-pushed the experimental-ambient-multicluster-telemetry branch from 0758b62 to bb3bb1c Compare January 27, 2026 02:41

istio-testing removed the needs-rebase Indicates a PR needs to be rebased before being merged label Jan 27, 2026

keithmattix and others added 2 commits January 27, 2026 06:44

Fix some test compilation errors

2c1835e

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

istio-testing added the needs-rebase Indicates a PR needs to be rebased before being merged label Jan 27, 2026

Merge branch 'master' into experimental-ambient-multicluster-telemetry

5c4e694

istio-testing removed the needs-rebase Indicates a PR needs to be rebased before being merged label Jan 27, 2026

keithmattix added the cherrypick/release-1.29 Set this label on a PR to auto-merge it to the release-1.29 branch label Jan 27, 2026

zirain approved these changes Jan 28, 2026

View reviewed changes

istio-testing merged commit 0a3c55b into master Jan 28, 2026
7 of 9 checks passed

keithmattix mentioned this pull request Jan 28, 2026

[release-1.29] Ambient Multicluster Telemetry #6799

Merged

stevenctl reviewed Jan 28, 2026

View reviewed changes

krinkinmu mentioned this pull request Jan 28, 2026

Add krinkinmu as a telemetry maintainer istio/community#1709

Merged

keithmattix mentioned this pull request Jan 28, 2026

xDS-based metadata discovery reports incorrect labels for multi-network deployments istio/istio#58790

Closed

2 tasks

Conversation

keithmattix commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

keithmattix commented Jan 27, 2026

Uh oh!

keithmattix commented Jan 27, 2026

Uh oh!

keithmattix commented Jan 27, 2026

Uh oh!

keithmattix commented Jan 27, 2026

Uh oh!

keithmattix commented Jan 27, 2026

Uh oh!

keithmattix commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

istio-testing commented Jan 27, 2026

Uh oh!

zirain commented Jan 28, 2026

Uh oh!

keithmattix commented Jan 28, 2026

Uh oh!

Uh oh!

istio-testing commented Jan 28, 2026

Uh oh!

istio-testing commented Jan 28, 2026

Uh oh!

stevenctl Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

grnmeira Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

keithmattix commented Jan 27, 2026 •

edited

Loading

keithmattix commented Jan 27, 2026 •

edited

Loading