Skip to content

Add e2e test for proxy baggage-based metadata discovery + fixes#6798

Merged
istio-testing merged 17 commits into
istio:masterfrom
krinkinmu:experimental-ambient-multicluster-telemetry3
Jan 29, 2026
Merged

Add e2e test for proxy baggage-based metadata discovery + fixes#6798
istio-testing merged 17 commits into
istio:masterfrom
krinkinmu:experimental-ambient-multicluster-telemetry3

Conversation

@krinkinmu
Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

This adds an e2e test for new proxy filters that verifies both baggage
propagation as well as metrics that stats filter will generate.

I also made some changes to how we parse and generate baggage header.
Basically, WorkloadMetadataObject supports cases where app and service
might have different values. In xDS-based metadata discovery
implementation however app is always derived from the service name, at
least in ztunnel.

So there is a bit of a mismatch there. In practice this mismatch should
not matter, but purely hypothetically, if I didn't change the logic, we
could end up in a situation where waypoint node metadata is configured
in such a way that app is set, but service is not. And if that happens,
waypoint will generate a baggage, that ztunnels cannot interpret
correctly yet and it will result in metrics with unknown values.

I figured that I can rewrite code in a way that accounts for all corner
cases like that by making sure that:

  1. When we parse baggage we set both app and service, and if any of
    those is not provided in the baggage, we use the other to backfill
  2. When we generate baggage, we always generate service (because ztunnel
    needs it), and we generate app, if it's different from the service.

All-in-all, under normal circumstances in the baggage we will only have
service; if app is different from the service - we will add it to the
baggage as well; and when parsing baggage we will use either service or
app, depending on what is provided.

Which issue this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close that issue when PR gets merged): fixes #

Special notes for your reviewer:

+cc @keithmattix @Stevenjin8 @grnmeira

krinkinmu and others added 15 commits January 27, 2026 02:38
…ent-multicluster-telemetry branch (istio#6772)

* Include myself, Steven and Gustavo as owners of the
experimental-ambient-multicluster-telemetry branch

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

* Use single match - creating multiple matches means that the later overrides the earlier

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

---------

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>
* Add Baggage metadata propagation

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

* clang-tidy

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

* Go back to old baggage impl

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

* Fix baggage format

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

* Actually use new baggage approach

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

---------

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>
…stio#6771)

* Introduce new filters discovering peer metadata from baggage header

This a combination of two filters that have to be used together:
- regular network filter (expected to be configured in connect_originate
  or inner_connect_originate listeners before TCP Proxy filter)
- upstream network filter (expected to be configuration in all clusters
  that use HBONE or double-HBONE for endpoints)

Those two filters together basically create a tunnel. The tunnel
protocol just prepends a fixed size header to data stream coming from
regular network filter to the upstream network filter, followed by the
peer metadatra encoded as protobuf Any containing a protobuf Struct
inside (I'm just re-using existing code from Istio proxy, that's why
encoding is such as it is).

The regular network filter only triggers when there is some data coming
from upstream connection in response. It's not correct in general, but
in waypoints we do know that we proxy an L7 protocol (http or gRPC), so
we do expect a some data in reply.

The regular network filter relies on TCP Proxy filter extracting
response headers and saving them in the filter state. It then extracts
and parses the baggage header from the saved headers.

In all cases I explicitly communicate when no peer metadata has been
discovered by sending some data downstream. This ensures that upstream
network filter running downstream can always remove the prefix from the
data stream and does not really need to guess if it's there or not.

NOTE: We still do some checks to confirm that the prefix is there, but
we cannot really rely on those checks for correctness in all the cases.

The upstream network filter, as pointed out above, extracts the data
sent by the regular network filter from the data stream, it parses the
data and populates filter state based on that.

Unlike the HTTP peer metadata filter, this one runs in the context of
the upstream connection, so it populates the upstream filter state and
not the regular one.

I plan to add support to the HTTP peer metadata filter option for new
upstream metadata discovery via upstream filter metadata, thus
propagating it all the way to the istio stats filter.

NOTE: None of those filters are yet generated by pilot and there are
certainly some additional options to configure (e.g., maybe we can come
up with a good way to transfer metadata via Envoy TLS instead of
injecting it into the data stream directly - this way, in principle, we
could avoid creating a custom upstream filter all together, if http peer
metadata filter could get the peer metadata directly from
connect_originate listener). All-in-all, it's not the final
implementation.

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

* Fix BUILD formatting

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

* Fix formatting of C++ code

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

* Update HTTP peer_metadata filter to consume filter state set by upstream
peer_metadata filter

This basically taps the upstream peer metadata into the regular filter
state consumed by the istio stats filter. http peer metadata filter also
takes care of priorities between different discovery methods - we just
need to put different discovery methods in the right order in the
configuration.

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

* Populate peer principal in the upstream workload metadata as well

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

* Support propagating baggage header to upstream and additional safety checks for upstream network filter

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

* Only register UpstreamFilterState peer metadata discovery method for upstream peer discovery

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

* Move peer_metadata filter proto config in the same directory

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

* Fix typo

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

---------

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>
* Add Baggage metadata propagation

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

* clang-tidy

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

* basics for baggage discovery downstream

* removing unnecessary tests

* reverting crazy claude changes in release-binary.sh

* fixing tests, fixing baggage key tokens

* removing comment

* make lint

* fixing unit tests for metadata_object

* make lint

* suggestions from PR

* clarifying use of mappings for baggage and field access

* make lint

---------

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>
Co-authored-by: Keith Mattix II <keithmattix@microsoft.com>
* Add locality to proxy metadata

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

* Clang-tidy

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

* Buildifier format

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

* Rebase and fix some bugs

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

---------

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>
* Drop app labels from baggage and propagate principal

I think I confused folks a bit when I mentioned that app field is
missing from the baggage - it wasn't. In fact, canonical name of the
workload and app in ambient are the same thing, that's why baggage does
not actually need an app label - it already has service.name that
encodes what we need.

I updated the design document, but it happened after I mentioned here
and there that we need to add a missing field to the baggage.

This change corrects implementation and that makes istio stats populate
the app label correctly.

The other field that has not been populated is principal.
WorkloadMetadataObject contained that identity field that contained
principle in principle, but the methods used to conver
WorkloadMetadataObject to a protobuf Struct and back ignored that field
and never populated it, so it got lost and istio stats never used it.

We haven't noticed that before because in ambient we used xDS-based
peer metadata discovery by default and it triggers a different code
path that does not rely on the methods that convert protobuf Struct
to WorkloadMetadataObject, and the code path used there didn't have the
same issue.

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

* Keep backwards compatibility for app.service and app.version baggage fields

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

---------

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>
Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>
Co-authored-by: Keith Mattix II <keithmattix@microsoft.com>
Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>
Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>
Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>
Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>
* Automator: update envoy@ in istio/proxy@master (istio#6777)

* Automator: update envoy@ in istio/proxy@master (istio#6778)

* Don't do workload discovery for cross-network traffic (istio#6767)

* Get the implementation compiling

* Add tests for cross-network peer metadata

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

* clang-tidy

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

* One more tidy

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

* Switch to debug for logging

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

---------

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

* Automator: update envoy@ in istio/proxy@master (istio#6782)

* Automator: update envoy@ in istio/proxy@master (istio#6784)

* Automator: update go-control-plane in istio/proxy@master (istio#6786)

* Automator: update envoy@ in istio/proxy@master (istio#6787)

* Automator: update envoy@ in istio/proxy@master (istio#6788)

* update x-network header key (istio#6790)

Signed-off-by: Ian Rudie <ian.rudie@solo.io>

* Automator: update envoy@ in istio/proxy@master (istio#6794)

* Merge upstream/master and resolve merge conflicts

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

* Missed one

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

* Fixed a wrong one

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

---------

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>
Signed-off-by: Ian Rudie <ian.rudie@solo.io>
Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>
Co-authored-by: Istio Automation <istio-testing-bot@google.com>
Co-authored-by: Keith Mattix II <keithmattix@microsoft.com>
Co-authored-by: Ian Rudie <ilrudie@gmail.com>
This adds an e2e test for new proxy filters that verifies both baggage
propagation as well as metrics that stats filter will generate.

I also made some changes to how we parse and generate baggage header.
Basically, WorkloadMetadataObject supports cases where app and service
might have different values. In xDS-based metadata discovery
implementation however app is always derived from the service name, at
least in ztunnel.

So there is a bit of a mismatch there. In practice this mismatch should
not matter, but purely hypothetically, if I didn't change the logic, we
could end up in a situation where waypoint node metadata is configured
in such a way that app is set, but service is not. And if that happens,
waypoint will generate a baggage, that ztunnels cannot interpret
correctly yet and it will result in metrics with unknown values.

I figured that I can rewrite code in a way that accounts for all corner
cases like that by making sure that:

1. When we parse baggage we set both app and service, and if any of
   those is not provided in the baggage, we use the other to backfill
2. When we generate baggage, we always generate service (because ztunnel
   needs it), and we generate app, if it's different from the service.

All-in-all, under normal circumstances in the baggage we will only have
service; if app is different from the service - we will add it to the
baggage as well; and when parsing baggage we will use either service or
app, depending on what is provided.

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>
…ter-telemetry' into experimental-ambient-multicluster-telemetry3
@krinkinmu krinkinmu requested a review from a team as a code owner January 28, 2026 02:27
@istio-testing istio-testing added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jan 28, 2026
Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>
@krinkinmu
Copy link
Copy Markdown
Contributor Author

/retest

1 similar comment
@krinkinmu
Copy link
Copy Markdown
Contributor Author

/retest

@keithmattix
Copy link
Copy Markdown
Contributor

@krinkinmu Can you make this PR into master?

@krinkinmu krinkinmu changed the base branch from experimental-ambient-multicluster-telemetry to master January 28, 2026 16:18
@istio-testing istio-testing added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jan 28, 2026
@istio-testing istio-testing added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Jan 28, 2026
@krinkinmu
Copy link
Copy Markdown
Contributor Author

/retest-required

@keithmattix
Copy link
Copy Markdown
Contributor

/retest

@keithmattix
Copy link
Copy Markdown
Contributor

/retest

@keithmattix keithmattix added the cherrypick/release-1.29 Set this label on a PR to auto-merge it to the release-1.29 branch label Jan 29, 2026
@keithmattix
Copy link
Copy Markdown
Contributor

Can come after branch cut if necessary; it's mostly just a test addition

// on top of that
// - server plays a role of a destination workload ztunnel - it terminates HBONE connection
// - client plays a role of a waypoint
// 2. For connection from the test to the client we don't use HBONE:
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: We should note that because we don't use HBONE to the client, there's no source principal (which is expected)

@istio-testing istio-testing merged commit 2b43757 into istio:master Jan 29, 2026
7 checks passed
@istio-testing
Copy link
Copy Markdown
Collaborator

In response to a cherrypick label: #6798 failed to apply on top of branch "release-1.29":

Applying: Include myself, Steven and Gustavo as owners of the experimental-ambient-multicluster-telemetry branch (#6772)
Using index info to reconstruct a base tree...
M	CODEOWNERS
Falling back to patching base and 3-way merge...
Auto-merging CODEOWNERS
CONFLICT (content): Merge conflict in CODEOWNERS
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
Patch failed at 0001 Include myself, Steven and Gustavo as owners of the experimental-ambient-multicluster-telemetry branch (#6772)

@istio-testing
Copy link
Copy Markdown
Collaborator

In response to a cherrypick label: new issue could not be created for failed cherrypick: status code 410 not one of [201], body: {"message":"Issues has been disabled in this repository.","documentation_url":"https://docs.github.com/rest/issues/issues#create-an-issue","status":"410"}

krinkinmu added a commit to krinkinmu/proxy that referenced this pull request Jan 29, 2026
…o#6798)

* Include myself, Steven and Gustavo as owners of the experimental-ambient-multicluster-telemetry branch (istio#6772)

* Include myself, Steven and Gustavo as owners of the
experimental-ambient-multicluster-telemetry branch

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

* Use single match - creating multiple matches means that the later overrides the earlier

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

---------

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

* Add Baggage metadata propagation (istio#6776)

* Add Baggage metadata propagation

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

* clang-tidy

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

* Go back to old baggage impl

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

* Fix baggage format

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

* Actually use new baggage approach

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

---------

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

* Introduce new filters discovering peer metadata from baggage header (istio#6771)

* Introduce new filters discovering peer metadata from baggage header

This a combination of two filters that have to be used together:
- regular network filter (expected to be configured in connect_originate
  or inner_connect_originate listeners before TCP Proxy filter)
- upstream network filter (expected to be configuration in all clusters
  that use HBONE or double-HBONE for endpoints)

Those two filters together basically create a tunnel. The tunnel
protocol just prepends a fixed size header to data stream coming from
regular network filter to the upstream network filter, followed by the
peer metadatra encoded as protobuf Any containing a protobuf Struct
inside (I'm just re-using existing code from Istio proxy, that's why
encoding is such as it is).

The regular network filter only triggers when there is some data coming
from upstream connection in response. It's not correct in general, but
in waypoints we do know that we proxy an L7 protocol (http or gRPC), so
we do expect a some data in reply.

The regular network filter relies on TCP Proxy filter extracting
response headers and saving them in the filter state. It then extracts
and parses the baggage header from the saved headers.

In all cases I explicitly communicate when no peer metadata has been
discovered by sending some data downstream. This ensures that upstream
network filter running downstream can always remove the prefix from the
data stream and does not really need to guess if it's there or not.

NOTE: We still do some checks to confirm that the prefix is there, but
we cannot really rely on those checks for correctness in all the cases.

The upstream network filter, as pointed out above, extracts the data
sent by the regular network filter from the data stream, it parses the
data and populates filter state based on that.

Unlike the HTTP peer metadata filter, this one runs in the context of
the upstream connection, so it populates the upstream filter state and
not the regular one.

I plan to add support to the HTTP peer metadata filter option for new
upstream metadata discovery via upstream filter metadata, thus
propagating it all the way to the istio stats filter.

NOTE: None of those filters are yet generated by pilot and there are
certainly some additional options to configure (e.g., maybe we can come
up with a good way to transfer metadata via Envoy TLS instead of
injecting it into the data stream directly - this way, in principle, we
could avoid creating a custom upstream filter all together, if http peer
metadata filter could get the peer metadata directly from
connect_originate listener). All-in-all, it's not the final
implementation.

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

* Fix BUILD formatting

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

* Fix formatting of C++ code

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

* Update HTTP peer_metadata filter to consume filter state set by upstream
peer_metadata filter

This basically taps the upstream peer metadata into the regular filter
state consumed by the istio stats filter. http peer metadata filter also
takes care of priorities between different discovery methods - we just
need to put different discovery methods in the right order in the
configuration.

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

* Populate peer principal in the upstream workload metadata as well

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

* Support propagating baggage header to upstream and additional safety checks for upstream network filter

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

* Only register UpstreamFilterState peer metadata discovery method for upstream peer discovery

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

* Move peer_metadata filter proto config in the same directory

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

* Fix typo

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

---------

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

* Baggage discovery (istio#6779)

* Add Baggage metadata propagation

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

* clang-tidy

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

* basics for baggage discovery downstream

* removing unnecessary tests

* reverting crazy claude changes in release-binary.sh

* fixing tests, fixing baggage key tokens

* removing comment

* make lint

* fixing unit tests for metadata_object

* make lint

* suggestions from PR

* clarifying use of mappings for baggage and field access

* make lint

---------

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>
Co-authored-by: Keith Mattix II <keithmattix@microsoft.com>

* Add locality to proxy metadata (istio#6780)

* Add locality to proxy metadata

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

* Clang-tidy

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

* Buildifier format

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

* Rebase and fix some bugs

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

---------

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

* Drop app labels from baggage and propagate principal (istio#6791)

* Drop app labels from baggage and propagate principal

I think I confused folks a bit when I mentioned that app field is
missing from the baggage - it wasn't. In fact, canonical name of the
workload and app in ambient are the same thing, that's why baggage does
not actually need an app label - it already has service.name that
encodes what we need.

I updated the design document, but it happened after I mentioned here
and there that we need to add a missing field to the baggage.

This change corrects implementation and that makes istio stats populate
the app label correctly.

The other field that has not been populated is principal.
WorkloadMetadataObject contained that identity field that contained
principle in principle, but the methods used to conver
WorkloadMetadataObject to a protobuf Struct and back ignored that field
and never populated it, so it got lost and istio stats never used it.

We haven't noticed that before because in ambient we used xDS-based
peer metadata discovery by default and it triggers a different code
path that does not rely on the methods that convert protobuf Struct
to WorkloadMetadataObject, and the code path used there didn't have the
same issue.

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

* Keep backwards compatibility for app.service and app.version baggage fields

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

---------

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>
Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>
Co-authored-by: Keith Mattix II <keithmattix@microsoft.com>

* Fix some test compilation errors

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

* Merge upstream/master and resolve merge conflicts

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

* Missed one

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

* Fixed a wrong one

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

* Merge master branch and resolve merge conflicts properly (istio#6795)

* Automator: update envoy@ in istio/proxy@master (istio#6777)

* Automator: update envoy@ in istio/proxy@master (istio#6778)

* Don't do workload discovery for cross-network traffic (istio#6767)

* Get the implementation compiling

* Add tests for cross-network peer metadata

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

* clang-tidy

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

* One more tidy

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

* Switch to debug for logging

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

---------

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>

* Automator: update envoy@ in istio/proxy@master (istio#6782)

* Automator: update envoy@ in istio/proxy@master (istio#6784)

* Automator: update go-control-plane in istio/proxy@master (istio#6786)

* Automator: update envoy@ in istio/proxy@master (istio#6787)

* Automator: update envoy@ in istio/proxy@master (istio#6788)

* update x-network header key (istio#6790)

Signed-off-by: Ian Rudie <ian.rudie@solo.io>

* Automator: update envoy@ in istio/proxy@master (istio#6794)

* Merge upstream/master and resolve merge conflicts

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

* Missed one

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

* Fixed a wrong one

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

---------

Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>
Signed-off-by: Ian Rudie <ian.rudie@solo.io>
Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>
Co-authored-by: Istio Automation <istio-testing-bot@google.com>
Co-authored-by: Keith Mattix II <keithmattix@microsoft.com>
Co-authored-by: Ian Rudie <ilrudie@gmail.com>

* Add e2e test for proxy baggage-based metadata discovery + fixes

This adds an e2e test for new proxy filters that verifies both baggage
propagation as well as metrics that stats filter will generate.

I also made some changes to how we parse and generate baggage header.
Basically, WorkloadMetadataObject supports cases where app and service
might have different values. In xDS-based metadata discovery
implementation however app is always derived from the service name, at
least in ztunnel.

So there is a bit of a mismatch there. In practice this mismatch should
not matter, but purely hypothetically, if I didn't change the logic, we
could end up in a situation where waypoint node metadata is configured
in such a way that app is set, but service is not. And if that happens,
waypoint will generate a baggage, that ztunnels cannot interpret
correctly yet and it will result in metrics with unknown values.

I figured that I can rewrite code in a way that accounts for all corner
cases like that by making sure that:

1. When we parse baggage we set both app and service, and if any of
   those is not provided in the baggage, we use the other to backfill
2. When we generate baggage, we always generate service (because ztunnel
   needs it), and we generate app, if it's different from the service.

All-in-all, under normal circumstances in the baggage we will only have
service; if app is different from the service - we will add it to the
baggage as well; and when parsing baggage we will use either service or
app, depending on what is provided.

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

* Fix formatting

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>

---------

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>
Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>
Signed-off-by: Ian Rudie <ian.rudie@solo.io>
Co-authored-by: Keith Mattix II <keithmattix@microsoft.com>
Co-authored-by: Gustavo Meira <grnmeira@users.noreply.github.com>
Co-authored-by: Istio Automation <istio-testing-bot@google.com>
Co-authored-by: Ian Rudie <ilrudie@gmail.com>
istio-testing added a commit that referenced this pull request Feb 27, 2026
… (#6802)

* Include myself, Steven and Gustavo as owners of the experimental-ambient-multicluster-telemetry branch (#6772)

* Include myself, Steven and Gustavo as owners of the
experimental-ambient-multicluster-telemetry branch



* Use single match - creating multiple matches means that the later overrides the earlier



---------



* Add Baggage metadata propagation (#6776)

* Add Baggage metadata propagation



* clang-tidy



* Go back to old baggage impl



* Fix baggage format



* Actually use new baggage approach



---------



* Introduce new filters discovering peer metadata from baggage header (#6771)

* Introduce new filters discovering peer metadata from baggage header

This a combination of two filters that have to be used together:
- regular network filter (expected to be configured in connect_originate
  or inner_connect_originate listeners before TCP Proxy filter)
- upstream network filter (expected to be configuration in all clusters
  that use HBONE or double-HBONE for endpoints)

Those two filters together basically create a tunnel. The tunnel
protocol just prepends a fixed size header to data stream coming from
regular network filter to the upstream network filter, followed by the
peer metadatra encoded as protobuf Any containing a protobuf Struct
inside (I'm just re-using existing code from Istio proxy, that's why
encoding is such as it is).

The regular network filter only triggers when there is some data coming
from upstream connection in response. It's not correct in general, but
in waypoints we do know that we proxy an L7 protocol (http or gRPC), so
we do expect a some data in reply.

The regular network filter relies on TCP Proxy filter extracting
response headers and saving them in the filter state. It then extracts
and parses the baggage header from the saved headers.

In all cases I explicitly communicate when no peer metadata has been
discovered by sending some data downstream. This ensures that upstream
network filter running downstream can always remove the prefix from the
data stream and does not really need to guess if it's there or not.

NOTE: We still do some checks to confirm that the prefix is there, but
we cannot really rely on those checks for correctness in all the cases.

The upstream network filter, as pointed out above, extracts the data
sent by the regular network filter from the data stream, it parses the
data and populates filter state based on that.

Unlike the HTTP peer metadata filter, this one runs in the context of
the upstream connection, so it populates the upstream filter state and
not the regular one.

I plan to add support to the HTTP peer metadata filter option for new
upstream metadata discovery via upstream filter metadata, thus
propagating it all the way to the istio stats filter.

NOTE: None of those filters are yet generated by pilot and there are
certainly some additional options to configure (e.g., maybe we can come
up with a good way to transfer metadata via Envoy TLS instead of
injecting it into the data stream directly - this way, in principle, we
could avoid creating a custom upstream filter all together, if http peer
metadata filter could get the peer metadata directly from
connect_originate listener). All-in-all, it's not the final
implementation.



* Fix BUILD formatting



* Fix formatting of C++ code



* Update HTTP peer_metadata filter to consume filter state set by upstream
peer_metadata filter

This basically taps the upstream peer metadata into the regular filter
state consumed by the istio stats filter. http peer metadata filter also
takes care of priorities between different discovery methods - we just
need to put different discovery methods in the right order in the
configuration.



* Populate peer principal in the upstream workload metadata as well



* Support propagating baggage header to upstream and additional safety checks for upstream network filter



* Only register UpstreamFilterState peer metadata discovery method for upstream peer discovery



* Move peer_metadata filter proto config in the same directory



* Fix typo



---------



* Baggage discovery (#6779)

* Add Baggage metadata propagation



* clang-tidy



* basics for baggage discovery downstream

* removing unnecessary tests

* reverting crazy claude changes in release-binary.sh

* fixing tests, fixing baggage key tokens

* removing comment

* make lint

* fixing unit tests for metadata_object

* make lint

* suggestions from PR

* clarifying use of mappings for baggage and field access

* make lint

---------




* Add locality to proxy metadata (#6780)

* Add locality to proxy metadata



* Clang-tidy



* Buildifier format



* Rebase and fix some bugs



---------



* Drop app labels from baggage and propagate principal (#6791)

* Drop app labels from baggage and propagate principal

I think I confused folks a bit when I mentioned that app field is
missing from the baggage - it wasn't. In fact, canonical name of the
workload and app in ambient are the same thing, that's why baggage does
not actually need an app label - it already has service.name that
encodes what we need.

I updated the design document, but it happened after I mentioned here
and there that we need to add a missing field to the baggage.

This change corrects implementation and that makes istio stats populate
the app label correctly.

The other field that has not been populated is principal.
WorkloadMetadataObject contained that identity field that contained
principle in principle, but the methods used to conver
WorkloadMetadataObject to a protobuf Struct and back ignored that field
and never populated it, so it got lost and istio stats never used it.

We haven't noticed that before because in ambient we used xDS-based
peer metadata discovery by default and it triggers a different code
path that does not rely on the methods that convert protobuf Struct
to WorkloadMetadataObject, and the code path used there didn't have the
same issue.



* Keep backwards compatibility for app.service and app.version baggage fields



---------





* Fix some test compilation errors



* Merge upstream/master and resolve merge conflicts



* Missed one



* Fixed a wrong one



* Merge master branch and resolve merge conflicts properly (#6795)

* Automator: update envoy@ in istio/proxy@master (#6777)

* Automator: update envoy@ in istio/proxy@master (#6778)

* Don't do workload discovery for cross-network traffic (#6767)

* Get the implementation compiling

* Add tests for cross-network peer metadata



* clang-tidy



* One more tidy



* Switch to debug for logging



---------



* Automator: update envoy@ in istio/proxy@master (#6782)

* Automator: update envoy@ in istio/proxy@master (#6784)

* Automator: update go-control-plane in istio/proxy@master (#6786)

* Automator: update envoy@ in istio/proxy@master (#6787)

* Automator: update envoy@ in istio/proxy@master (#6788)

* update x-network header key (#6790)



* Automator: update envoy@ in istio/proxy@master (#6794)

* Merge upstream/master and resolve merge conflicts



* Missed one



* Fixed a wrong one



---------








* Add e2e test for proxy baggage-based metadata discovery + fixes

This adds an e2e test for new proxy filters that verifies both baggage
propagation as well as metrics that stats filter will generate.

I also made some changes to how we parse and generate baggage header.
Basically, WorkloadMetadataObject supports cases where app and service
might have different values. In xDS-based metadata discovery
implementation however app is always derived from the service name, at
least in ztunnel.

So there is a bit of a mismatch there. In practice this mismatch should
not matter, but purely hypothetically, if I didn't change the logic, we
could end up in a situation where waypoint node metadata is configured
in such a way that app is set, but service is not. And if that happens,
waypoint will generate a baggage, that ztunnels cannot interpret
correctly yet and it will result in metrics with unknown values.

I figured that I can rewrite code in a way that accounts for all corner
cases like that by making sure that:

1. When we parse baggage we set both app and service, and if any of
   those is not provided in the baggage, we use the other to backfill
2. When we generate baggage, we always generate service (because ztunnel
   needs it), and we generate app, if it's different from the service.

All-in-all, under normal circumstances in the baggage we will only have
service; if app is different from the service - we will add it to the
baggage as well; and when parsing baggage we will use either service or
app, depending on what is provided.



* Fix formatting



---------

Signed-off-by: Mikhail Krinkin <mkrinkin@microsoft.com>
Signed-off-by: Keith Mattix II <keithmattix@microsoft.com>
Signed-off-by: Ian Rudie <ian.rudie@solo.io>
Co-authored-by: Keith Mattix II <keithmattix@microsoft.com>
Co-authored-by: Gustavo Meira <grnmeira@users.noreply.github.com>
Co-authored-by: Istio Automation <istio-testing-bot@google.com>
Co-authored-by: Ian Rudie <ilrudie@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cherrypick/release-1.29 Set this label on a PR to auto-merge it to the release-1.29 branch size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants