Skip to content

Document the new xff_num_trusted_hops feature #479

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
htuch merged 4 commits into from
Feb 13, 2018
Merged

Document the new xff_num_trusted_hops feature #479

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
8207c8c
Document the new xff_num_trusted_hops feature
Feb 8, 2018
b8c0a37
fix formatting
Feb 9, 2018
3c9f340
fixes based on code review
Feb 9, 2018
aa9a01d
Merge remote-tracking branch 'upstream/master' into xff-doc/2503
Feb 13, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
133 changes: 116 additions & 17 deletions docs/root/configuration/http_conn_man/headers.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,13 +65,12 @@ the :option:`--service-node` option.
x-envoy-external-address
------------------------

It is a common case where a service wants to perform analytics based on the client IP address. Per
the lengthy discussion on :ref:`XFF <config_http_conn_man_headers_x-forwarded-for>`, this can get
quite complicated. A proper implementation involves forwarding XFF, and then choosing the first non
RFC1918 address *from the right*. Since this is such a common occurrence, Envoy simplifies this by
setting *x-envoy-external-address* during decoding if and only if the request ingresses externally
(i.e., it's from an external client). *x-envoy-external-address* is not set or overwritten for
internal requests. This header can be safely forwarded between internal services for analytics
It is a common case where a service wants to perform analytics based on the origin client's IP
address. Per the lengthy discussion on :ref:`XFF <config_http_conn_man_headers_x-forwarded-for>`,
this can get quite complicated, so Envoy simplifies this by setting *x-envoy-external-address*
to the :ref:`trusted client address <config_http_conn_man_headers_x-forwarded-for_trusted_client_address>`
if the request is from an external client. *x-envoy-external-address* is not set or overwritten
for internal requests. This header can be safely forwarded between internal services for analytics
purposes without having to deal with the complexities of XFF.

.. _config_http_conn_man_headers_x-envoy-force-trace:
Expand Down Expand Up @@ -156,8 +155,10 @@ operates in a transparent mode where it does not modify XFF.
.. attention::

In general, *use_remote_address* should be set to true when Envoy is deployed as an edge
node, whereas it may need to be set to false when Envoy is used as an internal service node
in a mesh deployment.
node (aka a front proxy), whereas it may need to be set to false when Envoy is used as
an internal service node in a mesh deployment.

.. _config_http_conn_man_headers_x-forwarded-for_trusted_client_address:

The value of *use_remote_address* controls how Envoy determines the *trusted client address*.
Given an HTTP request that has traveled through a series of zero or more proxies to reach
Expand All @@ -166,24 +167,116 @@ accurate. The source IP address of the immediate downstream node's connection to
trusted. XFF *sometimes* can be trusted. Malicious clients can forge XFF, but the last
address in XFF can be trusted if it was put there by a trusted proxy.

Envoy's rules for determining the trusted client address are:
Envoy's default rules for determining the trusted client address (*before* appending anything
to XFF) are:

* If *use_remote_address* is false and an XFF containing at least one IP address is
present in the request, the trusted client address is the *last* (rightmost) IP address in XFF.
* Otherwise, the trusted client address is the source IP address of the immediate downstream
node's connection to Envoy.

In an environment where there are one or more trusted proxies in front of an edge
Envoy instance, the *xff_num_trusted_hops* configuration option can be used to trust
additional addresses from XFF:

* If *use_remote_address* is false and *xff_num_trusted_hops* is set to a value *N* that is
greater than zero, the trusted client address is the (N+1)th address from the right end
of XFF. (If the XFF contains fewer than N+1 addresses, Envoy falls back to using the
immediate downstream connection's source address as trusted client address.)
* If *use_remote_address* is true and *xff_num_trusted_hops* is set to a value *N* that is
greater than zero, the trusted client address is the Nth address from the right end
of XFF. (If the XFF contains fewer than N addresses, Envoy falls back to using the
immediate downstream connection's source address as trusted client address.)

Copy link
Copy Markdown
Contributor

@alyssawilk alyssawilk Feb 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So first off, thank you for this amazing documentation, it makes it really clear what is going on over in Envoy #2559

The goal here is to preserve existing Envoy behavior while allowing configuring xff_num_trusted_hops, right? The one thing I'm still struggling with is the use of the combinations and permutations. If we were designing this from scratch, would there be a gain in having the two separate config options, or would we only want one, where we always considered the first hop to be the direct connected host?

Copy link
Copy Markdown
Contributor Author

@brian-pane brian-pane Feb 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on the discussion in Envoy #2559, I created a follow-up issue 2590 to simplify the configuration.

Envoy uses the trusted client address contents to determine whether a request originated
externally or internally. This influences whether the
:ref:`config_http_conn_man_headers_x-envoy-internal` header is set.

Testing IPv6 in a large multi-hop system can be difficult from a change management perspective. For
testing IPv6 compatibility of upstream services which parse XFF header values,
:ref:`represent_ipv4_remote_address_as_ipv4_mapped_ipv6
<envoy_api_field_config.filter.network.http_connection_manager.v2.HttpConnectionManager.represent_ipv4_remote_address_as_ipv4_mapped_ipv6>`
can be enabled in the v2 API. Envoy will append an IPv4 address in mapped IPv6 format, e.g.
::FFFF:50.0.0.1. This change will also apply to
:ref:`config_http_conn_man_headers_x-envoy-external-address`.
Example 1: Envoy as edge proxy, without a trusted proxy in front of it
Settings:
| use_remote_address = true
| xff_num_trusted_hops = 0

Request details:
| Downstream IP address = 192.0.2.5
| XFF = "203.0.113.128, 203.0.113.10, 203.0.113.1"

Result:
| Trusted client address = 192.0.2.5 (XFF is ignored)
| X-Envoy-External-Address is set to 192.0.2.5
| XFF is changed to "203.0.113.128, 203.0.113.10, 203.0.113.1, 192.0.2.5"
| X-Envoy-Internal is removed (if it was present in the incoming request)

Example 2: Envoy as internal proxy, with the Envoy edge proxy from Example 1 in front of it
Settings:
| use_remote_address = false
| xff_num_trusted_hops = 0

Request details:
| Downstream IP address = 10.11.12.13 (address of the Envoy edge proxy)
| XFF = "203.0.113.128, 203.0.113.10, 203.0.113.1, 192.0.2.5"

Result:
| Trusted client address = 192.0.2.5 (last address in XFF is trusted)
| X-Envoy-External-Address is not modified
| X-Envoy-Internal is removed (if it was present in the incoming request)

Example 3: Envoy as edge proxy, with two trusted external proxies in front of it
Settings:
| use_remote_address = true
| xff_num_trusted_hops = 2

Request details:
| Downstream IP address = 192.0.2.5
| XFF = "203.0.113.128, 203.0.113.10, 203.0.113.1"

Result:
| Trusted client address = 203.0.113.10 (2nd to last address in XFF is trusted)
| X-Envoy-External-Address is set to 203.0.113.10
| XFF is changed to "203.0.113.128, 203.0.113.10, 203.0.113.1, 192.0.2.5"
| X-Envoy-Internal is removed (if it was present in the incoming request)

Example 4: Envoy as internal proxy, with the edge proxy from Example 3 in front of it
Settings:
| use_remote_address = false
| xff_num_trusted_hops = 2

Request details:
| Downstream IP address = 10.11.12.13 (address of the Envoy edge proxy)
| XFF = "203.0.113.128, 203.0.113.10, 203.0.113.1, 192.0.2.5"

Result:
| Trusted client address = 203.0.113.10
| X-Envoy-External-Address is not modified
| X-Envoy-Internal is removed (if it was present in the incoming request)

Example 5: Envoy as an internal proxy, receiving a request from an internal client
Settings:
| use_remote_address = false
| xff_num_trusted_hops = 0

Request details:
| Downstream IP address = 10.20.30.40 (address of the internal client)
| XFF is not present

Result:
| Trusted client address = 10.20.30.40
| X-Envoy-External-Address remains unset
| X-Envoy-Internal is set to "true"

Example 6: The internal Envoy from Example 5, receiving a request proxied by another Envoy
Settings:
| use_remote_address = false
| xff_num_trusted_hops = 0

Request details:
| Downstream IP address = 10.20.30.50 (address of the Envoy instance proxying to this one)
| XFF = "10.20.30.40"

Result:
| Trusted client address = 10.20.30.40
| X-Envoy-External-Address remains unset
| X-Envoy-Internal is set to "true"

A few very important notes about XFF:

Expand All @@ -206,6 +299,12 @@ A few very important notes about XFF:
Envoy will not consider it internal. This is a known "bug" due to the simplification of how
XFF is parsed to determine if a request is internal. In this scenario, do not forward XFF and
allow Envoy to generate a new one with a single internal origin IP.
3. Testing IPv6 in a large multi-hop system can be difficult from a change management perspective.
For testing IPv6 compatibility of upstream services which parse XFF header values,
:ref:`represent_ipv4_remote_address_as_ipv4_mapped_ipv6 <envoy_api_field_config.filter.network.http_connection_manager.v2.HttpConnectionManager.represent_ipv4_remote_address_as_ipv4_mapped_ipv6>`
can be enabled in the v2 API. Envoy will append an IPv4 address in mapped IPv6 format, e.g.
::FFFF:50.0.0.1. This change will also apply to
:ref:`config_http_conn_man_headers_x-envoy-external-address`.

.. _config_http_conn_man_headers_x-forwarded-proto:

Expand Down
1 change: 0 additions & 1 deletion envoy/config/filter/network/http_connection_manager/v2/http_connection_manager.proto
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,6 @@ message HttpConnectionManager {
// :ref:`config_http_conn_man_headers_x-envoy-external-address` for more information.
google.protobuf.BoolValue use_remote_address = 14;

// [#not-implemented-hide:]
// The number of additional ingress proxy hops from the right side of the
// :ref:`config_http_conn_man_headers_x-forwarded-for` HTTP header to trust when
// determining the origin client's IP address. The default is zero if this option
Expand Down