From 8207c8c0a9cd89fa779e40ef7d68a47389e07801 Mon Sep 17 00:00:00 2001 From: Brian Pane Date: Thu, 8 Feb 2018 23:09:09 +0000 Subject: [PATCH 1/3] Document the new xff_num_trusted_hops feature Signed-off-by: Brian Pane --- .../configuration/http_conn_man/headers.rst | 88 +++++++++++++++++-- .../v2/http_connection_manager.proto | 1 - 2 files changed, 79 insertions(+), 10 deletions(-) diff --git a/docs/root/configuration/http_conn_man/headers.rst b/docs/root/configuration/http_conn_man/headers.rst index 16cda43d8..9b4558584 100644 --- a/docs/root/configuration/http_conn_man/headers.rst +++ b/docs/root/configuration/http_conn_man/headers.rst @@ -65,12 +65,11 @@ the :option:`--service-node` option. x-envoy-external-address ------------------------ -It is a common case where a service wants to perform analytics based on the client IP address. Per -the lengthy discussion on :ref:`XFF `, this can get -quite complicated. A proper implementation involves forwarding XFF, and then choosing the first non -RFC1918 address *from the right*. Since this is such a common occurrence, Envoy simplifies this by -setting *x-envoy-external-address* during decoding if and only if the request ingresses externally -(i.e., it's from an external client). *x-envoy-external-address* is not set or overwritten for +It is a common case where a service wants to perform analytics based on the origin client's IP +address. Per the lengthy discussion on :ref:`XFF `, +this can get quite complicated, so Envoy simplifies this by setting *x-envoy-external-address* +to the *trusted client address* (as defined in that XFF discussion) if the request is from +an external client. *x-envoy-external-address* is not set or overwritten for internal requests. This header can be safely forwarded between internal services for analytics purposes without having to deal with the complexities of XFF. @@ -156,8 +155,8 @@ operates in a transparent mode where it does not modify XFF. .. attention:: In general, *use_remote_address* should be set to true when Envoy is deployed as an edge - node, whereas it may need to be set to false when Envoy is used as an internal service node - in a mesh deployment. + node (aka a front proxy), whereas it may need to be set to false when Envoy is used as + an internal service node in a mesh deployment. The value of *use_remote_address* controls how Envoy determines the *trusted client address*. Given an HTTP request that has traveled through a series of zero or more proxies to reach @@ -166,17 +165,88 @@ accurate. The source IP address of the immediate downstream node's connection to trusted. XFF *sometimes* can be trusted. Malicious clients can forge XFF, but the last address in XFF can be trusted if it was put there by a trusted proxy. -Envoy's rules for determining the trusted client address are: +Envoy's default rules for determining the trusted client address (*before* appending anything +to XFF) are: * If *use_remote_address* is false and an XFF containing at least one IP address is present in the request, the trusted client address is the *last* (rightmost) IP address in XFF. * Otherwise, the trusted client address is the source IP address of the immediate downstream node's connection to Envoy. +In an environment where there are one or more trusted proxies in front of an edge +Envoy instance, the *xff_num_trusted_hops* configuration option can be used to trust +additional addresses from XFF: + +* If *use_remote_address* is false and *xff_num_trusted_hops* is set to a value *N* that is + greater than zero, the trusted client address is the *N+1* th address from the right end + of XFF. +* If *use_remote_address* is true and *xff_num_trusted_hops* is set to a value *N* that is + greater than zero, the trusted client address is the *N* th address from the right end + of XFF. + Envoy uses the trusted client address contents to determine whether a request originated externally or internally. This influences whether the :ref:`config_http_conn_man_headers_x-envoy-internal` header is set. +Example 1: Envoy as edge proxy, without a trusted proxy in front of it + Settings: + | use_remote_address = true + | xff_num_trusted_hops = 0 + + Request details: + | Downstream IP address = 192.0.2.5 + | XFF = "198.51.100.128, 198.51.100.10, 198.51.100.1" + + Result: + | Trusted client address = 192.0.2.5 (XFF is ignored) + | X-Envoy-External-Address is set to 192.0.2.5 + | XFF is changed to "198.51.100.128, 198.51.100.10, 198.51.100.1, 192.0.2.5" + | Request type = external + +Example 2: Envoy as internal proxy, with the Envoy edge proxy from Example 1 in front of it + Settings: + | use_remote_address = false + | xff_num_trusted_hops = 0 + + Request details: + | Downstream IP address = 10.11.12.13 (address of the Envoy edge proxy) + | XFF = "198.51.100.128, 198.51.100.10, 198.51.100.1, 192.0.2.5" + + Result: + | Trusted client address = 192.0.2.5 (last address in XFF is trusted) + | X-Envoy-External-Address is not modified + | Request type = internal + + +Example 3: Envoy as edge proxy, with two trusted external proxies in front of it + Settings: + | use_remote_address = true + | xff_num_trusted_hops = 2 + + Request details: + | Downstream IP address = 192.0.2.5 + | XFF = "198.51.100.128, 198.51.100.10, 198.51.100.1" + + Result: + | Trusted client address = 198.51.100.10 (2nd to last address in XFF is trusted) + | X-Envoy-External-Address is set to 198.51.100.10 + | XFF is changed to "198.51.100.128, 198.51.100.10, 198.51.100.1, 192.0.2.5" + | Request type = external + +Example 4: Envoy as internal proxy, with the edge proxy from Example 3 in front of it + Settings: + | use_remote_address = true + | xff_num_trusted_hops = 2 + + Request details: + | Downstream IP address = 10.11.12.13 (address of the Envoy edge proxy) + | XFF = "198.51.100.128, 198.51.100.10, 198.51.100.1, 192.0.2.5" + + Result: + | Trusted client address = 198.51.100.10 + | X-Envoy-External-Address is not modified + | Request type = internal + A few very important notes about XFF: 1. If *use_remote_address* is set to true, Envoy sets the diff --git a/envoy/config/filter/network/http_connection_manager/v2/http_connection_manager.proto b/envoy/config/filter/network/http_connection_manager/v2/http_connection_manager.proto index b0c37940e..bf019169f 100644 --- a/envoy/config/filter/network/http_connection_manager/v2/http_connection_manager.proto +++ b/envoy/config/filter/network/http_connection_manager/v2/http_connection_manager.proto @@ -160,7 +160,6 @@ message HttpConnectionManager { // :ref:`config_http_conn_man_headers_x-envoy-external-address` for more information. google.protobuf.BoolValue use_remote_address = 14; - // [#not-implemented-hide:] // The number of additional ingress proxy hops from the right side of the // :ref:`config_http_conn_man_headers_x-forwarded-for` HTTP header to trust when // determining the origin client's IP address. The default is zero if this option From b8c0a37c29c7032d6604850185d00e2b2e219d77 Mon Sep 17 00:00:00 2001 From: Brian Pane Date: Fri, 9 Feb 2018 03:34:22 +0000 Subject: [PATCH 2/3] fix formatting Signed-off-by: Brian Pane --- docs/root/configuration/http_conn_man/headers.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/root/configuration/http_conn_man/headers.rst b/docs/root/configuration/http_conn_man/headers.rst index 9b4558584..ec5e9bd7a 100644 --- a/docs/root/configuration/http_conn_man/headers.rst +++ b/docs/root/configuration/http_conn_man/headers.rst @@ -69,7 +69,7 @@ It is a common case where a service wants to perform analytics based on the orig address. Per the lengthy discussion on :ref:`XFF `, this can get quite complicated, so Envoy simplifies this by setting *x-envoy-external-address* to the *trusted client address* (as defined in that XFF discussion) if the request is from -an external client. *x-envoy-external-address* is not set or overwritten for +an external client. *x-envoy-external-address* is not set or overwritten for internal requests. This header can be safely forwarded between internal services for analytics purposes without having to deal with the complexities of XFF. From 3c9f34062a3a5b3216d5eb441be95fc413c0b369 Mon Sep 17 00:00:00 2001 From: Brian Pane Date: Fri, 9 Feb 2018 19:23:52 +0000 Subject: [PATCH 3/3] fixes based on code review Signed-off-by: Brian Pane --- .../configuration/http_conn_man/headers.rst | 75 +++++++++++++------ 1 file changed, 53 insertions(+), 22 deletions(-) diff --git a/docs/root/configuration/http_conn_man/headers.rst b/docs/root/configuration/http_conn_man/headers.rst index ec5e9bd7a..cc1705777 100644 --- a/docs/root/configuration/http_conn_man/headers.rst +++ b/docs/root/configuration/http_conn_man/headers.rst @@ -68,9 +68,9 @@ x-envoy-external-address It is a common case where a service wants to perform analytics based on the origin client's IP address. Per the lengthy discussion on :ref:`XFF `, this can get quite complicated, so Envoy simplifies this by setting *x-envoy-external-address* -to the *trusted client address* (as defined in that XFF discussion) if the request is from -an external client. *x-envoy-external-address* is not set or overwritten for -internal requests. This header can be safely forwarded between internal services for analytics +to the :ref:`trusted client address ` +if the request is from an external client. *x-envoy-external-address* is not set or overwritten +for internal requests. This header can be safely forwarded between internal services for analytics purposes without having to deal with the complexities of XFF. .. _config_http_conn_man_headers_x-envoy-force-trace: @@ -158,6 +158,8 @@ operates in a transparent mode where it does not modify XFF. node (aka a front proxy), whereas it may need to be set to false when Envoy is used as an internal service node in a mesh deployment. +.. _config_http_conn_man_headers_x-forwarded-for_trusted_client_address: + The value of *use_remote_address* controls how Envoy determines the *trusted client address*. Given an HTTP request that has traveled through a series of zero or more proxies to reach Envoy, the trusted client address is the earliest source IP address that is known to be @@ -178,11 +180,13 @@ Envoy instance, the *xff_num_trusted_hops* configuration option can be used to t additional addresses from XFF: * If *use_remote_address* is false and *xff_num_trusted_hops* is set to a value *N* that is - greater than zero, the trusted client address is the *N+1* th address from the right end - of XFF. + greater than zero, the trusted client address is the (N+1)th address from the right end + of XFF. (If the XFF contains fewer than N+1 addresses, Envoy falls back to using the + immediate downstream connection's source address as trusted client address.) * If *use_remote_address* is true and *xff_num_trusted_hops* is set to a value *N* that is - greater than zero, the trusted client address is the *N* th address from the right end - of XFF. + greater than zero, the trusted client address is the Nth address from the right end + of XFF. (If the XFF contains fewer than N addresses, Envoy falls back to using the + immediate downstream connection's source address as trusted client address.) Envoy uses the trusted client address contents to determine whether a request originated externally or internally. This influences whether the @@ -195,13 +199,13 @@ Example 1: Envoy as edge proxy, without a trusted proxy in front of it Request details: | Downstream IP address = 192.0.2.5 - | XFF = "198.51.100.128, 198.51.100.10, 198.51.100.1" + | XFF = "203.0.113.128, 203.0.113.10, 203.0.113.1" Result: | Trusted client address = 192.0.2.5 (XFF is ignored) | X-Envoy-External-Address is set to 192.0.2.5 - | XFF is changed to "198.51.100.128, 198.51.100.10, 198.51.100.1, 192.0.2.5" - | Request type = external + | XFF is changed to "203.0.113.128, 203.0.113.10, 203.0.113.1, 192.0.2.5" + | X-Envoy-Internal is removed (if it was present in the incoming request) Example 2: Envoy as internal proxy, with the Envoy edge proxy from Example 1 in front of it Settings: @@ -210,13 +214,12 @@ Example 2: Envoy as internal proxy, with the Envoy edge proxy from Example 1 in Request details: | Downstream IP address = 10.11.12.13 (address of the Envoy edge proxy) - | XFF = "198.51.100.128, 198.51.100.10, 198.51.100.1, 192.0.2.5" + | XFF = "203.0.113.128, 203.0.113.10, 203.0.113.1, 192.0.2.5" Result: | Trusted client address = 192.0.2.5 (last address in XFF is trusted) | X-Envoy-External-Address is not modified - | Request type = internal - + | X-Envoy-Internal is removed (if it was present in the incoming request) Example 3: Envoy as edge proxy, with two trusted external proxies in front of it Settings: @@ -225,27 +228,55 @@ Example 3: Envoy as edge proxy, with two trusted external proxies in front of it Request details: | Downstream IP address = 192.0.2.5 - | XFF = "198.51.100.128, 198.51.100.10, 198.51.100.1" + | XFF = "203.0.113.128, 203.0.113.10, 203.0.113.1" Result: - | Trusted client address = 198.51.100.10 (2nd to last address in XFF is trusted) - | X-Envoy-External-Address is set to 198.51.100.10 - | XFF is changed to "198.51.100.128, 198.51.100.10, 198.51.100.1, 192.0.2.5" - | Request type = external + | Trusted client address = 203.0.113.10 (2nd to last address in XFF is trusted) + | X-Envoy-External-Address is set to 203.0.113.10 + | XFF is changed to "203.0.113.128, 203.0.113.10, 203.0.113.1, 192.0.2.5" + | X-Envoy-Internal is removed (if it was present in the incoming request) Example 4: Envoy as internal proxy, with the edge proxy from Example 3 in front of it Settings: - | use_remote_address = true + | use_remote_address = false | xff_num_trusted_hops = 2 Request details: | Downstream IP address = 10.11.12.13 (address of the Envoy edge proxy) - | XFF = "198.51.100.128, 198.51.100.10, 198.51.100.1, 192.0.2.5" + | XFF = "203.0.113.128, 203.0.113.10, 203.0.113.1, 192.0.2.5" Result: - | Trusted client address = 198.51.100.10 + | Trusted client address = 203.0.113.10 | X-Envoy-External-Address is not modified - | Request type = internal + | X-Envoy-Internal is removed (if it was present in the incoming request) + +Example 5: Envoy as an internal proxy, receiving a request from an internal client + Settings: + | use_remote_address = false + | xff_num_trusted_hops = 0 + + Request details: + | Downstream IP address = 10.20.30.40 (address of the internal client) + | XFF is not present + + Result: + | Trusted client address = 10.20.30.40 + | X-Envoy-External-Address remains unset + | X-Envoy-Internal is set to "true" + +Example 6: The internal Envoy from Example 5, receiving a request proxied by another Envoy + Settings: + | use_remote_address = false + | xff_num_trusted_hops = 0 + + Request details: + | Downstream IP address = 10.20.30.50 (address of the Envoy instance proxying to this one) + | XFF = "10.20.30.40" + + Result: + | Trusted client address = 10.20.30.40 + | X-Envoy-External-Address remains unset + | X-Envoy-Internal is set to "true" A few very important notes about XFF: