[pulsar-doc] Add document for Pulasr SNI routing with ATS #7421
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rdhabalia
added
the
doc
Jennifer88huang-zz
suggested changes
Jul 2, 2020
Contributor
Jennifer88huang-zz
left a comment
Jennifer88huang-zz
left a comment
There was a problem hiding this comment.
Thank you very much for your contributions. I have several tiny suggestions on this doc
- Use second person rather than first person(I, we).
- Use clear and concise sentences, rather than long-winded sentences.
- Use active voice.
- Use present tense, and try to avoid using "can".
You can learn those style guides at https://developers.google.com/style
|
|
||
| A proxy server is a go‑between or intermediary server that forwards requests from multiple clients to different servers across the Internet. The proxy server can act as a “traffic cop,” in both forward and reverse proxy scenarios, and adds various benefits in your system such as load balancing, performance, security, auto-scaling, etc.. There are already many proxy servers already available in the market which are fast, scalable and more importantly covers various essential security aspects that are needed by the large organization to securely share their confidential data over the network. Pulsar already provides proxy implementation which acts as a reverse proxy and creates a gateway in front of brokers. However, pulsar doesn’t provide support to use other proxies such as Apache traffic server (ATS), HAProxy, Nginx, Envoy those are more scalable and secured. Most of these proxy-servers support SNI ROUTING which can route traffic to a destination without having to terminate the SSL connection. Routing at layer 4 gives greater transparency because the outbound connection is determined by examining the destination address in the client TCP packets. | ||
|
|
||
| [PIP-60](https://github.com/apache/pulsar/wiki/PIP-60:-Support-Proxy-server-with-SNI-routing) eexplains SNI routing protocol and how Pulsar clients support SNI routing protocol to connect to brokers via proxy. This document explains how to setup ATS proxy and pulsar-client to enable SNI routing and connect pulsar-client to pulsar-broker via ATS proxy. |
Contributor
There was a problem hiding this comment.
Suggested change
| [PIP-60](https://github.com/apache/pulsar/wiki/PIP-60:-Support-Proxy-server-with-SNI-routing) eexplains SNI routing protocol and how Pulsar clients support SNI routing protocol to connect to brokers via proxy. This document explains how to setup ATS proxy and pulsar-client to enable SNI routing and connect pulsar-client to pulsar-broker via ATS proxy. | |
| [PIP-60](https://github.com/apache/pulsar/wiki/PIP-60:-Support-Proxy-server-with-SNI-routing) explains SNI routing protocol and how Pulsar clients support SNI routing protocol to connect to brokers via proxy. This document explains how to setup ATS proxy and pulsar-client to enable SNI routing and connect pulsar-client to pulsar-broker via ATS proxy. |
|
|
||
| # Pulsar Proxy with SNI routing | ||
|
|
||
| A proxy server is a go‑between or intermediary server that forwards requests from multiple clients to different servers across the Internet. The proxy server can act as a “traffic cop,” in both forward and reverse proxy scenarios, and adds various benefits in your system such as load balancing, performance, security, auto-scaling, etc.. There are already many proxy servers already available in the market which are fast, scalable and more importantly covers various essential security aspects that are needed by the large organization to securely share their confidential data over the network. Pulsar already provides proxy implementation which acts as a reverse proxy and creates a gateway in front of brokers. However, pulsar doesn’t provide support to use other proxies such as Apache traffic server (ATS), HAProxy, Nginx, Envoy those are more scalable and secured. Most of these proxy-servers support SNI ROUTING which can route traffic to a destination without having to terminate the SSL connection. Routing at layer 4 gives greater transparency because the outbound connection is determined by examining the destination address in the client TCP packets. |
Contributor
There was a problem hiding this comment.
Suggested change
| A proxy server is a go‑between or intermediary server that forwards requests from multiple clients to different servers across the Internet. The proxy server can act as a “traffic cop,” in both forward and reverse proxy scenarios, and adds various benefits in your system such as load balancing, performance, security, auto-scaling, etc.. There are already many proxy servers already available in the market which are fast, scalable and more importantly covers various essential security aspects that are needed by the large organization to securely share their confidential data over the network. Pulsar already provides proxy implementation which acts as a reverse proxy and creates a gateway in front of brokers. However, pulsar doesn’t provide support to use other proxies such as Apache traffic server (ATS), HAProxy, Nginx, Envoy those are more scalable and secured. Most of these proxy-servers support SNI ROUTING which can route traffic to a destination without having to terminate the SSL connection. Routing at layer 4 gives greater transparency because the outbound connection is determined by examining the destination address in the client TCP packets. | |
| A proxy server is a go‑between or intermediary server that forwards requests from multiple clients to different servers across the Internet. The proxy server acts as a “traffic cop,” in both forward and reverse proxy scenarios, and adds various benefits in your system such as load balancing, performance, security, auto-scaling, etc.. There are already many proxy servers already available in the market which are fast, scalable and more importantly covers various essential security aspects that are needed by the large organization to securely share their confidential data over the network. Pulsar provides proxy implementation which acts as a reverse proxy and creates a gateway in front of brokers. However, pulsar doesn’t provide support to use other proxies such as Apache traffic server (ATS), HAProxy, Nginx, Envoy those are more scalable and secured. Most of these proxy-servers support SNI ROUTING which can route traffic to a destination without having to terminate the SSL connection. Routing at layer 4 gives greater transparency because the outbound connection is determined by examining the destination address in the client TCP packets. |
| [PIP-60](https://github.com/apache/pulsar/wiki/PIP-60:-Support-Proxy-server-with-SNI-routing) eexplains SNI routing protocol and how Pulsar clients support SNI routing protocol to connect to brokers via proxy. This document explains how to setup ATS proxy and pulsar-client to enable SNI routing and connect pulsar-client to pulsar-broker via ATS proxy. | ||
|
|
||
| ## ATS-SNI Routing in Pulsar | ||
| [ATS supports layer-4 SNI routing](https://docs.trafficserver.apache.org/en/latest/admin-guide/layer-4-routing.en.html) with requirement that inbound connection must be TLS. Pulsar client supports SNI routing protocol and we can use ATS proxy in front of brokers, so pulsar client can connect to broker via ATS proxy. Pulsar also supports SNI routing for geo-replication, so brokers can connect to cross cluster brokers via ATS proxy. |
Contributor
There was a problem hiding this comment.
- I did not get the meaning of the first sentence.
- Try to use the second person when writing technical document, see https://developers.google.com/style/person
- Try not use modal verbs like "can" to avoid ambiguity. "Can" have two meanings:
a. be able to do sth
b. the possibility to do sth
Contributor
There was a problem hiding this comment.
what does "in front of brokers" mean?
|
|
||
| ### ATS Proxy setup for layer-4 SNI routing | ||
|
|
||
| This section explains ATS-setup to enable layer-4 SNI routing and use ATS proxy as a forward proxy where the pulsar broker runs behind the ATS proxy and client connects to pulsar-broker through this ATS proxy. |
Contributor
There was a problem hiding this comment.
It would be better if you can write clear and concise sentences, it's easy to read and understand.
Avoid writing long-winded sentences, see https://developers.google.com/style/tone#some-things-to-avoid-where-possible
|
|
||
|  | ||
|
|
||
| We have to configure two conf files into ATS proxy to support SNI routing. |
Contributor
There was a problem hiding this comment.
Suggested change
| We have to configure two conf files into ATS proxy to support SNI routing. | |
| To support SNI routing, you need to xxx. |
Huanli-Meng
reviewed
Jul 2, 2020
Contributor
Huanli-Meng
left a comment
Huanli-Meng
left a comment
There was a problem hiding this comment.
Leave my comments, PTAL.
And, please update the sidebar doc. only by doing that, the doc is shown on the Pulsar website after the PR is merged.
| sidebar_label: Proxy support with SNI routing | ||
| --- | ||
|
|
||
| # Pulsar Proxy with SNI routing |
Contributor
There was a problem hiding this comment.
Pulsar docs use level 2 heading level as started. Therefore, you should check through the document and change the heading levels one by one.
|
|
||
| # Pulsar Proxy with SNI routing | ||
|
|
||
| A proxy server is a go‑between or intermediary server that forwards requests from multiple clients to different servers across the Internet. The proxy server can act as a “traffic cop,” in both forward and reverse proxy scenarios, and adds various benefits in your system such as load balancing, performance, security, auto-scaling, etc.. There are already many proxy servers already available in the market which are fast, scalable and more importantly covers various essential security aspects that are needed by the large organization to securely share their confidential data over the network. Pulsar already provides proxy implementation which acts as a reverse proxy and creates a gateway in front of brokers. However, pulsar doesn’t provide support to use other proxies such as Apache traffic server (ATS), HAProxy, Nginx, Envoy those are more scalable and secured. Most of these proxy-servers support SNI ROUTING which can route traffic to a destination without having to terminate the SSL connection. Routing at layer 4 gives greater transparency because the outbound connection is determined by examining the destination address in the client TCP packets. |
Contributor
There was a problem hiding this comment.
Suggested change
| A proxy server is a go‑between or intermediary server that forwards requests from multiple clients to different servers across the Internet. The proxy server can act as a “traffic cop,” in both forward and reverse proxy scenarios, and adds various benefits in your system such as load balancing, performance, security, auto-scaling, etc.. There are already many proxy servers already available in the market which are fast, scalable and more importantly covers various essential security aspects that are needed by the large organization to securely share their confidential data over the network. Pulsar already provides proxy implementation which acts as a reverse proxy and creates a gateway in front of brokers. However, pulsar doesn’t provide support to use other proxies such as Apache traffic server (ATS), HAProxy, Nginx, Envoy those are more scalable and secured. Most of these proxy-servers support SNI ROUTING which can route traffic to a destination without having to terminate the SSL connection. Routing at layer 4 gives greater transparency because the outbound connection is determined by examining the destination address in the client TCP packets. | |
| A proxy server is an intermediary server that forwards requests from multiple clients to different servers across the Internet. The proxy server acts as a "traffic cop" in both forward and reverse proxy scenarios, and brings various benefits to your system such as load balancing, performance, security, auto-scaling, etc. There are already many proxy servers available in the market which are fast and scalable. More importantly, these proxy servers cover various essential security aspects that are needed by the large organizations to securely share their confidential data over the network. Pulsar already provides proxy implementation which acts as a reverse proxy and creates a gateway in front of brokers. However, pulsar does not supports to use other proxies such as Apache Traffic Server (ATS), HAProxy, Nginx, and Envoy. These proxies are more scalable and secured. Most of these proxy-servers support **SNI routing** which is used to route traffic to a destination without terminating the SSL connection. Routing at layer 4 provides greater transparency because the outbound connection is determined by examining the destination address in the client TCP packets. |
|
|
||
| A proxy server is a go‑between or intermediary server that forwards requests from multiple clients to different servers across the Internet. The proxy server can act as a “traffic cop,” in both forward and reverse proxy scenarios, and adds various benefits in your system such as load balancing, performance, security, auto-scaling, etc.. There are already many proxy servers already available in the market which are fast, scalable and more importantly covers various essential security aspects that are needed by the large organization to securely share their confidential data over the network. Pulsar already provides proxy implementation which acts as a reverse proxy and creates a gateway in front of brokers. However, pulsar doesn’t provide support to use other proxies such as Apache traffic server (ATS), HAProxy, Nginx, Envoy those are more scalable and secured. Most of these proxy-servers support SNI ROUTING which can route traffic to a destination without having to terminate the SSL connection. Routing at layer 4 gives greater transparency because the outbound connection is determined by examining the destination address in the client TCP packets. | ||
|
|
||
| [PIP-60](https://github.com/apache/pulsar/wiki/PIP-60:-Support-Proxy-server-with-SNI-routing) eexplains SNI routing protocol and how Pulsar clients support SNI routing protocol to connect to brokers via proxy. This document explains how to setup ATS proxy and pulsar-client to enable SNI routing and connect pulsar-client to pulsar-broker via ATS proxy. |
Contributor
There was a problem hiding this comment.
Suggested change
| [PIP-60](https://github.com/apache/pulsar/wiki/PIP-60:-Support-Proxy-server-with-SNI-routing) eexplains SNI routing protocol and how Pulsar clients support SNI routing protocol to connect to brokers via proxy. This document explains how to setup ATS proxy and pulsar-client to enable SNI routing and connect pulsar-client to pulsar-broker via ATS proxy. | |
| [PIP-60](https://github.com/apache/pulsar/wiki/PIP-60:-Support-Proxy-server-with-SNI-routing) explains SNI routing protocol and how Pulsar clients support SNI routing protocol to connect to brokers through the proxy. This document explains how to set up the ATS proxy and the Pulsar client to enable SNI routing and connect Pulsar client to the broker through the ATS proxy. |
| [PIP-60](https://github.com/apache/pulsar/wiki/PIP-60:-Support-Proxy-server-with-SNI-routing) eexplains SNI routing protocol and how Pulsar clients support SNI routing protocol to connect to brokers via proxy. This document explains how to setup ATS proxy and pulsar-client to enable SNI routing and connect pulsar-client to pulsar-broker via ATS proxy. | ||
|
|
||
| ## ATS-SNI Routing in Pulsar | ||
| [ATS supports layer-4 SNI routing](https://docs.trafficserver.apache.org/en/latest/admin-guide/layer-4-routing.en.html) with requirement that inbound connection must be TLS. Pulsar client supports SNI routing protocol and we can use ATS proxy in front of brokers, so pulsar client can connect to broker via ATS proxy. Pulsar also supports SNI routing for geo-replication, so brokers can connect to cross cluster brokers via ATS proxy. |
Contributor
There was a problem hiding this comment.
Suggested change
| [ATS supports layer-4 SNI routing](https://docs.trafficserver.apache.org/en/latest/admin-guide/layer-4-routing.en.html) with requirement that inbound connection must be TLS. Pulsar client supports SNI routing protocol and we can use ATS proxy in front of brokers, so pulsar client can connect to broker via ATS proxy. Pulsar also supports SNI routing for geo-replication, so brokers can connect to cross cluster brokers via ATS proxy. | |
| ATS supports [Layer 4 SNI routing](https://docs.trafficserver.apache.org/en/latest/admin-guide/layer-4-routing.en.html) with a requirement that inbound connection must be a TLS connection. The Pulsar client supports SNI routing protocol and we can use the ATS proxy in front of brokers. Therefore, the Pulsar client can connect to broker through the ATS proxy. Pulsar also supports SNI routing for geo-replication, so brokers can connect to cross cluster brokers through the ATS proxy. |
| 1. `records.conf`: | ||
| The [records.config fil](https://docs.trafficserver.apache.org/en/latest/admin-guide/files/records.config.en.html) (by default, located in `/usr/local/etc/trafficserver/`) is a list of configurable variables used by the Traffic Server and we have to update this file with tls port (`http.server_ports`) on which proxy can listen and proxy certs (`ssl.client.cert.path` and `ssl.client.cert.filename`) for secure tls tunneling. We also have to configure a range of server ports (`http.connect_ports`) that can be used for tunneling to pulsar-broker. If Pulsar brokers are listening on 4443 and 6651 then add brokers’ service port in http.connect_ports configuration. | ||
|
|
||
| ``` |
Contributor
There was a problem hiding this comment.
please note the indent of the code example, same comments on the below code examples.
|
|
||
| ### Pulsar geo-replication with SNI routing | ||
|
|
||
| We can also use ATS proxy for geo-replication. Pulsar broker has capability to connect to cross colo brokers for geo-replication using SNI routing. In order to enable SNI routing for cross cluster broker connection we have to configure SNI proxy URL to cluster metadata. If cluster metadata has SNI proxy url configured then broker will connect to cross cluster broker via proxy over SNI routing. |
Contributor
There was a problem hiding this comment.
Suggested change
| We can also use ATS proxy for geo-replication. Pulsar broker has capability to connect to cross colo brokers for geo-replication using SNI routing. In order to enable SNI routing for cross cluster broker connection we have to configure SNI proxy URL to cluster metadata. If cluster metadata has SNI proxy url configured then broker will connect to cross cluster broker via proxy over SNI routing. | |
| We can also use ATS proxy for geo-replication. The Pulsar broker can connect to cross colo brokers for geo-replication using SNI routing. In order to enable SNI routing for cross cluster broker connection, we have to configure SNI proxy URL to the cluster metadata. If the cluster metadata has SNI proxy URL configured, the broker connects to cross cluster broker through the proxy over SNI routing. |
what does the colo mean? a typo or a technical word?
|
|
||
| We can also use ATS proxy for geo-replication. Pulsar broker has capability to connect to cross colo brokers for geo-replication using SNI routing. In order to enable SNI routing for cross cluster broker connection we have to configure SNI proxy URL to cluster metadata. If cluster metadata has SNI proxy url configured then broker will connect to cross cluster broker via proxy over SNI routing. | ||
|
|
||
|  |
Contributor
There was a problem hiding this comment.
In general, all images related about pulsar docs are placed in the asset folder. could you please put the original image to that folder and update the local path for the image?
|
|
||
|  | ||
|
|
||
| In this example, we have a pulsar cluster deployed into two separate regions, us-west and us-east. We have also configured ATS proxy in both the regions, and brokers in each region running behind this ATS proxy. Now, we will configure cluster metadata for both the clusters, so brokers in one cluster will use SNI routing and connect to brokers in other clusters via ATS proxy. |
Contributor
There was a problem hiding this comment.
Suggested change
| In this example, we have a pulsar cluster deployed into two separate regions, us-west and us-east. We have also configured ATS proxy in both the regions, and brokers in each region running behind this ATS proxy. Now, we will configure cluster metadata for both the clusters, so brokers in one cluster will use SNI routing and connect to brokers in other clusters via ATS proxy. | |
| In this example, we have a pulsar cluster deployed into two separate regions, us-west and us-east. We have also configured ATS proxy in both the regions, and brokers in each region run behind this ATS proxy. Now, we configure the cluster metadata for both the clusters, so brokers in one cluster can use SNI routing and connect to brokers in other clusters through the ATS proxy. |
|
|
||
| In this example, we have a pulsar cluster deployed into two separate regions, us-west and us-east. We have also configured ATS proxy in both the regions, and brokers in each region running behind this ATS proxy. Now, we will configure cluster metadata for both the clusters, so brokers in one cluster will use SNI routing and connect to brokers in other clusters via ATS proxy. | ||
|
|
||
| (a) Configure cluster metadata for us-east with us-east broker service url and us-east ATS proxy url with SNI proxy-protocol. |
Contributor
There was a problem hiding this comment.
If this is an ordered task, use "1,2,3..." instead of "a, b,c..."
Suggested change
| (a) Configure cluster metadata for us-east with us-east broker service url and us-east ATS proxy url with SNI proxy-protocol. | |
| (a) Configure the cluster metadata for us-east with us-east broker service URL and us-east ATS proxy URL with SNI proxy-protocol. |
| --proxy-url pulsar+ssl://east-ats-proxy:443 | ||
| ``` | ||
|
|
||
| (b) Configure cluster metadata for us-east with us-east broker service url and us-east ATS proxy url with SNI proxy-protocol. |
Contributor
There was a problem hiding this comment.
Suggested change
| (b) Configure cluster metadata for us-east with us-east broker service url and us-east ATS proxy url with SNI proxy-protocol. | |
| (b) Configure the cluster metadata for us-east with us-east broker service URL and us-east ATS proxy URL with SNI proxy-protocol. |
us-west?
Contributor
|
@jennifer88huang , PIP-60 is a feature released in Pulsar 2.6.0, please cherry-pick the doc to the Release 2.6.0 and 2.6.1. Thanks. |
Contributor
Done. Thanks for your reminder. |
Contributor
Author
|
@Huanli-Meng for useful feedback. I addressed all the comments. |
Anonymitaet
reviewed
Jul 3, 2020
Co-authored-by: Yu Liu <50226895+Anonymitaet@users.noreply.github.com>
Co-authored-by: Yu Liu <50226895+Anonymitaet@users.noreply.github.com>
Co-authored-by: Yu Liu <50226895+Anonymitaet@users.noreply.github.com>
Co-authored-by: Yu Liu <50226895+Anonymitaet@users.noreply.github.com>
Co-authored-by: Yu Liu <50226895+Anonymitaet@users.noreply.github.com>
Contributor
Author
|
addressed the comments. PTAL again. |
jiazhai
approved these changes
Jul 7, 2020
Jennifer88huang-zz
approved these changes
Jul 7, 2020
Contributor
|
@rdhabalia Thank you very much for your contribution. |
wolfstudy
pushed a commit
that referenced
this pull request
Jul 29, 2020
* [pulsar-doc] Add document for Pulasr SNI routing with ATS * address comments * add image * Update site2/docs/concepts-proxy-sni-routing.md Co-authored-by: Yu Liu <50226895+Anonymitaet@users.noreply.github.com> * Update site2/docs/concepts-proxy-sni-routing.md Co-authored-by: Yu Liu <50226895+Anonymitaet@users.noreply.github.com> * Update site2/docs/concepts-proxy-sni-routing.md Co-authored-by: Yu Liu <50226895+Anonymitaet@users.noreply.github.com> * Update site2/docs/concepts-proxy-sni-routing.md Co-authored-by: Yu Liu <50226895+Anonymitaet@users.noreply.github.com> * Update site2/docs/concepts-proxy-sni-routing.md Co-authored-by: Yu Liu <50226895+Anonymitaet@users.noreply.github.com> Co-authored-by: Yu Liu <50226895+Anonymitaet@users.noreply.github.com> (cherry picked from commit ee1b810)
huangdx0726
pushed a commit
to huangdx0726/pulsar
that referenced
this pull request
Aug 24, 2020
* [pulsar-doc] Add document for Pulasr SNI routing with ATS * address comments * add image * Update site2/docs/concepts-proxy-sni-routing.md Co-authored-by: Yu Liu <50226895+Anonymitaet@users.noreply.github.com> * Update site2/docs/concepts-proxy-sni-routing.md Co-authored-by: Yu Liu <50226895+Anonymitaet@users.noreply.github.com> * Update site2/docs/concepts-proxy-sni-routing.md Co-authored-by: Yu Liu <50226895+Anonymitaet@users.noreply.github.com> * Update site2/docs/concepts-proxy-sni-routing.md Co-authored-by: Yu Liu <50226895+Anonymitaet@users.noreply.github.com> * Update site2/docs/concepts-proxy-sni-routing.md Co-authored-by: Yu Liu <50226895+Anonymitaet@users.noreply.github.com> Co-authored-by: Yu Liu <50226895+Anonymitaet@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Add documentation for PIP-60 with instrunction to setup proxy and configuration.