From fa9fc379b62faff083b07a69dd87cc8603fbaba1 Mon Sep 17 00:00:00 2001 From: Amin Jamali Date: Fri, 31 May 2024 14:32:07 +0000 Subject: [PATCH 1/3] Update docs --- ...pment_guide.md => 01-development-guide.md} | 21 ++++++++++++++++- ...iguration.md => 02-nats-configurations.md} | 23 +++++++++++++++++-- 2 files changed, 41 insertions(+), 3 deletions(-) rename docs/{gorouter_development_guide.md => 01-development-guide.md} (92%) rename docs/{nats_configuration.md => 02-nats-configurations.md} (84%) diff --git a/docs/gorouter_development_guide.md b/docs/01-development-guide.md similarity index 92% rename from docs/gorouter_development_guide.md rename to docs/01-development-guide.md index b5eff4db4..906297525 100644 --- a/docs/gorouter_development_guide.md +++ b/docs/01-development-guide.md @@ -1,4 +1,23 @@ -# Development Guide for Gorouter +--- +title: Development Guide +expires_at: never +tags: [routing-release,gorouter] +--- + + + +* [Development Guide](#development-guide) + * [Golang TCP Networking Basics](#golang-tcp-networking-basics) + * [General Gorouter Architecture](#general-gorouter-architecture) + * [Ifrit processes](#ifrit-processes) + * [What does Gorouter do?](#what-does-gorouter-do) + * [What are all these extra components in the Gorouter request flow?](#what-are-all-these-extra-components-in-the-gorouter-request-flow) + * [Negroni Handlers](#negroni-handlers) + * [ProxyRoundTripper](#proxyroundtripper) + + + +# Development Guide Recommended reading before diving into Gorouter code: - [Hypertext Transfer Protocol](https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol#Message_format) diff --git a/docs/nats_configuration.md b/docs/02-nats-configurations.md similarity index 84% rename from docs/nats_configuration.md rename to docs/02-nats-configurations.md index 5cac161f7..a42f8216b 100644 --- a/docs/nats_configuration.md +++ b/docs/02-nats-configurations.md @@ -1,3 +1,20 @@ +--- +title: NATS Configuration +expires_at: never +tags: [routing-release,gorouter] +--- + + + +* [NATS Configuration](#nats-configuration) + * [Consistency over Availability:](#consistency-over-availability) + * [Relation between DropletStaleThreshold, NATs PingInterval and MinimumRegistrationInterval](#relation-between-dropletstalethreshold-nats-pinginterval-and-minimumregistrationinterval) + * [Definitions:](#definitions) + + + +# NATS Configuration + ## Consistency over Availability: In the context of Cloud Foundry, when an application instance crashes or is stopped as a result of the app being stopped or scaled down, the allocated IP and port are released to the pool. The same IP and port may then be assigned to a new instance of another application, as when a new app is started, scaled up, or a crashed instance is recreated. Under normal operation, each of these events will result in a change to Gorouter's routing table. Updates to the routing table depend on a message being sent by a client to NATS (e.g. Route Emitter is responsible for sending changes to routing data for apps running on Diego), and on Gorouter fetching the message from NATS. @@ -8,7 +25,9 @@ To prevent stale routes, Gorouter is by default optimized for consistency over a If an operator would prefer to favor availability over consistency, the configuration property [`suspend_pruning_if_nats_unavailable`](../config/config.go#L203) can be used to ignore route TTL and prevent pruning in the event that Gorouter cannot connect to NATS. This config option will also set max reconnect in the NATS client to -1 (no limit) which prevents Gorouter from crashing and losing its in-memory routing table. This configuration option is set to false by default. ->**Warning**: There is a significant probability of routing to an incorrect backend endpoint in the case of port re-use. Suspending route pruning should be used with caution. +> [!WARNING] +> +> There is a significant probability of routing to an incorrect backend endpoint in the case of port re-use. Suspending route pruning should be used with caution. ## Relation between DropletStaleThreshold, NATs PingInterval and MinimumRegistrationInterval @@ -44,4 +63,4 @@ minimumRegistrationInterval) - (NATS Timeout * NumberOfNatsServers))/3 Currently we do not allow the operator to set the value for DropletStaleThreshold and StartResponseDelayInterval, hence there is no real need for the above equation to calculate the ping interval yet. After long - consideration of different scenarios we have decided configure interval with value [`20` seconds](https://github.com/cloudfoundry/gorouter/blob/master/config/config.go#L199). + consideration of different scenarios we have decided configure interval with value [`20` seconds](https://github.com/cloudfoundry/gorouter/blob/main/config/config.go#L199). From cbf48cbe07cd0d0cd4fc0a24aa5d09df1ae46ce4 Mon Sep 17 00:00:00 2001 From: Amin Jamali Date: Fri, 31 May 2024 18:44:24 +0000 Subject: [PATCH 2/3] Remove ADR since we are not actively maintaining it --- .../0001-record-architecture-decisions.md | 19 -------- .../0002-change-tls-route-pruning-behavior.md | 43 ------------------- 2 files changed, 62 deletions(-) delete mode 100644 docs/decisions/0001-record-architecture-decisions.md delete mode 100644 docs/decisions/0002-change-tls-route-pruning-behavior.md diff --git a/docs/decisions/0001-record-architecture-decisions.md b/docs/decisions/0001-record-architecture-decisions.md deleted file mode 100644 index e91ba6ba9..000000000 --- a/docs/decisions/0001-record-architecture-decisions.md +++ /dev/null @@ -1,19 +0,0 @@ -# 1. Record architecture decisions - -Date: 2018-09-12 - -## Status - -Accepted - -## Context - -We need to record the architectural decisions made on this project. - -## Decision - -We will use Architecture Decision Records, as [described by Michael Nygard](http://thinkrelevance.com/blog/2011/11/15/documenting-architecture-decisions). - -## Consequences - -See Michael Nygard's article, linked above. For a lightweight ADR toolset, see Nat Pryce's [adr-tools](https://github.com/npryce/adr-tools). diff --git a/docs/decisions/0002-change-tls-route-pruning-behavior.md b/docs/decisions/0002-change-tls-route-pruning-behavior.md deleted file mode 100644 index 3ab412fb6..000000000 --- a/docs/decisions/0002-change-tls-route-pruning-behavior.md +++ /dev/null @@ -1,43 +0,0 @@ -# 2. Change TLS endpoint pruning behavior - -Date: 2018-09-13 - -## Status - -Accepted - -## Context - -This is related to story [#158847588](https://www.pivotaltracker.com/story/show/158847588) - -Prior to the story above, when route-integrity was turned on (soon to be the -default) we did not prune routes that received most [retriable]() errors. The -code ensures that there are only two types of errors that [result in a -prune](https://github.com/cloudfoundry/gorouter/blob/b63e6fb16c2a422ec5108a19debc9adb81f2d1dd/route/pool.go#L369-L372): - -[Hostname Mismatch and Attempted TLS with Non TLS -Backend](https://github.com/cloudfoundry/gorouter/blob/b63e6fb16c2a422ec5108a19debc9adb81f2d1dd/proxy/fails/classifier_group.go#L17-L20) - -The prune operation should have little user impact - the route will get added -again the next time the route-registrar runs if the application is still -running. - -## Decision - -We will prune any TLS route that has had a failure immediately. Consequently, -we are immediately pruning on more errors, such that the final list includes the -following errors: AttemptedTLSWithNonTLSBackend, Dial, RemoteFailedCertCheck, -RemoteHandshakeFailure, HostnameMismatch, UntrustedCert - -We will also add logging to the cases when an endpoint is pruned. - -## Consequences - -If a developer's app is flapping, they may start to see a new pattern: a 502 -followed by a series of 404s (until the route is re-emitted). - -Logging will be introduced into the route pool pruning behavior, giving -operators a view into whether a prune or fail has occurred and what error caused -it. - -[As of this date, endpoint retry logic will not change](https://docs.cloudfoundry.org/concepts/http-routing.html#transparent) From f1cee7fed03bd54e6c7db34ad46aeaac3dd1879b Mon Sep 17 00:00:00 2001 From: App Platform Runtime Working Group CI Bot Date: Mon, 10 Jun 2024 13:54:45 +0000 Subject: [PATCH 3/3] Sync README.md --- README.md | 880 ++---------------------------------------------------- 1 file changed, 29 insertions(+), 851 deletions(-) diff --git a/README.md b/README.md index 9bfd0f36e..062750320 100644 --- a/README.md +++ b/README.md @@ -1,862 +1,40 @@ -[![Go Report Card](https://goreportcard.com/badge/github.com/cloudfoundry/gorouter)](https://goreportcard.com/report/github.com/cloudfoundry/gorouter) +# gorouter -# Gorouter -This repository contains the source code for the Cloud Foundry L7 HTTP router. -Gorouter is deployed by default with Cloud Foundry -([cf-deployment](https://github.com/cloudfoundry/cf-deployment)) which includes -[routing-release](https://github.com/cloudfoundry/routing-release) as submodule. +[![Go Report +Card](https://goreportcard.com/badge/code.cloudfoundry.org/gorouter)](https://goreportcard.com/report/code.cloudfoundry.org/gorouter) +[![Go +Reference](https://pkg.go.dev/badge/code.cloudfoundry.org/gorouter.svg)](https://pkg.go.dev/code.cloudfoundry.org/gorouter) -> **Note**: This repository should be imported as -> `code.cloudfoundry.org/gorouter`. - -## Reporting issues and requesting features - -Please report all issues and feature requests in [cloudfoundry/routing-release](https://github.com/cloudfoundry/routing-release). - -## Contributing - -Please read the [contributors' -guide](https://github.com/cloudfoundry/gorouter/blob/main/CONTRIBUTING.md) and our [Development Guide for -Gorouter](https://github.com/cloudfoundry/gorouter/blob/main/docs/gorouter_development_guide.md). - -### Setup - -Gorouter dependencies are managed with -[routing-release](https://github.com/cloudfoundry/routing-release#). Do not -clone the gorouter repo directly; instead, follow instructions at -https://github.com/cloudfoundry/routing-release#get-the-code (summarized below). - -```bash -git clone https://github.com/cloudfoundry/routing-release -cd routing-release -./scripts/update -cd src/code.cloudfoundry.org/gorouter -``` - -### Running Tests - -Tests in this repo cannot be run on their own, only as part of Routing Release. - -Follow the instructions for [running tests in -docker](https://github.com/cloudfoundry/routing-release#in-a-docker-container) -in the routing release readme. - -### Building -Building creates an executable in the gorouter/ dir: - -```bash -go build -``` - -### Installing -Installing creates an executable in the $GOPATH/bin dir: - -```bash -go install -``` - -### Start - -```bash -# Start NATS server in daemon mode -git clone https://github.com/nats-io/nats-server -cd nats-server/ -go install -nats-server & - -# Start gorouter -gorouter -``` - -## Performance - -See [Routing Release 0.144.0 Release Notes] -(https://github.com/cloudfoundry/routing-release/releases/tag/0.144.0) - -## Dynamic Routing Table - -Gorouters routing table is updated dynamically via the NATS message bus. NATS -can be deployed via BOSH with -([cf-deployment](https://github.com/cloudfoundry/cf-deployment)) or standalone -using [nats-release](https://github.com/cloudfoundry/nats-release). - -To add or remove a record from the routing table, a NATS client must send -register or unregister messages. Records in the routing table have a maximum TTL -of 120 seconds, so clients must heartbeat registration messages periodically; we -recommend every 20s. [Route -Registrar](https://github.com/cloudfoundry/route-registrar) is a BOSH job that -comes with [Routing Release](https://github.com/cloudfoundry/routing-release) -that automates this process. - -When deployed with Cloud Foundry, registration of routes for apps pushed to CF -occurs automatically without user involvement. For details, see [Routes and -Domains] -(https://docs.cloudfoundry.org/devguide/deploy-apps/routes-domains.html). - -### Registering Routes via NATS - -When the gorouter starts, it sends a `router.start` message to NATS. This -message contains an interval that other components should then send -`router.register` on, `minimumRegisterIntervalInSeconds`. It is recommended that -clients should send `router.register` messages on this interval. This -`minimumRegisterIntervalInSeconds` value is configured through the -`start_response_delay_interval` configuration property. Gorouter will prune -routes that it considers to be stale based upon a separate "staleness" value, -`droplet_stale_threshold`, which defaults to 120 seconds. Gorouter will check if -routes have become stale on an interval defined by -`prune_stale_droplets_interval`, which defaults to 30 seconds. All of these -values are represented in seconds and will always be integers. - -The format of the `router.start` message is as follows: - -```json -{ - "id": "some-router-id", - "hosts": ["1.2.3.4"], - "minimumRegisterIntervalInSeconds": 20, - "prunteThresholdInSeconds": 120 -} -``` - -After a `router.start` message is received by a client, the client should send -`router.register` messages. This ensures that the new router can update its -routing table and synchronize with existing routers. - -If a component comes online after the router, it must make a NATS request called -`router.greet` in order to determine the interval. The response to this message -will be the same format as `router.start`. - -The format of the `router.register` message is as follows: - -```json -{ - "host": "127.0.0.1", - "port": 4567, - "tls_port": 1234, - "protocol": "http1", - "uris": [ - "my_first_url.localhost.routing.cf-app.com", - "my_second_url.localhost.routing.cf-app.com" - ], - "tags": { - "another_key": "another_value", - "some_key": "some_value" - }, - "app": "some_app_guid", - "stale_threshold_in_seconds": 120, - "private_instance_id": "some_app_instance_id", - "isolation_segment": "some_iso_seg_name", - "server_cert_domain_san": "some_subject_alternative_name" -} -``` - -`stale_threshold_in_seconds` is the custom staleness threshold for the route -being registered. If this value is not sent, it will default to the router's -default staleness threshold. - -`app` is a unique identifier for an application that the endpoint is registered -for. This value will be included in router access logs with the label `app_id`, -as well as being sent with requests to the endpoint in an HTTP header -`X-CF-ApplicationId`. - -`private_instance_id` is a unique identifier for an instance associated with the -app identified by the `app` field. Gorouter includes an HTTP header -`X-CF-InstanceId` set to this value with requests to the registered endpoint. - -`isolation_segment` determines which routers will register route. Only Gorouters -configured with the matching isolation segment will register the route. If a -value is not provided, the route will be registered only by Gorouters set to the -`all` or `shared-and-segments` router table sharding modes. Refer to the job -properties for [Gorouter] -(https://github.com/cloudfoundry/routing-release/blob/develop/jobs/gorouter/spec) -for more information. - -`tls_port` is the port that Gorouter will use to attempt TLS connections with -the registered backends. Supported only when `router.backend.enable_tls: true` -is configured in the manifest. `router.ca_certs` may be optionally configured -with a CA, for backends certificates signed by custom CAs. For mutual -authentication with backends, `router.backends.tls_pem` may be optionally -provided. When `router.backend.enable_tls: true`, Gorouter will prefer -`tls_port` over `port` if present in the NATS message. Otherwise, `port` will be -preferred, and messages with only `tls_port` will be rejected and an error -message logged. +This repository contains the source code for the Cloud Foundry L7 HTTP +router. Gorouter is deployed by default with Cloud Foundry +([cf-deployment](https://github.com/cloudfoundry/cf-deployment)) which +includes +[routing-release](https://github.com/cloudfoundry/routing-release) as +submodule. -`server_cert_domain_san` (required when `tls_port` is present) Indicates a -string that Gorouter will look for in a Subject Alternative Name (SAN) of the -TLS certificate hosted by the backend to validate instance identity. When the -value of `server_cert_domain_san` does not match a SAN in the server -certificate, Gorouter will prune the backend and retry another backend for the -route if one exists, or return a 503 if it cannot validate the identity of any -backend in three tries. - -Additionally, if the `host` and `tls_port` pair matches an already registered -`host` and `port` pair, the previously registered route will be overwritten and -Gorouter will now attempt TLS connections with the `host` and `tls_port` pair. -The same is also true if the `host` and `port` pair matches an already -registered `host` and `tls_port` pair, except Gorouter will no longer attempt -TLS connections with the backend. - -Such a message can be sent to both the `router.register` subject to register -URIs, and to the `router.unregister` subject to unregister URIs, respectively. - -### Deleting a Route - -Routes can be deleted with the `router.unregister` nats message. The format of -the `router.unregister` message the same as the `router.register` message, but -most information is ignored. Any route that matches the `host`, `port` and -`uris` fields will be deleted. - -### Example - -Create a simple app -```bash -$ nohup ruby -rsinatra -e 'get("/") { "Hello!" }' & -``` - -Send a register message -```bash -$ nats-pub 'router.register' '{"host":"127.0.0.1","port":4567,"uris":["my_first_url.localhost.routing.cf-app.com","my_second_url.localhost.routing.cf-app.com"],"tags":{"another_key":"another_value","some_key":"some_value"}}' - -Published [router.register] : '{"host":"127.0.0.1","port":4567,"uris":["my_first_url.localhost.routing.cf-app.com","my_second_url.localhost.routing.cf-app.com"],"tags":{"another_key":"another_value","some_key":"some_value"}}' -``` - -See that it works! -```bash -$ curl my_first_url.localhost.routing.cf-app.com:8081 -Hello! -``` - -Unregister the route -```bash -$ nats-pub 'router.unregister' '{"host":"127.0.0.1","port":4567,"tls_port":1234,"uris":["my_first_url.localhost.routing.cf-app.com","my_second_url.localhost.routing.cf-app.com"]}' - -Published [router.unregister] : '{"host":"127.0.0.1","port":4567,"tls_port":1234,"uris":["my_first_url.localhost.routing.cf-app.com","my_second_url.localhost.routing.cf-app.com"]}' -``` - -See that the route is gone - -```bash -$ curl my_first_url.localhost.routing.cf-app.com:8081 -404 Not Found: Requested route ('my_first_url.localhost.routing.cf-app.com') does not exist. -``` - -If `router.backends.enable_tls` has been set to true, `tls_port` will be used as -the definitive port when unregistering a route if present, otherwise `port` will -be used. If `router.backends.enable_tls` is set to false, `port` will be -preferred and any requests with only `tls_port` will be rejected and an error -logged to the gorouter logs. - -Note that if `router.backends.enable_tls` is true and `host` and `tls_port` -happens to match a registered `host` and `port` pair, this `host` and `port` -pair will be unregistered. The reverse is also true. - -> **Note:** In order to use `nats-pub` to register a route, you must install the -> [gem](https://github.com/nats-io/ruby-nats) on a Cloud Foundry VM. It's -> easiest on a VM that has ruby as a package, such as the API VM. Find the ruby -> installed in `/var/vcap/packages`, export your PATH variable to include the bin -> directory, and then run `gem install nats`. Find the nats login info from your -> gorouter config and use it to connect to the nats cluster. - -## Healthchecking from a Load Balancer - -To scale Gorouter horizontally for high-availability or throughput capacity, you -must deploy it behind a highly-available load balancer (F5, AWS ELB, etc). - -Gorouter has a health endpoint `/health` on port 8443 (with TLS) and -on 8080 (without TLS) that returns a 200 OK which indicates the Gorouter instance -is healthy; any other response indicates unhealthy. These port can be configured -via the `router.status.port` and `router.status.tls.port` properties in the BOSH -deployment manifest or via the `status.port` and `status.tls.port` properties -under `/var/vcap/jobs/gorouter/config/gorouter.yml` - - -```bash -$ curl -v http://10.0.32.15:8080/health -* Trying 10.0.32.15.. -* Connected to 10.0.32.15 (10.0.32.15) port 8080 (#0) -> GET /health HTTP/1.1 -> Host: 10.0.32.15:8080 -> User-Agent: curl/7.43.0 -> Accept: */* +> \[!NOTE\] > -< HTTP/1.1 200 OK -< Cache-Control: private, max-age=0 -< Expires: 0 -< Date: Thu, 22 Sep 2016 00:13:54 GMT -< Content-Length: 3 -< Content-Type: text/plain; charset=utf-8 -< -ok -* Connection #0 to host 10.0.32.15 left intact -``` - -**DEPRECATED:** Your load balancer can be configured to send an HTTP -healthcheck on port 80 with the `User-Agent` HTTP header set to -`HTTP-Monitor/1.1`. A 200 response indicates the Gorouter instance is healthy; -any other response indicates unhealthy. Gorouter can be configured to accept -alternate values for the User Agent header using the `healthcheck_user_agent` -configuration property; as an example, AWS ELBS send `User-Agent: -ELB-HealthChecker/1.0`. - -```bash -$ curl -v -A "HTTP-Monitor/1.1" "http://10.0.32.15" -* Rebuilt URL to: http://10.0.32.15/ -* Hostname was NOT found in DNS cache -* Trying 10.0.32.15... -* Connected to 10.0.32.15 (10.0.32.15) port 80 (#0) -> GET / HTTP/1.1 -> User-Agent: HTTP-Monitor/1.1 -> Host: 10.0.32.15 -> Accept: */* -> -< HTTP/1.1 200 OK -< Cache-Control: private, max-age=0 -< Expires: 0 -< X-Vcap-Request-Id: 04ad84c6-43dd-4d20-7818-7c47595d9442 -< Date: Thu, 07 Jan 2016 22:30:02 GMT -< Content-Length: 3 -< Content-Type: text/plain; charset=utf-8 -< -ok -* Connection #0 to host 10.0.32.15 left intact -``` - -**DEPRECATED:** The `/healthz` endpoint is now an alias for the `/health` endpoint -to ensure backward compatibility. - -## Instrumentation - -### The Routing Table - -The `/routes` endpoint returns the entire routing table as JSON. This endpoint -requires basic authentication and is served on port `8082`. This port is configurable -via the `router.status.routes.port` property in the BOSH deployment manifest, or via -the `status.routes.port` property in `/var/vcap/jobs/gorouter/config/gorouter.yml`. -Route information is available via localhost only. - -Each route has an associated array of host:port entries, formatted as follows: - -```bash -$ curl "http://someuser:somepass@localhost:8080/routes" -{ - "api.catwoman.cf-app.com": [ - { - "address": "10.244.0.138:9022", - "ttl": 0, - "tags": { - "component": "CloudController" - } - } - ], - "dora-dora.catwoman.cf-app.com": [ - { - "address": "10.244.16.4:60035", - "ttl": 0, - "tags": { - "component": "route-emitter" - } - }, - { - "address": "10.244.16.4:60060", - "ttl": 0, - "tags": { - "component": "route-emitter" - } - } - ] -} -``` -**NOTE:** This endpoint is internal only, and may change in the future. To safeguard -against changes, rely on the `/var/vcap/jobs/gorouter/bin/retrieve-local-routes` script -to get this information. - -Because of the nature of the data present in `/varz` and `/routes`, they require -http basic authentication credentials. These credentials can be found the BOSH -manifest for cf-deployment under the `router` job: - -```bash -properties: - router: - status: - password: zed292_bevesselled - port: - user: paronymy61-polaric -``` - -If `router.status.user` is not set in the manifest, the default is -`router-status` as can be seen from [the job -spec](https://github.com/cloudfoundry/routing-release/blob/develop/jobs/gorouter/spec). - -Or on the Gorouter VM under `/var/vcap/jobs/gorouter/config/gorouter.yml`: - -```yaml -status: - port: 8080 - user: some_user - pass: some_password -``` - -### Metrics - -The `/varz` endpoint provides status and metrics. This endpoint requires basic -authentication. - -
- Metrics response (click to expand) - -```bash -$ curl "http://someuser:somepass@10.0.32.15:8080/varz" -{ - "bad_gateways": 0, - "bad_requests": 20, - "cpu": 0, - "credentials": [ - "user", - "pass" - ], - "droplets": 26, - "host": "10.0.32.15:8080", - "index": 0, - "latency": { - "50": 0.001418144, - "75": 0.00180639025, - "90": 0.0070607187, - "95": 0.009561058849999996, - "99": 0.01523927838000001, - "samples": 1, - "value": 5e-07 - }, - "log_counts": { - "info": 9, - "warn": 40 - }, - "mem": 19672, - "ms_since_last_registry_update": 1547, - "num_cores": 2, - "rate": [ - 1.1361328993362565, - 1.1344545494448148, - 1.1365784133171992 - ], - "requests": 13832, - "requests_per_sec": 1.1361328993362565, - "responses_2xx": 13814, - "responses_3xx": 0, - "responses_4xx": 9, - "responses_5xx": 0, - "responses_xxx": 0, - "start": "2016-01-07 19:04:40 +0000", - "tags": { - "component": { - "CloudController": { - "latency": { - "50": 0.009015199, - "75": 0.0107408015, - "90": 0.015104917100000005, - "95": 0.01916497394999999, - "99": 0.034486261410000024, - "samples": 1, - "value": 5e-07 - }, - "rate": [ - 0.13613289933245148, - 0.13433569936308343, - 0.13565885617276216 - ], - "requests": 1686, - "responses_2xx": 1684, - "responses_3xx": 0, - "responses_4xx": 2, - "responses_5xx": 0, - "responses_xxx": 0 - }, - "HM9K": { - "latency": { - "50": 0.0033354, - "75": 0.00751815875, - "90": 0.011916812100000005, - "95": 0.013760064, - "99": 0.013760064, - "samples": 1, - "value": 5e-07 - }, - "rate": [ - 1.6850238803894876e-12, - 5.816129919395257e-05, - 0.00045864309255845694 - ], - "requests": 12, - "responses_2xx": 6, - "responses_3xx": 0, - "responses_4xx": 6, - "responses_5xx": 0, - "responses_xxx": 0 - }, - "dea-0": { - "latency": { - "50": 0.001354994, - "75": 0.001642107, - "90": 0.0020699939000000003, - "95": 0.0025553900499999996, - "99": 0.003677146940000006, - "samples": 1, - "value": 5e-07 - }, - "rate": [ - 1.0000000000000013, - 1.0000000002571303, - 0.9999994853579043 - ], - "requests": 12103, - "responses_2xx": 12103, - "responses_3xx": 0, - "responses_4xx": 0, - "responses_5xx": 0, - "responses_xxx": 0 - }, - "uaa": { - "latency": { - "50": 0.038288465, - "75": 0.245610809, - "90": 0.2877324668, - "95": 0.311816554, - "99": 0.311816554, - "samples": 1, - "value": 5e-07 - }, - "rate": [ - 8.425119401947438e-13, - 2.9080649596976205e-05, - 0.00022931374141467497 - ], - "requests": 17, - "responses_2xx": 17, - "responses_3xx": 0, - "responses_4xx": 0, - "responses_5xx": 0, - "responses_xxx": 0 - } - } - }, - "top10_app_requests": [ - { - "application_id": "063f95f9-492c-456f-b569-737f69c04899", - "rpm": 60, - "rps": 1 - } - ], - "type": "Router", - "uptime": "0d:3h:22m:31s", - "urls": 21, - "uuid": "0-c7fd7d76-f8d8-46b7-7a1c-7a59bcf7e286" -} -``` -
- -### Profiling the Server - -The Gorouter runs the -[debugserver](https://github.com/cloudfoundry/debugserver), which is a wrapper -around the go pprof tool. In order to generate this profile, do the following: - -```bash -# Establish a SSH tunnel to your server (not necessary if you can connect directly) -ssh -L localhost:8080:[INTERNAL_SERVER_IP]:17001 vcap@[BOSH_DIRECTOR] -# Run the profile tool. -go tool pprof http://localhost:8080/debug/pprof/profile -``` - -## Load Balancing - -The Gorouter is, in simple terms, a reverse proxy that load balances between -many backend instances. The default load balancing algorithm that Gorouter will -use is a simple **round-robin** strategy. Gorouter will retry a request if the -chosen backend does not accept the TCP connection. - -### Round-Robin -Default load balancing algorithm that gorouter will use or may be explicitly set -in **gorouter.yml** `yaml default_balancing_algorithm: round-robin` - -### Least-Connection -The Gorouter also supports least connection based routing and this can be -enabled in **gorouter.yml** - -```yaml -default_balancing_algorithm: least-connection -``` - -Least connection based load balancing will select the endpoint with the least -number of connections. If multiple endpoints match with the same number of least -connections, it will select a random one within those least connections. - -_NOTE: Gorouter currently only supports changing the load balancing strategy at -the gorouter level and does not yet support a finer-grained level such as -route-level. Therefore changing the load balancing algorithm from the default -(round-robin) should be proceeded with caution._ - -## When terminating TLS in front of Gorouter with a component that does not support sending HTTP headers - -### Enabling apps and CF to detect that request was encrypted using X-Forwarded-Proto - -If you terminate TLS in front of Gorouter, your component should send the -`X-Forwarded-Proto` HTTP header in order for applications and Cloud Foundry -system components to correctly detect when the original request was encrypted. -As an example, UAA will reject requests that do not include `X-Forwarded-Proto: -https`. - -If your TLS-terminating component does not support sending HTTP headers, we -recommend also terminating TLS at Gorouter. In this scenario you should only -disable TLS at Gorouter if your TLS-terminating component rejects unencrypted -requests **and** your private network is completely trusted. In this case, use -the following property to inform applications and CF system components that -requests are secure. - -```yaml -properties: - router: - force_forwarded_proto_https: true -``` - -### Enabling apps to detect the requestor's IP address using PROXY Protocol - -If you terminate TLS in front of Gorouter, your component should also send the -`X-Forwarded-Proto` HTTP header in order for `X-Forwarded-For` header to -applications can detect the requestor's IP address. - -If your TLS-terminating component does not support sending HTTP headers, you can -use the PROXY protocol to send Gorouter the requestor's IP address. - -If your TLS-terminating component supports the PROXY protocol, enable the PROXY -protocol on Gorouter using the following cf-deployment manifest property: - -```yaml -properties: - router: - enable_proxy: true -``` - -You can test this feature manually: - -```bash -echo -e "PROXY TCP4 1.2.3.4 [GOROUTER IP] 12345 [GOROUTER PORT]\r\nGET / HTTP/1.1\r\nHost: [APP URL]\r\n" | nc [GOROUTER IP] [GOROUTER PORT] -``` - -You should see in the access logs on the Gorouter that the `X-Forwarded-For` -header is `1.2.3.4`. You can read more about the PROXY Protocol -[here](http://www.haproxy.org/download/1.5/doc/proxy-protocol.txt). - -## HTTP/2 Support - -The Gorouter supports ingress and egress HTTP/2 connections when the BOSH -deployment manifest property is enabled. - -```yaml -properties: - router: - enable_http2: true -``` - -By default, connections will be proxied to backends over HTTP/1.1, regardless of -ingress protocol. Backends can be configured with the `http2` protocol to enable -end-to-end HTTP/2 routing for use cases like gRPC. - -Example `router.register` message with `http2` protocol: -```json -{ - "host": "127.0.0.1", - "port": 4567, - "protocol": "http2", - "...": "..." -} -``` - -## Logs - -The router's logging is specified in its YAML configuration file. It supports -the following log levels: - -* `fatal` - A fatal error has occurred that makes gorouter unable to handle any - requests. Examples: the router can't bind to its TCP port, a CF component has - published invalid data to the router. -* `error` - An unexpected error has occurred. Examples: the router failed to - fetch token from UAA service. -* `info` - An expected event has occurred. Examples: the router started or - exited, the router has begun to prune routes for stale droplets. -* `debug` - A lower-level event has occurred. Examples: route registration, - route unregistration. - -Sample log message in gorouter. - -`[2017-02-01 22:54:08+0000] {"log_level":0,"timestamp":"2019-11-21T22:16:18.750673404Z","message":"endpoint-registered","source":"vcap.gorouter.registry","data":{"uri":"0-*.login.bosh-lite.com","backend":"10.123.0.134:8080","modification_tag":{"guid":"","index":0}}}` - -- `log_level`: This represents logging level of the message -- `timestamp`: Time of the log in either RFC 3339 (default) or epoch format -- `message`: Content of the log line -- `source`: The function within Gorouter that initiated the log message -- `data`: Additional information that varies based on the message - -### Route table change logs - -The following log messages are emitted any time the routing table changes: - -- `route-registered`: a new route is added to the table -- `route-unregistered`: an existing route is removed from the table -- `endpoint-registered`: a new backend is added to the table - e.g. an app is scaled up and a new app instance is started -- `endpoint-unregistered`: a backend is removed from the table - e.g. an app is scaled down and an app instance is stopped - -Examples: - -Route mapped to existing application with 1 app instance: -``` -{"log_level":1,"timestamp":"2020-08-27T22:59:43.462087363Z","message":"route-registered","source":"vcap.gorouter.registry","data":{"uri":"a.springgreen.cf-app.com"}} -{"log_level":1,"timestamp":"2020-08-27T22:59:43.462279999Z","message":"endpoint-registered","source":"vcap.gorouter.registry","data":{"uri":"a.springgreen.cf-app.com","backend":"10.0.1.11:61002","modification_tag":{"guid":"","index":0},"isolation_segment":"-","isTLS":true}} -``` - -App with two mapped routes scaled up from 1 instance to 2: -``` -{"log_level":1,"timestamp":"2020-08-27T22:59:59.350998043Z","message":"endpoint-registered","source":"vcap.gorouter.registry","data":{"uri":"a.springgreen.cf-app.com","backend":"10.0.1.11:61006","modification_tag":{"guid":"","index":0},"isolation_segment":"-","isTLS":true}} -{"log_level":1,"timestamp":"2020-08-27T22:59:59.351131999Z","message":"endpoint-registered","source":"vcap.gorouter.registry","data":{"uri":"foo.springgreen.cf-app.com","backend":"10.0.1.11:61006","modification_tag":{"guid":"","index":0},"isolation_segment":"-","isTLS":true}} -``` - -App with two mapped routes scaled down from 2 instances to 1: -``` -{"log_level":1,"timestamp":"2020-08-27T23:00:27.122616625Z","message":"endpoint-unregistered","source":"vcap.gorouter.registry","data":{"uri":"a.springgreen.cf-app.com","backend":"10.0.1.11:61006","modification_tag":{"guid":"","index":0},"isolation_segment":"-","isTLS":true}} -{"log_level":1,"timestamp":"2020-08-27T23:00:27.123043785Z","message":"endpoint-unregistered","source":"vcap.gorouter.registry","data":{"uri":"foo.springgreen.cf-app.com","backend":"10.0.1.11:61006","modification_tag":{"guid":"","index":0},"isolation_segment":"-","isTLS":true}} -``` - -Route unmapped from application with 1 app instance: -``` -{"log_level":1,"timestamp":"2020-08-27T23:00:46.702876112Z","message":"endpoint-unregistered","source":"vcap.gorouter.registry","data":{"uri":"a.springgreen.cf-app.com","backend":"10.0.1.11:61002","modification_tag":{"guid":"","index":0},"isolation_segment":"-","isTLS":true}} -{"log_level":1,"timestamp":"2020-08-27T23:00:46.703133349Z","message":"route-unregistered","source":"vcap.gorouter.registry","data":{"uri":"a.springgreen.cf-app.com"}} -``` - - -### Access logs - -Access logs provide information for the following fields when receiving a -request: - -` - [] " -" -"" "" -x_forwarded_for:"" x_forwarded_proto:"" -vcap_request_id: response_time: -gorouter_time: app_id: -app_index: instance_id:"" -failed_attempts: failed_attempts_time: -dns_time: dial_time: tls_time: -backend_time: x_cf_routererror: -` - -* Status Code, Response Time, Gorouter Time, Application ID, Application Index, - X-Cf-RouterError, and Extra Headers are all optional fields. The absence of - Status Code, Response Time, Application ID, Application Index, or - X-Cf-RouterError will result in a "-" in the corresponding field. - -* `Response Time` is the total time it takes for the request to go through the - Gorouter to the app and for the response to travel back through the Gorouter. - This includes the time the request spends traversing the network to the app - and back again to the Gorouter. It also includes the time the app spends - forming a response. - -* `Gorouter Time` is the total time it takes for the request to go through the - Gorouter initially plus the time it takes for the response to travel back - through the Gorouter. This does not include the time the request spends - traversing the network to the app. This also does not include the time the app - spends forming a response. - -* `failed_attempts`, `failed_attempts_time`, `dns_time`, `dial_time`, - `tls_time` and `backend_time` are only logged if - `logging.enable_attempts_details` is set to true. The `*_time` will only be - provided for the last, successful attempt, if the request fails they will be - empty and the error log can be consulted to get the details about each - attempt. `failed_attempts_time` contains the total time spent performing - attempts that failed. - -* `X-CF-RouterError` is populated if the Gorouter encounters an error. This can - help distinguish if a non-2xx response code is due to an error in the Gorouter - or the backend. For more information on the possible Router Error causes go to - the [#router-errors](#router-errors) section. - -Access logs are also redirected to syslog. - -## Headers - -If a user wants to send requests to a specific app instance, the header -`X-CF-APP-INSTANCE` can be added to indicate the specific instance to be -targeted. The format of the header value should be `X-Cf-App-Instance: -APP_GUID:APP_INDEX`. If the instance cannot be found or the format is wrong, a -400 status code is returned. In addition, Gorouter will return a -`X-Cf-Routererror` header. If the instance guid provided is incorrectly -formatted, the value of the header will be `invalid_cf_app_instance_header`. If -the instance guid provided is correctly formatted, but the guid does not exist, -the value of this header will be `unknown_route`, and the request body will -contain `400 Bad Request: Requested instance ('1') with guid -('aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa') does not exist for route -('dora.superman.routing.cf-app.com')`. - -Usage of the `X-Cf-App-Instance` header is only available for users on the Diego -architecture. - -### Router Errors - -The value of the `X-Cf-Routererror` header can be one of the following: - -| Value | Description | -|--------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| invalid_cf_app_instance_header | The provided value for the "X-Cf-App-Instance" header does not match the required format of `APP_GUID:INSTANCE_ID`. | -| empty_host | The value for the "Host" header is empty, or the "Host" header is equivalent to the remote address. Some LB's optimistically set the "Host" header value with their IP address when there is no value present. | -| unknown_route | The desired route does not exist in the gorouter's route table. | -| no_endpoints | There is an entry in the route table for the desired route, but there are no healthy endpoints available. | -| Connection Limit Reached | The backends associated with the route have reached their max number of connections. The max connection number is set via the spec property `router.backends.max_conns`. | -| route_service_unsupported | Route services are not enabled. This can be configured via the spec property `router.route_services_secret`. If the property is empty, route services are disabled. | -| endpoint_failure | The registered endpoint for the desired route failed to handle the request. - -## Supported Cipher Suites - -The Gorouter supports both RFC and OpenSSL formatted values. Refer to [golang -1.9](https://github.com/golang/go/blob/release-branch.go1.9/src/crypto/tls/cipher_suites.go#L369-L390) -for the list of supported cipher suites for Gorouter. Refer to [this -documentation](https://testssl.sh/openssl-rfc.mapping.html) for a list of -OpenSSL RFC mappings. Example configurations enabling the -TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 cipher suite for Gorouter: - -```yaml -enable_ssl: true -cipher_suite: "ECDHE-ECDSA-AES128-GCM-SHA256" -``` - -or - -```yaml -enable_ssl: true -cipher_suite: "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256" -``` - -## Docs - -There is a separate [docs](docs) folder which contains more advanced -topics. - -## Troubleshooting - -Refer -[doc](https://docs.pivotal.io/pivotalcf/adminguide/troubleshooting_slow_requests.html) -to learn more troubleshooting slow requests. +> This repository should be imported as +> `code.cloudfoundry.org/gorouter`. -## Development +# Docs -### Dependencies +- [Development Guide](./docs/01-development-guide.md) +- [NATS Configuration](./docs/02-nats-configurations.md) -This repository's dependencies are managed using -[routing-release](https://github.com/cloudfoundry/routing-release). Please refer to documentation in that repository for setting up tests +# Contributing -### Executables +See the [Contributing.md](./.github/CONTRIBUTING.md) for more +information on how to contribute. -1. `bin/test.bash`: This file is used to run test in Docker & CI. Please refer to [Dependencies](#dependencies) for setting up tests. +# Working Group Charter -### Reporting issues and requesting features +This repository is maintained by [App Runtime +Platform](https://github.com/cloudfoundry/community/blob/main/toc/working-groups/app-runtime-platform.md) +under `Networking` area. -Please report all issues and feature requests in [cloudfoundry/routing-release](https://github.com/cloudfoundry/routing-release). +> \[!IMPORTANT\] +> +> Content in this file is managed by the [CI task +> `sync-readme`](https://github.com/cloudfoundry/wg-app-platform-runtime-ci/blob/c83c224ad06515ed52f51bdadf6075f56300ec93/shared/tasks/sync-readme/metadata.yml) +> and is generated by CI following a convention.