aggregate cluster integration tests by bazmurphy · Pull Request #30 · bazmurphy/envoy

bazmurphy · 2025-03-28T14:16:58Z

Aggregate Cluster Integration Test

Work In Progress 🚧

max_requests
max_connections
max_pending_requests
max_retries
max_connection_pools

bazmurphy · 2025-04-02T15:39:26Z

---------- 00 TEST START
---------- 03 INITIALIZE START
---------- 01 CONFIG MODIFY START
aggregate_cluster name(): aggregate_cluster
BEFORE temp_aggregate_cluster_typed_config.clusters_size(): 2
BEFORE temp_aggregate_cluster_typed_config clusters[0]: cluster_1
BEFORE temp_aggregate_cluster_typed_config clusters[1]: cluster_2
AFTER temp_aggregate_cluster_typed_config.clusters_size(): 1
AFTER temp_aggregate_cluster_typed_config clusters[0]: cluster_1
BEFORE aggregate_cluster has_circuit_breakers(): 0
BEFORE aggregate_cluster thresholds_size(): 0
AFTER aggregate_cluster thresholds_size(): 1
AFTER aggregate_cluster has_circuit_breakers(): 1
AFTER aggregate_cluster thresholds_size(): 1
AFTER aggregate_cluster thresholds_size(): 1
aggregate_cluster threshold default max_connections().value(): 1000000
aggregate_cluster threshold default max_pending_requests().value(): 1000000
aggregate_cluster threshold default max_requests().value(): 1
aggregate_cluster threshold default max_retries().value(): 1000000
aggregate_cluster threshold default track_remaining(): 1
---------- 02 CONFIG MODIFY FINISH
---------- 04 INITIALIZE FINISH
cluster1_ name(): cluster_1
BEFORE cluster1_ has_circuit_breakers(): 0
BEFORE cluster1_circuit_breakers thresholds_size(): 0
AFTER cluster1_ has_circuit_breakers(): 1
AFTER cluster1_circuit_breakers thresholds_size(): 1
cluster1_ threshold default max_connections().value(): 1000000
cluster1_ threshold default max_pending_requests().value(): 1000000
cluster1_ threshold default max_requests().value(): 1
cluster1_ threshold default max_retries().value(): 1000000
cluster1_ threshold default track_remaining(): 1
---------- 05 UPDATING XDS CONFIG FOR cluster1_
---------- 06 CIRCUIT BREAKER STATS [BEFORE]
BEFORE request/response1 aggregate_cluster rq_open: 0
BEFORE request/response1 aggregate_cluster remaining_rq: 1
BEFORE request/response1 cluster_1 rq_open: 0
BEFORE request/response1 cluster_1 remaining_rq: 1
---------- 07 MAKING HTTP CONNECTION
---------- 08 SENDING REQUEST TO /aggregatecluster
---------- 09 WAIT FOR REQUEST TO ARRIVE AT cluster_1
---------- 10 WAIT FOR THE REQUEST TO TRIGGER THE CIRCUIT BREAKER(S)
---------- 11 CIRCUIT BREAKER STATS [DURING]
DURING request/response1 aggregate_cluster rq_open: 0
DURING request/response1 aggregate_cluster remaining_rq: 1
DURING request/response1 cluster_1 rq_open: 1
DURING request/response1 cluster_1 remaining_rq: 0
---------- 12 ENCODING HEADERS AND ENDING STREAM, RETURNING RESPONSE
---------- 13 CIRCUIT BREAKER STATS [AFTER]
AFTER request/response1 aggregate_cluster rq_open: 0
AFTER request/response1 aggregate_cluster remaining_rq: 1
AFTER request/response1 cluster_1 rq_open: 0
AFTER request/response1 cluster_1 remaining_rq: 1
---------- 99 TEST END

From the above^

A request is sent to /aggregatecluster
Makes it way to cluster_1
Trips the cluster_1 max_requests circuit breaker
Importantly it does NOT trip the aggregate_cluster max_requests circuit breaker
Then when the response completes the cluster_1 circuit breaker state returns to its initial state

DURING request/response1 aggregate_cluster rq_open: 0
DURING request/response1 aggregate_cluster remaining_rq: 1
DURING request/response1 cluster_1 rq_open: 1  <--- cluster_1 max_requests circuit breaker open
DURING request/response1 cluster_1 remaining_rq: 0  <--- cluster_1 remaining_requests 0

bazmurphy · 2025-04-03T16:32:02Z

Max Connections Circuit Breaker (HTTP2, max_concurrent_streams: 1)

---------- 00 TEST START
aggregate_cluster max_concurrent_streams: 1
cluster1_ max_concurrent_streams: 1
BEFORE aggregate_cluster cx_open: 0
BEFORE aggregate_cluster remaining_cx: 1
BEFORE cluster_1 cx_open: 0
BEFORE cluster_1 remaining_cx: 1
DURING aggregate_cluster cx_open: 0
DURING aggregate_cluster remaining_cx: 1
DURING cluster_1 cx_open: 1    <--- circuit breaker triggered
DURING cluster_1 remaining_cx: 0    <--- 0 remaining connections
AFTER aggregate_cluster cx_open: 0
AFTER aggregate_cluster remaining_cx: 1
AFTER cluster_1 cx_open: 0
AFTER cluster_1 remaining_cx: 1
---------- 99 TEST END

bazmurphy · 2025-04-07T10:27:45Z

Max Pending Requests (HTTP2)

set the circuit breakers for max_connections to 1 per upstream (so we use a single connection for the tests)
set the circuit breakers for max_pending_requests per upstream to 1
set a limit of 1 max_concurrent_streams per connection

send request1 - and pause its completion
send request2 - this is the first pending request queued, it triggers the the max_pending_requests circuit breaker
send request3 - this automatically fails (rejected by the circuit breaker)

complete request/response1
complete request/response2
the circuit breaker returns to its initial state

2 pending requests were logged
only 1 connection was used

---------- 00 TEST START
aggregate_cluster max_concurrent_streams: 1
cluster1_ max_concurrent_streams: 1
--------------------
BEFORE aggregate_cluster rq_pending_open: 0
BEFORE aggregate_cluster remaining_pending: 1
BEFORE aggregate_cluster upstream_rq_active: 0
BEFORE aggregate_cluster upstream_rq_total: 0
BEFORE aggregate_cluster upstream_rq_pending_active: 0
BEFORE aggregate_cluster upstream_rq_pending_total: 0
BEFORE aggregate_cluster upstream_cx_active: 0
BEFORE aggregate_cluster upstream_cx_total: 0
BEFORE cluster_1 rq_pending_open: 0
BEFORE cluster_1 remaining_pending: 1
BEFORE cluster_1 upstream_rq_active: 0
BEFORE cluster_1 upstream_rq_total: 0
BEFORE cluster_1 upstream_rq_pending_active: 0
BEFORE cluster_1 upstream_rq_pending_total: 0
BEFORE cluster_1 upstream_cx_active: 0
BEFORE cluster_1 upstream_cx_total: 0
--------------------
DURING [request1] aggregate_cluster rq_pending_open: 0
DURING [request1] aggregate_cluster remaining_pending: 1
DURING [request1] aggregate_cluster upstream_rq_active: 0
DURING [request1] aggregate_cluster upstream_rq_total: 0
DURING [request1] aggregate_cluster upstream_rq_pending_active: 0
DURING [request1] aggregate_cluster upstream_rq_pending_total: 0
DURING [request1] aggregate_cluster upstream_cx_active: 0
DURING [request1] aggregate_cluster upstream_cx_total: 0
DURING [request1] cluster_1 rq_pending_open: 0
DURING [request1] cluster_1 remaining_pending: 1
DURING [request1] cluster_1 upstream_rq_active: 1
DURING [request1] cluster_1 upstream_rq_total: 1
DURING [request1] cluster_1 upstream_rq_pending_active: 0
DURING [request1] cluster_1 upstream_rq_pending_total: 1
DURING [request1] cluster_1 upstream_cx_active: 1
DURING [request1] cluster_1 upstream_cx_total: 1
--------------------
DURING [request2] aggregate_cluster rq_pending_open: 0
DURING [request2] aggregate_cluster remaining_pending: 1
DURING [request2] aggregate_cluster upstream_rq_active: 0
DURING [request2] aggregate_cluster upstream_rq_total: 0
DURING [request2] aggregate_cluster upstream_rq_pending_active: 0
DURING [request2] aggregate_cluster upstream_rq_pending_total: 0
DURING [request2] aggregate_cluster upstream_cx_active: 0
DURING [request2] aggregate_cluster upstream_cx_total: 0
DURING [request2] cluster_1 rq_pending_open: 1 <-- circuit breaker triggered
DURING [request2] cluster_1 remaining_pending: 0
DURING [request2] cluster_1 upstream_rq_active: 1
DURING [request2] cluster_1 upstream_rq_total: 1
DURING [request2] cluster_1 upstream_rq_pending_active: 1
DURING [request2] cluster_1 upstream_rq_pending_total: 2
DURING [request2] cluster_1 upstream_cx_active: 1
DURING [request2] cluster_1 upstream_cx_total: 1
--------------------
AFTER [all responses] aggregate_cluster rq_pending_open: 0
AFTER [all responses] aggregate_cluster remaining_pending: 1
AFTER [all responses] aggregate_cluster upstream_rq_active: 0
AFTER [all responses] aggregate_cluster upstream_rq_total: 0
AFTER [all responses] aggregate_cluster upstream_rq_pending_active: 0
AFTER [all responses] aggregate_cluster upstream_rq_pending_total: 0
AFTER [all responses] aggregate_cluster upstream_cx_active: 0
AFTER [all responses] aggregate_cluster upstream_cx_total: 0
AFTER [all responses] cluster_1 rq_pending_open: 0
AFTER [all responses] cluster_1 remaining_pending: 1
AFTER [all responses] cluster_1 upstream_rq_active: 0
AFTER [all responses] cluster_1 upstream_rq_total: 2
AFTER [all responses] cluster_1 upstream_rq_pending_active: 0
AFTER [all responses] cluster_1 upstream_rq_pending_total: 2
AFTER [all responses] cluster_1 upstream_cx_active: 1
AFTER [all responses] cluster_1 upstream_cx_total: 1
---------- 99 TEST END

bazmurphy · 2025-04-09T08:23:01Z

Max Retries Circuit Breaker Test (current state)

---------- 00 TEST START
route we are applying the retry changes to: /aggregatecluster
--------------------
BEFORE aggregate_cluster rq_retry_open: 0
BEFORE aggregate_cluster remaining_retries: 1
BEFORE aggregate_cluster upstream_rq_retry: 0
BEFORE aggregate_cluster upstream_rq_retry_success: 0
BEFORE aggregate_cluster upstream_rq_retry_overflow: 0
BEFORE aggregate_cluster upstream_rq_retry_limit_exceeded: 0
BEFORE aggregate_cluster upstream_rq_retry_backoff_exponential: 0
BEFORE aggregate_cluster upstream_rq_retry_backoff_ratelimited: 0
BEFORE aggregate_cluster upstream_rq_total: 0
BEFORE aggregate_cluster upstream_rq_active: 0
BEFORE aggregate_cluster upstream_rq_cancelled: 0
BEFORE aggregate_cluster upstream_rq_timeout: 0
BEFORE aggregate_cluster upstream_rq_completed: 0
BEFORE aggregate_cluster upstream_rq_max_duration_reached: 0
BEFORE aggregate_cluster upstream_rq_per_try_timeout: 0
BEFORE aggregate_cluster upstream_rq_rx_reset: 0
BEFORE aggregate_cluster upstream_rq_tx_reset: 0
BEFORE aggregate_cluster upstream_cx_active: 0
BEFORE aggregate_cluster upstream_cx_total: 0
--------------------
BEFORE cluster_1 rq_retry_open: 0
BEFORE cluster_1 remaining_retries: 1
BEFORE cluster_1 upstream_rq_retry: 0
BEFORE cluster_1 upstream_rq_retry_success: 0
BEFORE cluster_1 upstream_rq_retry_overflow: 0
BEFORE cluster_1 upstream_rq_retry_limit_exceeded: 0
BEFORE cluster_1 upstream_rq_retry_backoff_exponential: 0
BEFORE cluster_1 upstream_rq_retry_backoff_ratelimited: 0
BEFORE cluster_1 upstream_rq_total: 0
BEFORE cluster_1 upstream_rq_active: 0
BEFORE cluster_1 upstream_rq_cancelled: 0
BEFORE cluster_1 upstream_rq_timeout: 0
BEFORE cluster_1 upstream_rq_completed: 0
BEFORE cluster_1 upstream_rq_max_duration_reached: 0
BEFORE cluster_1 upstream_rq_per_try_timeout: 0
BEFORE cluster_1 upstream_rq_rx_reset: 0
BEFORE cluster_1 upstream_rq_tx_reset: 0
BEFORE cluster_1 upstream_cx_active: 0
BEFORE cluster_1 upstream_cx_total: 0
--------------------
DOWNSTREAM CLIENT: MAKING HTTP CONNECTION
--------------------
DOWNSTREAM CLIENT: SENDING REQUEST-1
--------------------
UPSTREAM CLUSTER: WAITING FOR REQUEST-1
--------------------
REQUEST-1 aggregate_cluster rq_retry_open: 0
REQUEST-1 aggregate_cluster remaining_retries: 1
REQUEST-1 aggregate_cluster upstream_rq_retry: 0
REQUEST-1 aggregate_cluster upstream_rq_retry_success: 0
REQUEST-1 aggregate_cluster upstream_rq_retry_overflow: 0
REQUEST-1 aggregate_cluster upstream_rq_retry_limit_exceeded: 0
REQUEST-1 aggregate_cluster upstream_rq_retry_backoff_exponential: 0
REQUEST-1 aggregate_cluster upstream_rq_retry_backoff_ratelimited: 0
REQUEST-1 aggregate_cluster upstream_rq_total: 0
REQUEST-1 aggregate_cluster upstream_rq_active: 0
REQUEST-1 aggregate_cluster upstream_rq_cancelled: 0
REQUEST-1 aggregate_cluster upstream_rq_timeout: 0
REQUEST-1 aggregate_cluster upstream_rq_completed: 0
REQUEST-1 aggregate_cluster upstream_rq_max_duration_reached: 0
REQUEST-1 aggregate_cluster upstream_rq_per_try_timeout: 0
REQUEST-1 aggregate_cluster upstream_rq_rx_reset: 0
REQUEST-1 aggregate_cluster upstream_rq_tx_reset: 0
REQUEST-1 aggregate_cluster upstream_cx_active: 0
REQUEST-1 aggregate_cluster upstream_cx_total: 0
--------------------
REQUEST-1 cluster_1 rq_retry_open: 0
REQUEST-1 cluster_1 remaining_retries: 1
REQUEST-1 cluster_1 upstream_rq_retry: 0
REQUEST-1 cluster_1 upstream_rq_retry_success: 0
REQUEST-1 cluster_1 upstream_rq_retry_overflow: 0
REQUEST-1 cluster_1 upstream_rq_retry_limit_exceeded: 0
REQUEST-1 cluster_1 upstream_rq_retry_backoff_exponential: 0
REQUEST-1 cluster_1 upstream_rq_retry_backoff_ratelimited: 0
REQUEST-1 cluster_1 upstream_rq_total: 1
REQUEST-1 cluster_1 upstream_rq_active: 1
REQUEST-1 cluster_1 upstream_rq_cancelled: 0
REQUEST-1 cluster_1 upstream_rq_timeout: 0
REQUEST-1 cluster_1 upstream_rq_completed: 0
REQUEST-1 cluster_1 upstream_rq_max_duration_reached: 0
REQUEST-1 cluster_1 upstream_rq_per_try_timeout: 0
REQUEST-1 cluster_1 upstream_rq_rx_reset: 0
REQUEST-1 cluster_1 upstream_rq_tx_reset: 0
REQUEST-1 cluster_1 upstream_cx_active: 1
REQUEST-1 cluster_1 upstream_cx_total: 1
--------------------
UPSTREAM CLUSTER: RESPOND TO REQUEST-1 WITH 503
--------------------
UPSTREAM CLUSTER: WAITING FOR REQUEST-1-RETRY
--------------------
CHECKING STATS (both clusters) rq_retry_open AND remaining_retries
--------------------
REQUEST-1-RETRY aggregate_cluster rq_retry_open: 1 <-- circuit breaker triggered
REQUEST-1-RETRY aggregate_cluster remaining_retries: 0 <-- 0 remaining retries
REQUEST-1-RETRY aggregate_cluster upstream_rq_retry: 1
REQUEST-1-RETRY aggregate_cluster upstream_rq_retry_success: 0
REQUEST-1-RETRY aggregate_cluster upstream_rq_retry_overflow: 0
REQUEST-1-RETRY aggregate_cluster upstream_rq_retry_limit_exceeded: 0
REQUEST-1-RETRY aggregate_cluster upstream_rq_retry_backoff_exponential: 1
REQUEST-1-RETRY aggregate_cluster upstream_rq_retry_backoff_ratelimited: 0
REQUEST-1-RETRY aggregate_cluster upstream_rq_total: 0
REQUEST-1-RETRY aggregate_cluster upstream_rq_active: 0
REQUEST-1-RETRY aggregate_cluster upstream_rq_cancelled: 0
REQUEST-1-RETRY aggregate_cluster upstream_rq_timeout: 0
REQUEST-1-RETRY aggregate_cluster upstream_rq_completed: 0
REQUEST-1-RETRY aggregate_cluster upstream_rq_max_duration_reached: 0
REQUEST-1-RETRY aggregate_cluster upstream_rq_per_try_timeout: 0
REQUEST-1-RETRY aggregate_cluster upstream_rq_rx_reset: 0
REQUEST-1-RETRY aggregate_cluster upstream_rq_tx_reset: 0
REQUEST-1-RETRY aggregate_cluster upstream_cx_active: 0
REQUEST-1-RETRY aggregate_cluster upstream_cx_total: 0
--------------------
REQUEST-1-RETRY cluster_1 rq_retry_open: 0
REQUEST-1-RETRY cluster_1 remaining_retries: 1
REQUEST-1-RETRY cluster_1 upstream_rq_retry: 0
REQUEST-1-RETRY cluster_1 upstream_rq_retry_success: 0
REQUEST-1-RETRY cluster_1 upstream_rq_retry_overflow: 0
REQUEST-1-RETRY cluster_1 upstream_rq_retry_limit_exceeded: 0
REQUEST-1-RETRY cluster_1 upstream_rq_retry_backoff_exponential: 0
REQUEST-1-RETRY cluster_1 upstream_rq_retry_backoff_ratelimited: 0
REQUEST-1-RETRY cluster_1 upstream_rq_total: 2
REQUEST-1-RETRY cluster_1 upstream_rq_active: 1
REQUEST-1-RETRY cluster_1 upstream_rq_cancelled: 0
REQUEST-1-RETRY cluster_1 upstream_rq_timeout: 0
REQUEST-1-RETRY cluster_1 upstream_rq_completed: 0
REQUEST-1-RETRY cluster_1 upstream_rq_max_duration_reached: 0
REQUEST-1-RETRY cluster_1 upstream_rq_per_try_timeout: 0
REQUEST-1-RETRY cluster_1 upstream_rq_rx_reset: 0
REQUEST-1-RETRY cluster_1 upstream_rq_tx_reset: 0
REQUEST-1-RETRY cluster_1 upstream_cx_active: 1
REQUEST-1-RETRY cluster_1 upstream_cx_total: 1
--------------------
DOWNSTREAM CLIENT: SENDING REQUEST-2
--------------------
UPSTREAM CLUSTER: WAITING FOR REQUEST-2
--------------------
REQUEST-2 aggregate_cluster rq_retry_open: 1
REQUEST-2 aggregate_cluster remaining_retries: 0
REQUEST-2 aggregate_cluster upstream_rq_retry: 1
REQUEST-2 aggregate_cluster upstream_rq_retry_success: 0
REQUEST-2 aggregate_cluster upstream_rq_retry_overflow: 0
REQUEST-2 aggregate_cluster upstream_rq_retry_limit_exceeded: 0
REQUEST-2 aggregate_cluster upstream_rq_retry_backoff_exponential: 1
REQUEST-2 aggregate_cluster upstream_rq_retry_backoff_ratelimited: 0
REQUEST-2 aggregate_cluster upstream_rq_total: 0
REQUEST-2 aggregate_cluster upstream_rq_active: 0
REQUEST-2 aggregate_cluster upstream_rq_cancelled: 0
REQUEST-2 aggregate_cluster upstream_rq_timeout: 0
REQUEST-2 aggregate_cluster upstream_rq_completed: 0
REQUEST-2 aggregate_cluster upstream_rq_max_duration_reached: 0
REQUEST-2 aggregate_cluster upstream_rq_per_try_timeout: 0
REQUEST-2 aggregate_cluster upstream_rq_rx_reset: 0
REQUEST-2 aggregate_cluster upstream_rq_tx_reset: 0
REQUEST-2 aggregate_cluster upstream_cx_active: 0
REQUEST-2 aggregate_cluster upstream_cx_total: 0
--------------------
REQUEST-2 cluster_1 rq_retry_open: 0
REQUEST-2 cluster_1 remaining_retries: 1
REQUEST-2 cluster_1 upstream_rq_retry: 0
REQUEST-2 cluster_1 upstream_rq_retry_success: 0
REQUEST-2 cluster_1 upstream_rq_retry_overflow: 0
REQUEST-2 cluster_1 upstream_rq_retry_limit_exceeded: 0
REQUEST-2 cluster_1 upstream_rq_retry_backoff_exponential: 0
REQUEST-2 cluster_1 upstream_rq_retry_backoff_ratelimited: 0
REQUEST-2 cluster_1 upstream_rq_total: 3
REQUEST-2 cluster_1 upstream_rq_active: 2
REQUEST-2 cluster_1 upstream_rq_cancelled: 0
REQUEST-2 cluster_1 upstream_rq_timeout: 0
REQUEST-2 cluster_1 upstream_rq_completed: 0
REQUEST-2 cluster_1 upstream_rq_max_duration_reached: 0
REQUEST-2 cluster_1 upstream_rq_per_try_timeout: 0
REQUEST-2 cluster_1 upstream_rq_rx_reset: 0
REQUEST-2 cluster_1 upstream_rq_tx_reset: 0
REQUEST-2 cluster_1 upstream_cx_active: 1
REQUEST-2 cluster_1 upstream_cx_total: 1
--------------------
UPSTREAM CLUSTER: RESPOND TO REQUEST-2 WITH 503
--------------------
CHECKING STATS upstream_rq_retry_overflow AND upstream_rq_retry_limit_exceeded
--------------------
REQUEST-2-RETRY aggregate_cluster rq_retry_open: 1
REQUEST-2-RETRY aggregate_cluster remaining_retries: 0
REQUEST-2-RETRY aggregate_cluster upstream_rq_retry: 1
REQUEST-2-RETRY aggregate_cluster upstream_rq_retry_success: 0
REQUEST-2-RETRY aggregate_cluster upstream_rq_retry_overflow: 1 <-- circuit breaker overflow
REQUEST-2-RETRY aggregate_cluster upstream_rq_retry_limit_exceeded: 0
REQUEST-2-RETRY aggregate_cluster upstream_rq_retry_backoff_exponential: 1
REQUEST-2-RETRY aggregate_cluster upstream_rq_retry_backoff_ratelimited: 0
REQUEST-2-RETRY aggregate_cluster upstream_rq_total: 0
REQUEST-2-RETRY aggregate_cluster upstream_rq_active: 0
REQUEST-2-RETRY aggregate_cluster upstream_rq_cancelled: 0
REQUEST-2-RETRY aggregate_cluster upstream_rq_timeout: 0
REQUEST-2-RETRY aggregate_cluster upstream_rq_completed: 1
REQUEST-2-RETRY aggregate_cluster upstream_rq_max_duration_reached: 0
REQUEST-2-RETRY aggregate_cluster upstream_rq_per_try_timeout: 0
REQUEST-2-RETRY aggregate_cluster upstream_rq_rx_reset: 0
REQUEST-2-RETRY aggregate_cluster upstream_rq_tx_reset: 0
REQUEST-2-RETRY aggregate_cluster upstream_cx_active: 0
REQUEST-2-RETRY aggregate_cluster upstream_cx_total: 0
--------------------
REQUEST-2-RETRY cluster_1 rq_retry_open: 0
REQUEST-2-RETRY cluster_1 remaining_retries: 1
REQUEST-2-RETRY cluster_1 upstream_rq_retry: 0
REQUEST-2-RETRY cluster_1 upstream_rq_retry_success: 0
REQUEST-2-RETRY cluster_1 upstream_rq_retry_overflow: 0
REQUEST-2-RETRY cluster_1 upstream_rq_retry_limit_exceeded: 0
REQUEST-2-RETRY cluster_1 upstream_rq_retry_backoff_exponential: 0
REQUEST-2-RETRY cluster_1 upstream_rq_retry_backoff_ratelimited: 0
REQUEST-2-RETRY cluster_1 upstream_rq_total: 3
REQUEST-2-RETRY cluster_1 upstream_rq_active: 1
REQUEST-2-RETRY cluster_1 upstream_rq_cancelled: 0
REQUEST-2-RETRY cluster_1 upstream_rq_timeout: 0
REQUEST-2-RETRY cluster_1 upstream_rq_completed: 0
REQUEST-2-RETRY cluster_1 upstream_rq_max_duration_reached: 0
REQUEST-2-RETRY cluster_1 upstream_rq_per_try_timeout: 0
REQUEST-2-RETRY cluster_1 upstream_rq_rx_reset: 0
REQUEST-2-RETRY cluster_1 upstream_rq_tx_reset: 0
REQUEST-2-RETRY cluster_1 upstream_cx_active: 1
REQUEST-2-RETRY cluster_1 upstream_cx_total: 1
--------------------
UPSTREAM CLUSTER: RESPOND TO REQUEST-1-RETRY WITH 503
--------------------
DOWNSTREAM CLIENT: RESPONSE-1 WAIT FOR END STREAM
--------------------
DOWNSTREAM CLIENT: RESPONSE-1 CHECKING HEADER STATUS IS 503
--------------------
DOWNSTREAM CLIENT: RESPONSE-2 WAIT FOR END STREAM
--------------------
DOWNSTREAM CLIENT: RESPONSE-2 CHECKING HEADER STATUS IS 503
--------------------
CHECKING STATS (both clusters) rq_retry_open AND remaining_retries AND upstream_rq_retry_limit_exceeded
--------------------
AFTER aggregate_cluster rq_retry_open: 0
AFTER aggregate_cluster remaining_retries: 1
AFTER aggregate_cluster upstream_rq_retry: 1
AFTER aggregate_cluster upstream_rq_retry_success: 0
AFTER aggregate_cluster upstream_rq_retry_overflow: 1
AFTER aggregate_cluster upstream_rq_retry_limit_exceeded: 1 <-- retry limit exceeded
AFTER aggregate_cluster upstream_rq_retry_backoff_exponential: 1
AFTER aggregate_cluster upstream_rq_retry_backoff_ratelimited: 0
AFTER aggregate_cluster upstream_rq_total: 0
AFTER aggregate_cluster upstream_rq_active: 0
AFTER aggregate_cluster upstream_rq_cancelled: 0
AFTER aggregate_cluster upstream_rq_timeout: 0
AFTER aggregate_cluster upstream_rq_completed: 2 <-- only request1 and request2 completed
AFTER aggregate_cluster upstream_rq_max_duration_reached: 0
AFTER aggregate_cluster upstream_rq_per_try_timeout: 0
AFTER aggregate_cluster upstream_rq_rx_reset: 0
AFTER aggregate_cluster upstream_rq_tx_reset: 0
AFTER aggregate_cluster upstream_cx_active: 0
AFTER aggregate_cluster upstream_cx_total: 0
--------------------
AFTER cluster_1 rq_retry_open: 0
AFTER cluster_1 remaining_retries: 1
AFTER cluster_1 upstream_rq_retry: 0
AFTER cluster_1 upstream_rq_retry_success: 0
AFTER cluster_1 upstream_rq_retry_overflow: 0
AFTER cluster_1 upstream_rq_retry_limit_exceeded: 0
AFTER cluster_1 upstream_rq_retry_backoff_exponential: 0
AFTER cluster_1 upstream_rq_retry_backoff_ratelimited: 0
AFTER cluster_1 upstream_rq_total: 3 <-- only 3 requests reached the cluster_1 (the request2-retry was prevented)
AFTER cluster_1 upstream_rq_active: 0
AFTER cluster_1 upstream_rq_cancelled: 0
AFTER cluster_1 upstream_rq_timeout: 0
AFTER cluster_1 upstream_rq_completed: 0
AFTER cluster_1 upstream_rq_max_duration_reached: 0
AFTER cluster_1 upstream_rq_per_try_timeout: 0
AFTER cluster_1 upstream_rq_rx_reset: 0
AFTER cluster_1 upstream_rq_tx_reset: 0
AFTER cluster_1 upstream_cx_active: 1
AFTER cluster_1 upstream_cx_total: 1
---------- 99 TEST END

Baz Murphy and others added 19 commits March 28, 2025 07:10

work in progress

3e3817c

additions

7e7bf6f

additions

026d63f

additions

483977e

additions

1d46ca0

additions

fbbc9ce

adjustments

4eed39e

additions

0cd1ddc

adjust

9859ae2

additions

e367f58

additions

e804d3e

additions

7bb4902

additions

594e7e8

more notes

8abb154

Merge branch 'main' into bazmurphy-aggregate-cluster-integration-test

e28ef35

log

55c144d

additions

3973f81

make it complete the request

7b00744

success?

56eea42

Baz Murphy added 8 commits April 2, 2025 09:10

check response

f815e24

test2

8001b6e

close codec client connections

9a28068

update test run log, now passing

93f2f24

adjustments

5c32469

additions

4797676

additions

07b208e

adjustments

59c3c33

additions

8ec1f54

Baz Murphy added 9 commits April 4, 2025 05:26

additions

ef75f4d

additions

b9cee3c

adjustments

49d6e4c

max pending request success log

f78f8d5

adjustments

5da9f6a

adjustments

c049ff1

cleanup

bc108d8

adjustments

0170f7f

adjustments

b022f0c

Baz Murphy added 10 commits April 7, 2025 07:06

additions

0f2054e

adjust retry policy

d2f5f99

retry continued

a4cdf74

additions

3f49b7c

additions

1ee465a

additions

a13043b

additions

757459e

additions

6cb9e68

max_retries success

5e20115

additions

35f5ab5

Baz Murphy added 3 commits April 9, 2025 04:11

adjustments

99c70ad

additions

1b1a1a5

connection pool hell

1eb893c

This was referenced Apr 14, 2025

max connections #33

Open

max retries #34

Open

bazmurphy changed the title ~~aggregate cluster integration test [wip]~~ aggregate cluster integration tests Apr 15, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

aggregate cluster integration tests#30

aggregate cluster integration tests#30
bazmurphy wants to merge 50 commits into
mainfrom
bazmurphy-aggregate-cluster-integration-test

bazmurphy commented Mar 28, 2025 •

edited

Loading

Uh oh!

bazmurphy commented Apr 2, 2025 •

edited

Loading

Uh oh!

bazmurphy commented Apr 3, 2025 •

edited

Loading

Uh oh!

bazmurphy commented Apr 7, 2025 •

edited

Loading

Uh oh!

bazmurphy commented Apr 9, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

bazmurphy commented Mar 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Aggregate Cluster Integration Test

Work In Progress 🚧

Uh oh!

bazmurphy commented Apr 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bazmurphy commented Apr 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bazmurphy commented Apr 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bazmurphy commented Apr 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

bazmurphy commented Mar 28, 2025 •

edited

Loading

bazmurphy commented Apr 2, 2025 •

edited

Loading

bazmurphy commented Apr 3, 2025 •

edited

Loading

bazmurphy commented Apr 7, 2025 •

edited

Loading

bazmurphy commented Apr 9, 2025 •

edited

Loading