There is a limit on stream retry attempts

[youngm]

While doing some fault tolerance testing I noticed that if there is a service disruption while streaming the stream will only attempt to reconnect after 5 tries, and then it appears never to try again even when the connection is restored. After it is restored only Polling seems to work.

15:22:51.710 [PollThread] WARN  io.harness.cf.client.api.PollingProcessor - retrieveAll failed: CompletionException - io.harness.cf.client.connector.ConnectorException: UnknownHostException: No such host is known (config.ff.harness.io) - httpCode=400 httpReason=UnknownHostException: No such host is known (config.ff.harness.io)
ON
15:22:54.501 [OkHttp https://config.ff.harness.io/...] WARN  io.harness.cf.client.connector.NewRetryInterceptor - Request attempt 5 to https://config.ff.harness.io/api/1.0/stream?cluster=2 was not successful, [UnknownHostException: config.ff.harness.io], retry limited reached
15:22:54.502 [OkHttp https://config.ff.harness.io/...] WARN  io.harness.cf.client.connector.EventSource - SSE Stream aborted: Invalid SSE HTTP response: Response{protocol=h2, code=400, message=UnknownHostException: config.ff.harness.io, url=https://config.ff.harness.io/api/1.0/stream?cluster=2}

[erdirowlands]

Hi @youngm,

I’ve reproduced the issue and can confirm that this behavior is unexpected and inconsistent with our other SDKs. The expected behaviour is for the stream to continue retrying using exponential backoff.

I’ll start working on a fix to address this by:

1. Fixing the stream retry behaviour so that it continues to retry with the following exponential backoff pattern:

1st retry: 1 second (1000 ms)
2nd retry: 2 seconds (2000 ms)
3rd retry: 4 seconds (4000 ms)
4th retry: 8 seconds (8000 ms)
5th retry: 16 seconds (16,000 ms)
6th retry: 32 seconds (32,000 ms)
7th retry and beyond: 60 seconds (max cap)

2. Exposing a new retry configuration for the remaining SDK requests, and setting the default to limit to 10 attempts using the same exponential backoff.

These requests are:

• Authentication with Harness / Proxy
• Polling: Fetches all flags and target groups.
• Reacting to a stream event: Fetches a single flag or target group.
• Metrics: Posts analytics requests.

The SDK currently retries, and will continue to retry, on the following HTTP status codes:

429 Too Many Requests
408 Request Timeout
Any 5xx error (server errors)

[erdirowlands]

Hey @youngm we've released version 1.8.0 which resolves the issue. Please see https://github.com/harness/ff-java-server-sdk/releases/tag/1.8.0 for details.