Enable custom configuration for http connection in OrchestrationClient

[14f34890]

Describe the Problem

We would like have possibility to customize http connection settings for OrchestrationClient when create it.

Propose a Solution

For example it can be done:

HttpConnectionConfig httpConnectionConfig = HttpConnectionConfig.custom()
    .setConnectTimeout(timeout, TimeUnit.MILLISECONDS)
    .setResponseTimeout(timeout, TimeUnit.MILLISECONDS)
    .setSocketTimeout(timeout, TimeUnit.MILLISECONDS)
     ...
    .build();
OrchestrationClient orchestration = new OrchestrationClient(httpConnectionConfig)

Could you please provide such feature?

Describe Alternatives

No response

Affected Development Phase

Getting Started

Impact

No Impact

Timeline

No response

[MatKuhr]

You can already customize the HTTP client as described here.

Please note that this will affect all HTTP traffic performed by the AI SDK. If that is not desired, please share more information about your use case and why you want to change the configuration specifically only for orchestration calls.

[14f34890]

We would like to have possibility to run calls to OrchestrationClient with different http connection settings.
For example for health check calls to Orchestration we want to keep timeout short, as there are timeout restrictions in the platform itself which we use for health check.

And for usual calls (not health heck), we want to have longer timeout settings.

So, in case your SDK provides possibility to configure connection timeout settings at the creation of OrchestrationClient, we can just create two different clients and use one for health check (short timeout) the other for other calls (long timeout).

[MatKuhr]

If I understand you correctly, for those use cases the connection timeout is not suitable and will not achieve what you want. For example, a 2 minute connection timeout still allows for requests that take 10 minutes. It just means the TCP connection doesn't timeout, not that the server responds within 2 minutes.

Instead, please apply a resilience configuration such as:

var config = ResilienceConfiguration.of("orchestration-health-check")
            .timeLimiterConfiguration(ResilienceConfiguration.TimeLimiterConfiguration.of(Duration.ofSeconds(10)));

    try {
      ResilienceDecorator.executeSupplier(() -> healthCheck(), config);
    } catch (Exception e) {
      System.out.println("Health check failed: " + e.getMessage());
    }

Finally, we generally advise against performing LLM calls as part of a health check. Health checks should be fast, inexpensive and only include things an application can't meaningfully start or run without (e.g. a database).