Make the HTTP client that lives beneath fabric8 KubernetesClient configurable by capistrant · Pull Request #18540 · apache/druid

capistrant · 2025-09-16T22:45:48Z

Description

Allow an operator to select which HTTP client they want to use for their instance of fabric8 KubernetesClient

Http Client Selector

druid.indexer.runner.k8sAndWorker.http.httpClientType

Valid values: vertx (default), okhttp, jdk

why no jetty?

As of now no jetty http client support due to jetty dependency version clashing. It could be possible to look into this in the future though.

vertx

Extra configuration stays the same as it was in terms of key names and defaults, but the prefix changes to druid.indexer.runner.k8sAndWorker.http.vertx

okhttp

This is the old client that we used before #17913. Drawbacks are the default unbounded thread pool underpinning the client. This was a big reason behind making the change to vert.x in the first place (in addition to vert.x being the new default for fabric8 KubernetesClient in newer versions).

Extra configuration:

druid.indexer.runner.k8sAndWorker.http.okhttp.useCustomDispatcherExecutor
- Whether or not you want to override the default unbounded dispatcher.
- default = true
druid.indexer.runner.k8sAndWorker.http.okhttp.maxWorkerThreads
- The upper bound on the thread pool if you use the custom dispatcher
- default 50 (fixed size pool)
druid.indexer.runner.k8sAndWorker.http.okhttp.coreWorkerThreads
- The lower bound on the thread pool size if you use the custom dispatcher
- default = 50
druid.indexer.runner.k8sAndWorker.http.okhttp.workerThreadKeepAliveTime
- default = 60 (seconds)
- how long idle threads will live for the thread pool if you use the custom dispatcher

jdk

This uses the native http client in Java. Not recommended if runtime is java11.

Extra configuration:

druid.indexer.runner.k8sAndWorker.http.jdk.maxWorkerThreads
- The upper bound on the thread pool
- default 50 (fixed size pool)
druid.indexer.runner.k8sAndWorker.http.jdk.coreWorkerThreads
- The lower bound on the thread pool size
- default = 20
druid.indexer.runner.k8sAndWorker.http.jdk.workerThreadKeepAliveTime
- default = 50 (seconds)
- how long idle threads will live for the thread pool.

Release note

Adds new experimental configuration to allow users of the kubernetes-overlord-extensions to configure what HTTP client library Fabric8 KubernetesClient uses under the hood to communicate with the k8s server tasks run on. The default remains the same client and config that druid 34 uses. Additional options that are now available are okhttp and a native JDK HttpClient. The default client and config should suffice for most use cases.

Key changed/added classes in this PR

KubernetesOverlordModule
DruidKubernetesHttpClientFactory and each implementation of it.

This PR has:

been self-reviewed.
added documentation for new or modified features or behaviors.
a release note entry in the PR description.
added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
added or updated version, license, or notice information in licenses.yaml
added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
added integration tests.
been tested in a test Druid cluster.

gianm · 2025-09-17T11:26:27Z

  private static final String RUNNERSTRATEGY_PROPERTIES_FORMAT_STRING = K8SANDWORKER_PROPERTIES_PREFIX
                                                                        + ".runnerStrategy.%s";
-  private static final String HTTPCLIENT_PROPERITES_PREFIX = K8SANDWORKER_PROPERTIES_PREFIX + ".http";
+  private static final String HTTPCLIENT_TYPE_PROPERTY = K8SANDWORKER_PROPERTIES_PREFIX + ".http.httpClientType";


You haven't added documentation for this parameter, which I think is good because I think it's a parameter that we may remove "soon". I suggest noting that in a comment here. Its purpose is to help cluster operators evaluate different http clients with fabric8, and we're adding it because we've noticed issues with two of the major ones:

the old default (okhttp) can lead to excessive numbers of threads if you have a lot of tasks running

the new default (vert.x) has been observed to generate spurious API call failures, which leads to seemingly-random task failures

Ideally, through some evaluation we will determine which one is best to use and how best to configure it, and at that point we could remove the property. It will help us slim down the distribution since it won't need to ship all 3 fabric8 client modules.

I may or may not have been working on a commit to add docs after I realized I forgot 😆 but we can leave un-documented I guess. But maybe putting it in the extensions doc under an experimental label plus disclaimer that this is temporary while the community determines best path forward for a generic config that works well for all is good compromise. Then all users of the extension could help find the best path forward if they so choose.

That'd also be OK, I just don't want to create a reason to worry about removing it in the future. If the doc page says that it may be removed in a future release then that works.

FrankChen021 · 2025-09-17T12:36:47Z

why is the jdk not recommend for java 11?

Another point is that the type name jdk is ambiguous, people may think it's the http client shipped in the sun.net.www.http package.

capistrant · 2025-09-17T13:31:04Z

why is the jdk not recommend for java 11?

Another point is that the type name jdk is ambiguous, people may think it's the http client shipped in the sun.net.www.http package.

https://github.com/fabric8io/kubernetes-client/tree/main/httpclient-jdk just reading from their README they mentioned java16+ having fixes that some use cases need. Did not evaluate deeply since Druid is pending drop of java11 support due to jetty upgrade anyways.

FrankChen021 · 2025-09-17T14:55:48Z

why is the jdk not recommend for java 11?
Another point is that the type name jdk is ambiguous, people may think it's the http client shipped in the sun.net.www.http package.

https://github.com/fabric8io/kubernetes-client/tree/main/httpclient-jdk just reading from their README they mentioned java16+ having fixes that some use cases need. Did not evaluate deeply since Druid is pending drop of java11 support due to jetty upgrade anyways.

Thanks for the info.
From the link, I notice that the problem is related to web sockets, not the normal http request. So I think it's not our case here.

It will not work with websocket requests containing queries with encoded characters until the fix openjdk/jdk@c07ce7e which is available in Java 16 - https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8245245

…reads

Also enforce proper setting of max threads for okhttp

cryptoe · 2025-09-19T07:27:37Z

+
+|Property| Possible Values |Description| Default |required|
+|--------|-----------------|-----------|---------|--------|
+|`druid.indexer.runner.k8sAndWorker.http.httpClientType`|`String` (e.g., `okhttp`, `vertx`, `javaStandardHttp`)|Specifies the HTTP client library to be used by the worker task runner for communication with worker nodes.|`vertx`|No|


Should the default be okhttp.
cc @gianm ?

It should be whatever we believe works best for the average user.

I do not think we are ready to say vert.x has enough problems to remove it from being default. If we still want this option to pick a client in druid35 so the community can work to identify the best long term approach, I'm on board with that. But as of now, I think leaving the default as is is best for druid 35.

This reverts commit 8b40ab7.

gianm · 2025-10-13T16:18:38Z

+
+:::
+
+The extension uses [fabric8 KubernetesClient](https://github.com/fabric8io/kubernetes-client) to communicate with the Kubernetes API server. This client creates an


Since this is area where there are problems and our understanding of them is evolving, I think it would be better to move most of this info to a GitHub issue and link it here. Name the GitHub issue something like "Determine best Kubernetes API client" and put the known issues and their mitigations there. That way, as our understanding evolves, old versions' documentation doesn't grow too stale. The only info we need here is a brief preamble, pointer to the GitHub issue, and the table with the descriptions of the configuration options.

makes sense, have created #18629 for the tracking and discussion of the issue and removed most of the docs content outside of the configs

gianm · 2025-10-15T12:16:41Z

TY @capistrant!

…igurable (apache#18540) * Make the httpclient backing fabric8 KubernetesClient pluggable * fix checkstyle * fix licenses * Cleanup prefixes for pluggable http client config * Default okhttp and native jdk http client threadpools to static 20 threads * experimental docs for http client config * Cleanup docs and make native jdk client name more specific * Fix unit tests * fix dependency analyzer * Make okhttp use the custom executor by default and bump its thread count Also enforce proper setting of max threads for okhttp * make native jdk http client configs more robust * fix checkstyle * Flip to okhttp as underlying http client for fabric8 * Revert "Flip to okhttp as underlying http client for fabric8" This reverts commit 8b40ab7. * Turn off custom dispatcher for okhttp * slim down docs for this experimental stuff and point to github issue

…igurable (#18540) * Make the httpclient backing fabric8 KubernetesClient pluggable * fix checkstyle * fix licenses * Cleanup prefixes for pluggable http client config * Default okhttp and native jdk http client threadpools to static 20 threads * experimental docs for http client config * Cleanup docs and make native jdk client name more specific * Fix unit tests * fix dependency analyzer * Make okhttp use the custom executor by default and bump its thread count Also enforce proper setting of max threads for okhttp * make native jdk http client configs more robust * fix checkstyle * Flip to okhttp as underlying http client for fabric8 * Revert "Flip to okhttp as underlying http client for fabric8" This reverts commit 8b40ab7. * Turn off custom dispatcher for okhttp * slim down docs for this experimental stuff and point to github issue

This commit adds non-blocking I/O support for Kubernetes API calls by backporting the Vertx HTTP client from Druid 35. This addresses thread pool exhaustion issues when running many concurrent tasks. Changes: - Add kubernetes-httpclient-vertx dependency (v6.7.2) - Add HttpClientType enum (VERTX default, OKHTTP fallback) - Add DruidKubernetesHttpClientFactory interface - Add DruidKubernetesVertxHttpClientConfig for thread pool configuration - Add DruidKubernetesVertxHttpClientFactory for Vertx instance management - Modify DruidKubernetesClient to accept custom HTTP client factory - Modify KubernetesTaskRunnerConfig with httpClientType and vertxHttpClientConfig - Modify KubernetesOverlordModule to select HTTP client based on config - Fix buildJob() method to accept taskType parameter (pre-existing bug) - Add comprehensive logging for debugging and verification Configuration: - Vertx is enabled by default (no config needed) - Fallback: druid.indexer.runner.httpClientType=OKHTTP - Optional tuning: druid.indexer.runner.vertxHttpClientConfig.* See VERTX_HTTP_CLIENT_BACKPORT.md for full documentation. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

Verified implementation against actual upstream code (commit cabada6): - Use VertxOptions.DEFAULT_* constants instead of hardcoded values - Align createVertxInstance() method with upstream exactly - Add TYPE_NAME constant to factory - Remove unnecessary toString() from config - Remove conditional eventLoopPoolSize check (always set like upstream) Keep our additions (not in upstream, but useful): - Logging for debugging and verification - close() method for clean Vertx shutdown Updated documentation with upstream comparison section. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

capistrant added 2 commits September 9, 2025 11:23

Make the httpclient backing fabric8 KubernetesClient pluggable

cabada6

fix checkstyle

2d41e5a

github-actions Bot added Area - Dependencies Kubernetes labels Sep 16, 2025

gianm reviewed Sep 17, 2025

View reviewed changes

capistrant added 2 commits September 17, 2025 08:55

fix licenses

17c81c7

Cleanup prefixes for pluggable http client config

c805eb9

capistrant added 2 commits September 17, 2025 10:59

Default okhttp and native jdk http client threadpools to static 20 th…

f535f98

…reads

experimental docs for http client config

434efa6

github-actions Bot added the Area - Documentation label Sep 17, 2025

capistrant added 6 commits September 17, 2025 13:44

Cleanup docs and make native jdk client name more specific

4f6eb1e

Fix unit tests

a8fe104

fix dependency analyzer

86940a6

Make okhttp use the custom executor by default and bump its thread count

328212b

Also enforce proper setting of max threads for okhttp

make native jdk http client configs more robust

ac500d3

fix checkstyle

6030a40

cryptoe reviewed Sep 19, 2025

View reviewed changes

capistrant added 6 commits September 19, 2025 10:33

Merge branch 'master' into k8s-client-plugable

5e7704a

Merge branch 'master' into k8s-client-plugable

a124d81

Flip to okhttp as underlying http client for fabric8

8b40ab7

Revert "Flip to okhttp as underlying http client for fabric8"

be5314a

This reverts commit 8b40ab7.

Turn off custom dispatcher for okhttp

d4db1a3

Merge branch 'master' into k8s-client-plugable

2b3adab

cecemei added this to the 35.0.0 milestone Oct 7, 2025

capistrant requested a review from cryptoe October 7, 2025 23:21

Merge branch 'master' into k8s-client-plugable

56d556b

capistrant added the Release Notes label Oct 13, 2025

cryptoe approved these changes Oct 13, 2025

View reviewed changes

gianm reviewed Oct 13, 2025

View reviewed changes

slim down docs for this experimental stuff and point to github issue

c223785

gianm approved these changes Oct 15, 2025

View reviewed changes

gianm merged commit ccb386e into apache:master Oct 15, 2025
130 of 132 checks passed

cecemei mentioned this pull request Oct 21, 2025

[Backport] Make the HTTP client that lives beneath fabric8 KubernetesClient configurable #18670

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make the HTTP client that lives beneath fabric8 KubernetesClient configurable#18540

Make the HTTP client that lives beneath fabric8 KubernetesClient configurable#18540
gianm merged 20 commits intoapache:masterfrom
capistrant:k8s-client-plugable

capistrant commented Sep 16, 2025 •

edited

Loading

Uh oh!

gianm Sep 17, 2025

Uh oh!

capistrant Sep 17, 2025

Uh oh!

gianm Sep 18, 2025

Uh oh!

FrankChen021 commented Sep 17, 2025

Uh oh!

capistrant commented Sep 17, 2025

Uh oh!

FrankChen021 commented Sep 17, 2025

Uh oh!

cryptoe Sep 19, 2025

Uh oh!

gianm Sep 24, 2025

Uh oh!

capistrant Oct 7, 2025

Uh oh!

gianm Oct 13, 2025

Uh oh!

capistrant Oct 13, 2025

Uh oh!

gianm commented Oct 15, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants


		:::

		The extension uses [fabric8 KubernetesClient](https://github.com/fabric8io/kubernetes-client) to communicate with the Kubernetes API server. This client creates an

Conversation

capistrant commented Sep 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Http Client Selector

vertx

okhttp

jdk

Release note

Key changed/added classes in this PR

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

FrankChen021 commented Sep 17, 2025

Uh oh!

capistrant commented Sep 17, 2025

Uh oh!

FrankChen021 commented Sep 17, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

gianm commented Oct 15, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

capistrant commented Sep 16, 2025 •

edited

Loading