Skip to content

Kubernetes executionconfig API should return effective config #19014

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
GWphua wants to merge 10 commits into
base: master
Choose a base branch
from
Open

Kubernetes executionconfig API should return effective config #19014

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
848e8af
Dynamic config retrieval logic.
GWphua Feb 11, 2026
54f6ed2
Effective strategy to be returned by API
GWphua Feb 12, 2026
3ec7648
Test for effective resource
GWphua Feb 12, 2026
ad04374
Test for effective resource + cleanup
GWphua Feb 12, 2026
5a41512
Nullable parameters in jsoncreator
GWphua Feb 12, 2026
0256a5e
Docs
GWphua Feb 27, 2026
49e013d
Docs
GWphua Feb 27, 2026
f2a0b34
Docs
GWphua Feb 27, 2026
043b252
Docs
GWphua Feb 27, 2026
8eae02a
Spelling check
GWphua Feb 27, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion docs/development/extensions-core/k8s-jobs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,9 @@ To use these APIs, ensure you have read and write permissions for the CONFIG res

#### Get dynamic configuration

Retrieves the current dynamic execution config for the Kubernetes task runner.
> Prior to Druid 37.0.0, this API will return an empty value when the dynamic config has not been updated via the POST method below. This has since changed to always reflect the dynamic config that will be used by the task runner to create K8s jobs.

Retrieve the current execution config used by the Kubernetes task runner.
Returns a JSON object with the dynamic configuration properties.

##### URL
Expand Down
4 changes: 4 additions & 0 deletions ...ava/org/apache/druid/k8s/overlord/execution/DefaultKubernetesTaskRunnerDynamicConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,21 +35,25 @@ public class DefaultKubernetesTaskRunnerDynamicConfig implements KubernetesTaskR

@JsonCreator
public DefaultKubernetesTaskRunnerDynamicConfig(
@Nullable
@JsonProperty("podTemplateSelectStrategy") PodTemplateSelectStrategy podTemplateSelectStrategy,
@Nullable
@JsonProperty("capacity") Integer capacity
)
{
this.podTemplateSelectStrategy = podTemplateSelectStrategy;
this.capacity = capacity;
}

@Nullable
@Override
@JsonProperty
public PodTemplateSelectStrategy getPodTemplateSelectStrategy()
{
return podTemplateSelectStrategy;
}

@Nullable
@Override
@JsonProperty
public Integer getCapacity()
Expand Down
27 changes: 13 additions & 14 deletions ...n/java/org/apache/druid/k8s/overlord/execution/KubernetesTaskExecutionConfigResource.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@
import org.apache.druid.common.config.JacksonConfigManager;
import org.apache.druid.java.util.common.Intervals;
import org.apache.druid.java.util.common.logger.Logger;
import org.apache.druid.k8s.overlord.KubernetesTaskRunnerEffectiveConfig;
import org.apache.druid.server.http.security.ConfigResourceFilter;
import org.apache.druid.server.security.AuthorizationUtils;
import org.joda.time.Interval;
Expand All @@ -43,7 +44,6 @@
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;
import java.util.List;
import java.util.concurrent.atomic.AtomicReference;

/**
* Resource that manages Kubernetes-specific execution configurations for running tasks.
Expand All @@ -57,16 +57,18 @@ public class KubernetesTaskExecutionConfigResource
private static final Logger log = new Logger(KubernetesTaskExecutionConfigResource.class);
private final JacksonConfigManager configManager;
private final AuditManager auditManager;
private AtomicReference<KubernetesTaskRunnerDynamicConfig> dynamicConfigRef = null;
private final KubernetesTaskRunnerEffectiveConfig effectiveConfig;

@Inject
public KubernetesTaskExecutionConfigResource(
final JacksonConfigManager configManager,
final AuditManager auditManager
final AuditManager auditManager,
final KubernetesTaskRunnerEffectiveConfig effectiveConfig
)
{
this.configManager = configManager;
this.auditManager = auditManager;
this.effectiveConfig = effectiveConfig;
}

/**
Expand All @@ -84,12 +86,9 @@ public Response setExecutionConfig(
@Context final HttpServletRequest req
)
{
KubernetesTaskRunnerDynamicConfig currentConfig = getDynamicConfig();
KubernetesTaskRunnerDynamicConfig mergedConfig = dynamicConfig;
KubernetesTaskRunnerDynamicConfig currentConfig = getCurrentConfiguration();
KubernetesTaskRunnerDynamicConfig mergedConfig = currentConfig.merge(dynamicConfig);
Copy link
Copy Markdown
Member

@FrankChen021 FrankChen021 Apr 25, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 POST persists static fallback values as dynamic overrides

Merging the request into getCurrentConfiguration() means the value saved to metadata is the effective config, not just the dynamic override. For example, with no dynamic config and a static druid.indexer.runner.capacity of 10, a POST that only changes podTemplateSelectStrategy will persist capacity=10 as dynamic config. If the operator later changes the static capacity to 20 and restarts, the persisted dynamic capacity still wins, so the task runner unexpectedly stays capped at 10. The POST path should merge against the persisted dynamic config, or otherwise avoid writing static fallback values into dynamic config.


if (currentConfig != null) {
mergedConfig = currentConfig.merge(dynamicConfig);
}
final ConfigManager.SetResult setResult = configManager.set(
KubernetesTaskRunnerDynamicConfig.CONFIG_KEY,
mergedConfig,
Expand Down Expand Up @@ -154,14 +153,14 @@ public Response getExecutionConfigHistory(
@ResourceFilters(ConfigResourceFilter.class)
public Response getExecutionConfig()
{
return Response.ok(getDynamicConfig()).build();
return Response.ok(getCurrentConfiguration()).build();
}

private KubernetesTaskRunnerDynamicConfig getDynamicConfig()
private KubernetesTaskRunnerDynamicConfig getCurrentConfiguration()
{
if (dynamicConfigRef == null) {
dynamicConfigRef = configManager.watch(KubernetesTaskRunnerDynamicConfig.CONFIG_KEY, KubernetesTaskRunnerDynamicConfig.class);
}
return dynamicConfigRef.get();
return new DefaultKubernetesTaskRunnerDynamicConfig(
effectiveConfig.getPodTemplateSelectStrategy(),
effectiveConfig.getCapacity()
);
}
}
Loading
Loading