Skip to content

Remove deprecated maxRowsPerSegment from batch task tuning configs#16811

Closed
kfaraz wants to merge 7 commits intoapache:masterfrom
kfaraz:remove_max_rows_per_segment
Closed

Remove deprecated maxRowsPerSegment from batch task tuning configs#16811
kfaraz wants to merge 7 commits intoapache:masterfrom
kfaraz:remove_max_rows_per_segment

Conversation

@kfaraz
Copy link
Copy Markdown
Contributor

@kfaraz kfaraz commented Jul 29, 2024 •
edited
Loading

Description

The following fields have been deprecated since #8141 , Druid 0.16.0.

  • maxRowsPerSegment
  • numShards
  • maxTotalRows

Instead of these fields, users can explicitly specify the desired partitionsSpec.

Changes

  • Remove deprecated fields from batch task tuning configs.
  • IndexTuningConfig used in index tasks
  • ParallelIndexTuningConfig used in index_parallel tasks
  • CompactionTuningConfig used in compact tasks
  • DataSourceCompactionConfig used by auto-compaction duty
  • Streaming tasks tuning configs need not change as they always use dynamic partitioning
    and it suffices to just specify the maxRowsPerSegment.
  • Remove publishTimeout as it is just an alias for pushTimeout

Release notes

The following deprecated fields are being removed from all batch task tuning configs:

  • maxRowsPerSegment
  • numShards
  • maxTotalRows
  • publishTimeout

Instead of these fields, users can explicitly specify the desired partitionsSpec in the tuningConfig.

The following configs are affected by this change:

  • tuning config for index_parallel, index and compact tasks
  • datasource compaction config posted on API /druid/coordinator/v1/config/compaction

Streaming task tuning configs will not be affected by this change.

This PR has:

  • been self-reviewed.
  • added documentation for new or modified features or behaviors.
  • a release note entry in the PR description.
  • added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
  • added or updated version, license, or notice information in licenses.yaml
  • added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
  • added integration tests.
  • been tested in a test Druid cluster.

@kfaraz kfaraz requested a review from clintropolis July 29, 2024 04:38
@github-actions github-actions Bot added Area - Batch Ingestion Kubernetes Area - Ingestion Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 labels Jul 29, 2024
github-advanced-security[bot]
github-advanced-security AI found potential problems Jul 29, 2024
View reviewed changes
Comment thread ...est/java/org/apache/druid/indexing/common/task/batch/parallel/HashPartitionTaskKillTest.java
ParallelIndexTaskRunner<T, R> retVal;
if (succeedsBeforeFailing == 0
&& this.getIngestionSchema().getTuningConfig().getNumShards() == null) {
&& partitionsSpec == null || partitionsSpec.getNumShards() == null) {

Check warning

Code scanning / CodeQL

Dereferenced variable may be null

Variable [partitionsSpec](1) may be null at this access as suggested by [this](2) null guard. Variable [partitionsSpec](1) may be null at this access as suggested by [this](3) null guard.
@kfaraz kfaraz marked this pull request as draft July 30, 2024 09:31
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Sep 29, 2024

This pull request has been marked as stale due to 60 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If you think
that's incorrect or this pull request should instead be reviewed, please simply
write any comment. Even if closed, you can still revive the PR at any time or
discuss it on the dev@druid.apache.org list.
Thank you for your contributions.

@github-actions github-actions Bot added the stale label Sep 29, 2024
@kfaraz kfaraz removed the stale label Sep 29, 2024
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Nov 29, 2024

This pull request has been marked as stale due to 60 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If you think
that's incorrect or this pull request should instead be reviewed, please simply
write any comment. Even if closed, you can still revive the PR at any time or
discuss it on the dev@druid.apache.org list.
Thank you for your contributions.

@github-actions github-actions Bot added the stale label Nov 29, 2024
@kfaraz kfaraz removed the stale label Dec 17, 2024
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Feb 16, 2025

This pull request has been marked as stale due to 60 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If you think
that's incorrect or this pull request should instead be reviewed, please simply
write any comment. Even if closed, you can still revive the PR at any time or
discuss it on the dev@druid.apache.org list.
Thank you for your contributions.

@github-actions github-actions Bot added the stale label Feb 16, 2025
@kfaraz kfaraz removed the stale label Feb 16, 2025
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 18, 2025

This pull request has been marked as stale due to 60 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If you think
that's incorrect or this pull request should instead be reviewed, please simply
write any comment. Even if closed, you can still revive the PR at any time or
discuss it on the dev@druid.apache.org list.
Thank you for your contributions.

@github-actions github-actions Bot added the stale label Apr 18, 2025
@kfaraz kfaraz removed the stale label Apr 18, 2025
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Jun 18, 2025

This pull request has been marked as stale due to 60 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If you think
that's incorrect or this pull request should instead be reviewed, please simply
write any comment. Even if closed, you can still revive the PR at any time or
discuss it on the dev@druid.apache.org list.
Thank you for your contributions.

@github-actions github-actions Bot added the stale label Jun 18, 2025
@kfaraz kfaraz removed the stale label Jul 14, 2025
@kfaraz
Copy link
Copy Markdown
Contributor Author

kfaraz commented Jul 14, 2025

This PR has too many conflicts now but keeping it around to refer to the set of changes, when needed.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Sep 13, 2025

This pull request has been marked as stale due to 60 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If you think
that's incorrect or this pull request should instead be reviewed, please simply
write any comment. Even if closed, you can still revive the PR at any time or
discuss it on the dev@druid.apache.org list.
Thank you for your contributions.

@github-actions github-actions Bot added the stale label Sep 13, 2025
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Oct 11, 2025

This pull request/issue has been closed due to lack of activity. If you think that
is incorrect, or the pull request requires review, you can revive the PR at any time.

@github-actions github-actions Bot closed this Oct 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@clintropolis clintropolis Awaiting requested review from clintropolis

Assignees

No one assigned

Labels

Area - Batch Ingestion Area - Documentation Area - Ingestion Area - MSQ For multi stage queries - https://github.com/apache/druid/issues/12262 Design Review Kubernetes stale

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@kfaraz @github-advanced-security