From 8559debed80e22afeaaca342663d46279016f531 Mon Sep 17 00:00:00 2001 From: brian Date: Thu, 9 Apr 2026 12:12:33 -0700 Subject: [PATCH 01/21] docs: druid 37 release notes --- docs/release-info/release-notes.md | 442 ++++++++++++++++++++++++++++- website/.spelling | 2 + 2 files changed, 442 insertions(+), 2 deletions(-) diff --git a/docs/release-info/release-notes.md b/docs/release-info/release-notes.md index 7698573f110e..ead6f35e4ea7 100644 --- a/docs/release-info/release-notes.md +++ b/docs/release-info/release-notes.md @@ -57,50 +57,488 @@ For tips about how to write a good release note, see [Release notes](https://git This section contains important information about new and existing features. +### Hadoop-based ingestion + +Support for Hadoop-based ingestion has been removed. The feature was deprecated in Druid 34. + +Use one of Druid's other supported ingestion methods, such as SQL-based ingestion or MiddleManager-less ingestion using Kubernetes. + +[#19109](https://github.com/apache/druid/pull/19109) + +### Query blocklist + +You can now create a query blocklist for dynamically blocking queries without restarting your deployment. You can block queries by datasource, query type, or query context using the new `/druid/coordinator/v1/config/broker` API. Block rules use `AND` logic, which means all criteria must match. Druid stores the rules in the metadata database. + +The following example blocks all groupBy queries on the `wikipedia` datasource with a query context parameter of `priority` equal to `0`: + +``` +POST /druid/coordinator/v1/config/broker + { + "queryBlocklist": [ + { + "ruleName": "block-wikipedia-groupbys", + "dataSources": ["wikipedia"], + "queryTypes": ["groupBy"], + "contextMatches": {"priority": "0"} + } + ] + } +``` + +[#19011](https://github.com/apache/druid/pull/19011) + +### Minor compaction for Overlord-based compaction (experimental) + +You can now configure minor compaction. With minor compaction, Druid only compacts newly ingested segments while upgrading existing compacted segments. When Druid upgrades segments, it updates the metadata instead of using resources to compact it again. Support is currently limited to Overlord-based compaction supervisors using the MSQ task engine. + +Use the `mostFragmentedFirst` compaction policy and set either a percentage of rows-based or byte-based threshold for minor compaction. + +[#19059](https://github.com/apache/druid/pull/19059) [#19205](https://github.com/apache/druid/pull/19205) + +### Thrift input format + +As part of the Thrift contributor extension, Druid now supports Thrift-encoded data for Kafka and Kinesis streaming ingestion. + +[#19111](https://github.com/apache/druid/pull/19111) + +To use this feature, you must add `druid-thrift-extensions` to your extension load list. + +### Kubernetes-based task management + +This extension is now generally available. + +[#19128](https://github.com/apache/druid/pull/19128) + +### Tombstones + +Tombstones for JSON-based native batch ingestion (the `dropExisting` flag for `ioConfig`) are now generally available. + +[#19128](https://github.com/apache/druid/pull/19128) + +### Dynamic default query context + +You can now add default query context parameters as a dynamic configuration to the Broker. This allows you to override static defaults set in your runtime properties without restarting your deployment or having to update multiple queries individually. Druid resolves which query context parameters to use based on the following priority: + +1. The query context included with the query +1. The query context set as a dynamic configuration on the Broker +1. The query context parameters set in the runtime properties +1. The defaults that ship with Druid + +Note that like other Broker dynamic configuration, this is best-effort. Settings may not be applied in certain +cases, such as when a Broker has recently started and hasn't received the config yet, or if the +Broker can't contact the Coordinator. If a query context parameter is critical for all your queries, set it in the runtime properties. + +[#19146](https://github.com/apache/druid/pull/19146) + +### `sys.queries` table (experimental) + +The new system queries table provides information about currently running and recently completed queries that use the Dart engine. This table is off by default. To enable the table, set the following config: + +``` +druid.sql.planner.enableSysQueriesTable = true +``` + +As part of this change, the `/druid/v2/sql/queries` API now supports an `includeComplete` parameter that shows recently completed queries. + +[#18923](https://github.com/apache/druid/pull/18923) + +### Auto-compaction with compaction supervisors + +Auto-compaction using compaction supervisors has been improved and is now generally available. + +As part of the improvement, instead of individual compacted segments having to store their full compaction state in the metadata store, states are now stored in a central location, a new `indexingStates` table, in the metadata store. Individual segments only need to store a unique reference (`indexing_state_fingerprint`) to their full compaction state. + +Since many segments in a single datasource share the same underlying compaction state, this greatly reduces metadata storage requirements for automatic compaction. + +For backwards compatibility, detailed compaction state will continue to be persisted in each segment for now. This will be removed in a future release. + +You can stop storing detailed compaction state by setting `storeCompactionStatePerSegment` to `false` in the cluster compaction config. If you turn it off and need to downgrade, Druid needs to re-compact any segments that have been compacted since you changed the config. + +This change has upgrade impacts for metadata storage and metadata caching. For more information, see the [Metadata storage for auto-compaction with compaction supervisors](#metadata-storage-for-auto-compaction-with-compaction-supervisors) upgrade note. + +[#19113](https://github.com/apache/druid/pull/19113) [#18844](https://github.com/apache/druid/pull/18844) + +### Broker tier selection for realtime servers + +Added `druid.broker.realtime.select.tier` and `druid.broker.realtime.balancer.type` on the Brokers to optionally override the Broker’s tier selection and balancer strategies for realtime servers. If these properties are not set (the default), realtime servers continue to use the existing `druid.broker.select` and `druid.broker.balancer` configurations that apply to both historical and realtime servers. + +[#19062](https://github.com/apache/druid/pull/19062) + +### Manual Broker routing in the web console + +You can now configure which Broker the Router uses for queries issued from the web console. You may want to do this if there are Brokers that don't have visibility into certain data tiers, and you know you're querying data available only on a certain tier. + +To specify a Broker, add the following config to `web-console/console-config.js`: + +```js +consoleBrokerService: 'druid/BROKER_NAME' +``` + +[#19069](https://github.com/apache/druid/pull/19069) + ## Functional area and related changes This section contains detailed release notes separated by areas. ### Web console +#### Changed storage column displays + +- Renamed **Current size** to **Assigned size**. +- Renamed **Max size** to **Effective size**. It now displays the smaller value between `max_size` and `storage_size`. The max size is still shown as a tooltip. +- Changed usage calculation to use `effective_size` + +[#19007](https://github.com/apache/druid/pull/19007) + #### Other web console improvements +- Added `workerDesc` to `WorkerStats`, which makes it easier to identify where a worker is running [#19171](https://github.com/apache/druid/pull/19171) +- Added the Dart unique execution ID (`dartQueryId`) and the `sqlQueryId` to the **Details** pane in the web console [#19185](https://github.com/apache/druid/pull/19185) +- Added support for showing completed Dart queries in the web console [#18940](https://github.com/apache/druid/pull/18940) +- Added a detail dialog to the **Services** page [#18960](https://github.com/apache/druid/pull/18960) +- Added icons to indicate when data is loaded into virtual storage, including a tooltip that shows all the counters for the data [#19010](https://github.com/apache/druid/pull/19010) +- Added support for Dart reports [#18897](https://github.com/apache/druid/pull/18897) +- Changed what's considered an active worker: any nonzero rows, files, bytes, frames, or wall time is enough to consider a worker active [#19183](https://github.com/apache/druid/pull/19183) +- Changed the **Cancel query** option to show only if a query is in an accepted or running state [#19182](https://github.com/apache/druid/pull/19182) + ### Ingestion #### SQL-based ingestion ##### Other SQL-based ingestion improvements +- Added support for virtual storage fabric when performing SQL-based ingestion [#18873](https://github.com/apache/druid/pull/18873) +- Added support for `StorageMonitor` so that MSQ task engine tasks always emit `taskId` and `groupId` [#19048](https://github.com/apache/druid/pull/19048) +- Improved worker cancellation [#18931](https://github.com/apache/druid/pull/18931) + #### Streaming ingestion +##### Changed how tasks get launched for autoscaling + +The behavior of `taskCountMin` and `taskCountStart` has been changed for autoscaling. Druid now computes the initial number of tasks to launch by checking the configs in the following order: `taskCountStart` (optional), then `taskCount` (in `ioConfig`), then `taskCountMin`. + +[#19091](https://github.com/apache/druid/pull/19091) + +##### Improved query isolation + +Added `serverPriorityToReplicas` parameter to the streaming supervisor specs (Kafka, Kinesis, and Rabbit). This allows operators to distribute task replicas across different server priorities for realtime indexing tasks. Similar to Historical tiering, this enables query isolation for mixed workload scenarios on the Peons, allowing some task replicas to handle queries of specific priorities. + +[#19040](https://github.com/apache/druid/pull/19040) + ##### Other streaming ingestion improvements +- Improved cost-based autoscaler performance in high lag scenarios [#19045](https://github.com/apache/druid/pull/19045) + + +#### Ingestion improvements + +- Sped up task scheduling on the Overlord [#19199](https://github.com/apache/druid/pull/19199) + ### Querying +#### groupBy query configuration + +Added a new groupBy query configuration property `druid.query.groupBy.maxSpillFileCount` to limit the maximum number of spill files created per query. When the limit is exceeded, the query fails with a clear error message instead of causing Historical nodes to run out of memory during spill file merging. The limit can also be overridden per query using the query context parameter `maxSpillFileCount`. + +[#19141](https://github.com/apache/druid/pull/19141) + +#### Improved handling of nested aggregates + +Druid can now merge two Aggregates with a Projection between them. For example, the following query: + +```sql +SELECT + hr, + UPPER(t1.x) x, + SUM(t1.cnt) cnt, + MIN(t1.mn) mn, + MAX(t1.mx) mx +FROM ( + SELECT + floor(__time to hour) hr, + dim2 x, + COUNT(*) cnt, + MIN(m1 * 5) mn, + MAX(m1 + m2) mx + FROM druid.foo + WHERE dim2 IN ('abc', 'def', 'a', 'b', '') + GROUP BY 1, 2 +) t1 +WHERE t1.x IN ('abc', 'foo', 'bar', 'a', '') +GROUP BY 1, 2 +``` + +can be simplified to the following: + +```sql +SELECT + FLOOR(__time TO hour) hr, + UPPER(dim2) x, + COUNT(*) cnt, + MIN(m1 * 5) mn, + MAX(m1 + m2) mx +FROM druid.foo +WHERE dim2 IN ('abc', 'a', '') +GROUP BY 1, 2 +``` + +[#18498](https://github.com/apache/druid/pull/18498) + #### Other querying improvements +- Added `durationMs` to Dart query reports [#19169](https://github.com/apache/druid/pull/19169) +- Improved error handling so that row signature column order is preserved when column analysis encounters an error [#19162](https://github.com/apache/druid/pull/19162) +- Improved GROUP BY performance [#18952](https://github.com/apache/druid/pull/18952) +- Improved expression filters to take advantage of specialized virtual columns when possible, resulting in better performance for the query [#18965](https://github.com/apache/druid/pull/18965) + ### Cluster management +#### New Broker tier selection strategies + +Operators can now configure two new Broker `TierSelectorStrategy` implementations: + +- `strict` - Only selects servers whose priorities match the configured list. Example configuration: `druid.broker.select.tier=strict` and `druid.broker.select.tier.strict.priorities=[1]`. +- `pooled` - Pools servers across the configured priorities and selects among them, allowing queries to utilize multiple priority tiers for improved availability. Example configuration: `druid.broker.select.tier=pooled` and `druid.broker.select.tier.pooled.priorities=[2,1]`. + +These strategies can also be configured for realtime servers using `druid.broker.realtime.select.tier`. + +[#19094](https://github.com/apache/druid/pull/19094) + +#### Druid operator + +The Druid operator now resides in its own repository: [`apache/druid-operator`](https://github.com/apache/druid-operator). + +[#19156](https://github.com/apache/druid/pull/19156) + +#### Cost-based autoscaler algorithm + +The algorithm for cost-based autoscaling has been changed: + +- Scale up more aggressively when per-partition lag is meaningful +- Relax the partitions-per-task increase limit based on lag severity and headroom +- Keep behavior conservative near `taskCountMax` and avoid negative headroom effects + +[#18936](https://github.com/apache/druid/pull/18936) + #### Other cluster management improvements +- Added `/status/ready` endpoint for service health so that external load balancers can handle a graceful shutdown better [#19148](https://github.com/apache/druid/pull/19148) +- Added a configurable option to scale-down during task run time for cost-based autoscaler [#18958](https://github.com/apache/druid/pull/18958) +- Added `storage_size` to `sys.servers` to facilitate retrieving disk cache size for Historicals when using the virtual storage fabric [#18979](https://github.com/apache/druid/pull/18979) +- Added a log for new task count computation for the cost-based auto scaler [#18929](https://github.com/apache/druid/pull/18929) +- Changed how the scaling is calculated from a square root-based scaling formula to a logarithmic formula that provides better emergency recovery at low task counts and millions of lag [#18976](https://github.com/apache/druid/pull/18976) +- Improved the load speed of cached segments during Historical startup [#18489](https://github.com/apache/druid/pull/18489) +- Improved Broker startup time by parallelizing buffer initialization [#19025](https://github.com/apache/druid/pull/19025) +- Improved the stack trace for MSQ task engine worker failures so that they're preserved [#19049](https://github.com/apache/druid/pull/19049) +- Improved the performance of the cost-based autoscaler during loaded lag conditions [#18991](https://github.com/apache/druid/pull/18991) + ### Data management +#### Durable storage cleaner + +The durable storage cleaner now supports configurable time-based retention for MSQ query results. Previously, query results were retained for all known tasks, which was unreliable for completed tasks. With this change, query results are retained for a configurable time period based on the task creation time. + +The new configuration property `druid.msq.intermediate.storage.cleaner.durationToRetain` controls the retention period for query results. The default retention period is 6 hours. + +[#19074](https://github.com/apache/druid/pull/19074) + #### Other data management improvements +- Changed the default value for `druid.indexing.formats.maxStringLength` to null from 0 [#19198](https://github.com/apache/druid/pull/19198) + ### Metrics and monitoring +#### Monitoring supervisor state + +Added a new `supervisor/count` metric when `SupervisorStatsMonitor` is enabled in `druid.monitoring.monitors`. The metric reports each supervisor’s state, such as `RUNNING` or `SUSPENDED`, for Prometheus, StatsD, and other metric systems. + +[#19114](https://github.com/apache/druid/pull/19114) + +#### Improved `groupBy` metrics + +`GroupByStatsMonitor` now provides the following metrics: + +- `mergeBuffer/bytesUsed` +- `mergeBuffer/maxAcquisitionTimeNs` +- `groupBy/maxSpilledBytes` +- `groupBy/maxMergeDictionarySize` + +[#18731](https://github.com/apache/druid/pull/18731) [#18970](https://github.com/apache/druid/pull/18970) [#18934](https://github.com/apache/druid/pull/18934) + +#### Filtering metrics + +Operators can limit which metrics the logging emitter writes by setting `druid.emitter.logging.shouldFilterMetrics=true` and, if desired, `druid.emitter.logging.allowedMetricsPath` to a JSON object file where the keys are metric names. A missing custom file results in a warning and use of the bundled `defaultMetrics.json`. Alerts and other non-metric events are always logged. + +[#19030](https://github.com/apache/druid/pull/19030) + +#### New Broker metrics + +Added `segment/schemaCache/rowSignature/changed` and `segment/schemaCache/rowSignature/column/count` metrics to get visibility into when the Broker's segment metadata cache's row signature for each datasource is initialized and updated. + +[#18966](https://github.com/apache/druid/pull/18966) + +#### Other metrics and monitoring improvements + +- Added the following metrics to the default for Prometheus: `mergeBuffer/bytesUsed` and `mergeBuffer/maxBytesUsed` [#19110](https://github.com/apache/druid/pull/19110) +- Added compaction mode to what gets emitted for the `compact/task/count` metric [#19151](https://github.com/apache/druid/pull/19151) +- Added support for logging and emitting SQL dynamic parameter values [#19067](https://github.com/apache/druid/pull/19067) +- Added `ingest/rows/published`, which gets emitted from all task types to denote the total row count of successfully published segments [#19177](https://github.com/apache/druid/pull/19177) +- Added `queries` and `totalQueries` counters, which reflect queries made to realtime servers to get realtime data +- Added `tier/storage/capacity` metric for the Coordinator to emit that is guaranteed to reflect the total `StorageLocation` size configured across all Historicals in a tier [#18962](https://github.com/apache/druid/pull/18962) +- Added new metrics for virtual storage fabric to the MSQ task engine `ChannelCounters`: `loadBytes`, `loadTime`, `loadWait`, and `loadFiles` [#18971](https://github.com/apache/druid/pull/18971) +- Added `storage/virtual/hit/bytes` metric to `StorageMonitor` +[#18895](https://github.com/apache/druid/pull/18895) +- Added `supervisorId` dimension for streaming tasks to `TaskCountStatsMonitor` [#18920](https://github.com/apache/druid/pull/18920) +- Changed `StorageMonitor` to always be on [#19048](https://github.com/apache/druid/pull/19048) +- Improved the metrics for autoscalers, so that they all emit the same metrics: `supervisorId`, `dataSource`, and `stream` [#19097](https://github.com/apache/druid/pull/19097) + ### Extensions -### Documentation improvements +#### gRPC + +The gRPC query extension now cancels in-flight queries when clients cancel or disconnect. + +[#19005](https://github.com/apache/druid/pull/19005) + +#### Iceberg + +You can now configure residual filters for non-partition columns when using the Iceberg input source. Set `residualFilterMode` in the Iceberg input source to one of the following: + +- `ignore`: (default) Ingest the residual rows with a warning log unless filtered by `transformSpec`. +- `fail`: Fail the ingestion job when residual filters are detected. Use this to ensure that filters only target partition columns. + +[#18953](https://github.com/apache/druid/pull/18953) + +#### HDFS storage + +Added support for `lz4` compression. As part of this change, the following metrics are now available: + +- `hdfs/pull/size` +- `hdfs/pull/duration` +- `hdfs/push/size` +- `hdfs/push/duration` + +[#18982](https://github.com/apache/druid/pull/18982) ## Upgrade notes and incompatible changes ### Upgrade notes +#### Hadoop-based ingestion + +Support for Hadoop-based ingestion has been removed. The feature was deprecated in Druid 34. + +Use one of Druid's other supported ingestion methods, such as SQL-based ingestion or MiddleManager-less ingestion using Kubernetes. + +[#19109](https://github.com/apache/druid/pull/19109) + +#### Segment metadata cache on by default + +Starting in Druid 37, the segment metadata cache is on by default. This feature allows the Broker to cache segment metadata polled from the Coordinator, rather than having to fetch metadata for every query against the `sys.segments` table. This improves performance but increases memory usage on Brokers. + +The `druid.sql.planner.metadataSegmentCacheEnable` config controls this feature. + +[#19075](https://github.com/apache/druid/pull/19075) + +#### Streaming ingestion `parser` + +Support for the deprecated `parser` has been removed for streaming ingest tasks such as Kafka and Kinesis. Operators must now specify `inputSource`/`inputFormat` on the `ioConfig` of the supervisor spec, and the `dataSchema` must not specify a parser. Do this before upgrading to Druid 37 or newer. + +[#19173](https://github.com/apache/druid/pull/19173) [#19166](https://github.com/apache/druid/pull/19166) + +#### Rolling upgrades from Druid versions prior to version 0.23 + +You can't perform a rolling upgrade from versions earlier than Druid 0.23. + +[#18961](https://github.com/apache/druid/pull/18961) + +#### Metadata storage for auto-compaction with compaction supervisors + +Automatic compaction using compaction supervisors now requires incremental segment metadata caching to be enabled on the Overlord and Coordinator via runtime properties. Specifically, the `druid.manager.segments.useIncrementalCache` config must be set to `always` or `ifSynced`. For more information about the config, see [Segment metadata cache](https://druid.apache.org/docs/latest/configuration/#segment-metadata-cache-experimental). + +Additionally, metadata store changes are required for this upgrade. + +If you already have `druid.metadata.storage.connector.createTables` set to `true`, no action is needed. + +If you have this feature turned off, you will need to alter the segments table and create the `compactionStates` table. The Postgres DDL is provided below as a guide: + +``` +-- create the indexing states lookup table and associated indices +CREATE TABLE druid_indexingStates ( + created_date VARCHAR(255) NOT NULL, + datasource VARCHAR(255) NOT NULL, + fingerprint VARCHAR(255) NOT NULL, + payload BYTEA NOT NULL, + used BOOLEAN NOT NULL, + pending BOOLEAN NOT NULL, + used_status_last_updated VARCHAR(255) NOT NULL, + PRIMARY KEY (fingerprint), + ); + + CREATE INDEX idx_druid_compactionStates_used ON druid_compactionStates(used, used_status_last_updated); +``` + +``` +-- modify druid_segments table to have a column for storing compaction state fingerprints +ALTER TABLE druid_segments ADD COLUMN indexing_state_fingerprint VARCHAR(255); +``` + +You may have to adapt the syntax to fit your table naming prefix and metadata store backend. + +[#18844](https://github.com/apache/druid/pull/18844) + +#### Segment locking + +Segment locking and `NumberedOverwriteShardSpec` are deprecated and will be removed in a future release. Use time chunk locking instead. You can make sure only time chunk locking is used by setting `druid.indexer.tasklock.forceTimeChunkLock` to `true`. + +[#19050](https://github.com/apache/druid/pull/19050) + ### Incompatible changes +#### Removed `defaultProcessingRate` config + +This config allowed scaling actions to begin prior to the first metrics becoming available. + +[#19028](https://github.com/apache/druid/pull/19028) + +#### Front-coding format + +Druid now defaults to v1 of the front-coded format instead of version 0 if enabled. Version 1 was introduced in Druid 26. Downgrading to or upgrading from a version of Druid prior to 26 may require reindexing if you have front-coding enabled with version 0. + +[#18984](https://github.com/apache/druid/pull/18984) + ### Developer notes +- Added `AGENTS.md` [#19084](https://github.com/apache/druid/pull/19084) +- Added a requirement to use [conventional commit syntax](https://www.conventionalcommits.org/en/v1.0.0/) [#19089](https://github.com/apache/druid/pull/19089) +- Updated `checkstyle` from `3.0.0` to `3.6.0` [#19064](https://github.com/apache/druid/pull/19064) + #### Dependency updates -The following dependencies have had their versions bumped: \ No newline at end of file +The following dependencies have been updated: + +- Added `software.amazon.awssdk` to support `WebIdentityTokenProvider` [#19178](https://github.com/apache/druid/pull/19178) +- `org.apache.iceberg` from `1.6.1` to `1.7.2` [#19172](https://github.com/apache/druid/pull/19172) +- `diff` node module from 4.0.1 to 4.0.4 [#18933](https://github.com/apache/druid/pull/18933) +- `org.apache.avro` from 1.11.4 to 1.11.5 [#19103](https://github.com/apache/druid/pull/19103) +- `bytebuddy` from `1.17.7` to `1.18.3` [#19000](https://github.com/apache/druid/pull/19000) +- `slf4j` from `2.0.16` to `2.0.17` [#18990](https://github.com/apache/druid/pull/18990) +- Apache Commons Codec from `1.16.1` to `1.17.1` [#18990](https://github.com/apache/druid/pull/18990) +- `jacoco` from `0.8.12` to `0.8.14` [#18990](https://github.com/apache/druid/pull/18990) +- `docker-java-bom` from `3.6.0` to `3.7.0` [#18990](https://github.com/apache/druid/pull/18990) +- `assertj-core` from `3.24.2` to `3.27.7` [#18994](https://github.com/apache/druid/pull/18994) +- `maven-surefire-plugin` from `3.2.5` to `3.5.4` [#18847](https://github.com/apache/druid/pull/18847) +- `guice` from `5.1.0` to `6.0.0` [#18986](https://github.com/apache/druid/pull/18986) +- JDK compiler from 11 to 17 [#18977](https://github.com/apache/druid/pull/18977) +- `vertx` from `4.5.14` to `4.5.24` [#18947](https://github.com/apache/druid/pull/18947) +- `fabric8` from `7.4.0` to `7.5.2` [#18947](https://github.com/apache/druid/pull/18947) +- `mockito` from `5.14.2` to `5.23` [#19145](https://github.com/apache/druid/pull/19145) +- `easymock` from `5.2.0` to `5.6.0` [#19145](https://github.com/apache/druid/pull/19145) +- `equalsverifier` from `3.15.8` to `4.4.1` [#19145](https://github.com/apache/druid/pull/19145) +- `bytebuddy` from `1.18.3` to `1.18.5` [#19145](https://github.com/apache/druid/pull/19145) +- Added `objenesis` `3.5` [#19145](https://github.com/apache/druid/pull/19145) +- `org.apache.zookeeper` from 3.8.4 to 3.8.6 [#19135](https://github.com/apache/druid/pull/19135) +- Added AWS SDK `2.40.0` [#18891](https://github.com/apache/druid/pull/18891) +- `com.lmax.disruptor` from `3.3.6` to `3.4.4` [#19122](https://github.com/apache/druid/pull/19122) +- `org.junit.junit-bom` from `5.13.3` to `5.14.3` [#19122](https://github.com/apache/druid/pull/19122) +- Several Kubernetes related dependencies have been updated. For more information, see [#19071](https://github.com/apache/druid/pull/19071) \ No newline at end of file diff --git a/website/.spelling b/website/.spelling index 99e767c48910..d4569666af32 100644 --- a/website/.spelling +++ b/website/.spelling @@ -120,6 +120,7 @@ GoogleCloudStorageInputSource GSSAPI GUIs GroupBy +gRPC Guice HDFS HLL @@ -822,6 +823,7 @@ InvalidNullByte QueryNotSupported QueryNotSupported RowTooLarge +tooltip TooManyBuckets TooManyInputFiles TooManyPartitions From 0a084f0527f6709786d63dd566ca0709ec0477c6 Mon Sep 17 00:00:00 2001 From: brian Date: Thu, 9 Apr 2026 12:14:55 -0700 Subject: [PATCH 02/21] upgrade notes --- docs/release-info/upgrade-notes.md | 153 ++++++++++++++++++++++++++++- 1 file changed, 152 insertions(+), 1 deletion(-) diff --git a/docs/release-info/upgrade-notes.md b/docs/release-info/upgrade-notes.md index 15b75867876c..371a14b7838d 100644 --- a/docs/release-info/upgrade-notes.md +++ b/docs/release-info/upgrade-notes.md @@ -28,6 +28,157 @@ For the full release notes for a specific version, see the [releases page](https ## Announcements +## 37.0.0 + +### Upgrade notes + +#### Hadoop-based ingestion + +Support for Hadoop-based ingestion has been removed. The feature was deprecated in Druid 34. + +Use one of Druid's other supported ingestion methods, such as SQL-based ingestion or MiddleManager-less ingestion using Kubernetes. + +[#19109](https://github.com/apache/druid/pull/19109) + +#### Segment metadata cache on by default + +Starting in Druid 37, the segment metadata cache is on by default. This feature allows the Broker to cache segment metadata polled from the Coordinator, rather than having to fetch metadata for every query against the `sys.segments` table. This improves performance but increases memory usage on Brokers. + +The `druid.sql.planner.metadataSegmentCacheEnable` config controls this feature. + +[#19075](https://github.com/apache/druid/pull/19075) + +#### Streaming ingestion `parser` + +Support for the deprecated `parser` has been removed for streaming ingest tasks such as Kafka and Kinesis. Operators must now specify `inputSource`/`inputFormat` on the `ioConfig` of the supervisor spec, and the `dataSchema` must not specify a parser. Do this before upgrading to Druid 37 or newer. + +[#19173](https://github.com/apache/druid/pull/19173) [#19166](https://github.com/apache/druid/pull/19166) + +#### Rolling upgrades from Druid versions prior to version 0.23 + +You can't perform a rolling upgrade from versions earlier than Druid 0.23. + +[#18961](https://github.com/apache/druid/pull/18961) + +#### Metadata storage for auto-compaction with compaction supervisors + +Automatic compaction using compaction supervisors now requires incremental segment metadata caching to be enabled on the Overlord and Coordinator via runtime properties. Specifically, the `druid.manager.segments.useIncrementalCache` config must be set to `always` or `ifSynced`. For more information about the config, see [Segment metadata cache](https://druid.apache.org/docs/latest/configuration/#segment-metadata-cache-experimental). + +Additionally, metadata store changes are required for this upgrade. + +If you already have `druid.metadata.storage.connector.createTables` set to `true`, no action is needed. + +If you have this feature turned off, you will need to alter the segments table and create the `compactionStates` table. The Postgres DDL is provided below as a guide: + +``` +-- create the indexing states lookup table and associated indices +CREATE TABLE druid_indexingStates ( + created_date VARCHAR(255) NOT NULL, + datasource VARCHAR(255) NOT NULL, + fingerprint VARCHAR(255) NOT NULL, + payload BYTEA NOT NULL, + used BOOLEAN NOT NULL, + pending BOOLEAN NOT NULL, + used_status_last_updated VARCHAR(255) NOT NULL, + PRIMARY KEY (fingerprint), + ); + + CREATE INDEX idx_druid_compactionStates_used ON druid_compactionStates(used, used_status_last_updated); +``` + +``` +-- modify druid_segments table to have a column for storing compaction state fingerprints +ALTER TABLE druid_segments ADD COLUMN indexing_state_fingerprint VARCHAR(255); +``` + +You may have to adapt the syntax to fit your table naming prefix and metadata store backend. + +[#18844](https://github.com/apache/druid/pull/18844) + +#### Segment locking + +Segment locking and `NumberedOverwriteShardSpec` are deprecated and will be removed in a future release. Use time chunk locking instead. You can make sure only time chunk locking is used by setting `druid.indexer.tasklock.forceTimeChunkLock` to `true`. + +[#19050](https://github.com/apache/druid/pull/19050) + +### Incompatible changes + +#### Removed `defaultProcessingRate` config + +This config allowed scaling actions to begin prior to the first metrics becoming available. + +[#19028](https://github.com/apache/druid/pull/19028) + +#### Front-coding format + +Druid now defaults to v1 of the front-coded format instead of version 0 if enabled. Version 1 was introduced in Druid 26. Downgrading to or upgrading from a version of Druid prior to 26 may require reindexing if you have front-coding enabled with version 0. + +[#18984](https://github.com/apache/druid/pull/18984) + +## 36.0.0 + +### Upgrade notes + +#### Deprecated metrics + +Monitors on peons that previously emitted the `id` dimension from `JettyMonitor`, `OshiSysMonitor`, `JvmMonitor`, `JvmCpuMonitor`, `JvmThreadsMonitor` and `SysMonitor` to represent the task ID are deprecated and will be removed in a future release. Use the `taskId` dimension instead. + +[#18709](https://github.com/apache/druid/pull/18709) + +#### Some `statsd` metrics removed + +The following obsolete metrics have been removed: + +* `segment/cost/raw` +* `segment/cost/normalized` +* `segment/cost/normalization` + +[#18846](https://github.com/apache/druid/pull/18846) + +## 35.0.0 + +### Upgrade notes + +#### Fallback vectorization on by default + +The `druid.expressions.allowVectorizeFallback` now defaults to `true`. Additionally, `SAFE_DIVIDE` can now vectorize as a fallback. + +[#18549](https://github.com/apache/druid/pull/18549) + +#### Java 11 support removed + +Upgrade to Java 17 or 21. Note that some versions of Java 21 encountered issues during test, specifically Java 21.05-21.07. If possible, avoid these versions. + +[#18424](https://github.com/apache/druid/pull/18424) + +#### Jetty 12 + +A new server configuration option has been added: `druid.server.http.uriCompliance`. Jetty 12 by default has strict enforcement of `RFC3986` URI format. This is a change from Jetty 9. To retain compatibility with legacy Druid, this config defaults to `LEGACY`, which uses the more permissive URI format enforcement that Jetty 9 used. If the cluster you operate does not require legacy compatibility, we recommend you use the upstream Jetty default of `RFC3986` in your Druid deployment. See the jetty documentation for more info. + +[#18424](https://github.com/apache/druid/pull/18424) + +#### Kerberos authentication + +The `druid.auth.authenticator.kerberos.cookieSignatureSecret` config is now mandatory. + +[#18368](https://github.com/apache/druid/pull/18368) + +#### Multi-stage query task engine + +The MSQ task engine is now a core capability of Druid rather than an extension. It has been in the default extension load list for several releases. + +Remove `druid-multi-stage-query` from `druid.extensions.loadList` in `common.runtimes.properties` before you upgrade. + +Druid 35.0.0 will ignore the extension if it's in the load list. Future versions of Druid will fail to start since it cannot locate the extension. + +[#18394](https://github.com/apache/druid/pull/18394) + +#### pac4j extension + +Due to the upgrade from `pac4j` 4 to 5, session serialization has changed from `pac4j`’s `JavaSerializer` to standard Java serialization. As a result, clients of clusters using the `pac4j` extension may be logged out during rolling upgrades and need to re‑authenticate. + +[#18259](https://github.com/apache/druid/pull/18259) + ## 34.0.0 ### Upgrade notes @@ -970,4 +1121,4 @@ As [ZooKeeper 3.4 has been end-of-life for a while](https://mail-archives.apache All columns in the `sys.segments` table are now serialized in the JSON format to make them consistent with other system tables. Column names now use the same "snake case" convention. -[#10481](https://github.com/apache/druid/pull/10481) +[#10481](https://github.com/apache/druid/pull/10481) \ No newline at end of file From 2c2078497cdf9657dc687b5b3dfc7996417121a2 Mon Sep 17 00:00:00 2001 From: brian Date: Fri, 10 Apr 2026 09:35:28 -0700 Subject: [PATCH 03/21] docs: 37.0.0 release notes --- docs/release-info/release-notes.md | 73 ++++++++++++++++++++++++++++-- docs/release-info/upgrade-notes.md | 9 ++++ 2 files changed, 77 insertions(+), 5 deletions(-) diff --git a/docs/release-info/release-notes.md b/docs/release-info/release-notes.md index ead6f35e4ea7..c53b00de1f70 100644 --- a/docs/release-info/release-notes.md +++ b/docs/release-info/release-notes.md @@ -89,11 +89,11 @@ POST /druid/coordinator/v1/config/broker ### Minor compaction for Overlord-based compaction (experimental) -You can now configure minor compaction. With minor compaction, Druid only compacts newly ingested segments while upgrading existing compacted segments. When Druid upgrades segments, it updates the metadata instead of using resources to compact it again. Support is currently limited to Overlord-based compaction supervisors using the MSQ task engine. +You can now configure minor compaction. With minor compaction, Druid only compacts newly ingested segments while upgrading existing compacted segments. When Druid upgrades segments, it updates the metadata instead of using resources to compact it again. You can use the native compaction engine or the MSQ task engine. Use the `mostFragmentedFirst` compaction policy and set either a percentage of rows-based or byte-based threshold for minor compaction. -[#19059](https://github.com/apache/druid/pull/19059) [#19205](https://github.com/apache/druid/pull/19205) +[#19059](https://github.com/apache/druid/pull/19059) [#19205](https://github.com/apache/druid/pull/19205) [#19016](https://github.com/apache/druid/pull/19016) ### Thrift input format @@ -176,6 +176,16 @@ consoleBrokerService: 'druid/BROKER_NAME' [#19069](https://github.com/apache/druid/pull/19069) +### Consul extension + +The contributor extension `druid-consul-extensions` lets Druid clusters use Consul for service discovery and +Coordinator/Overlord leader election instead of ZooKeeper. The extension supports ACLs, TLS/mTLS, and metrics. + +Before you switch to Consul, you need to set +`druid.serverview.type=http` and `druid.indexer.runner.type=httpRemote` cluster wide. + +[#18843](https://github.com/apache/druid/pull/18843) + ## Functional area and related changes This section contains detailed release notes separated by areas. @@ -184,6 +194,7 @@ This section contains detailed release notes separated by areas. #### Changed storage column displays +- Improved the compaction config view to - Renamed **Current size** to **Assigned size**. - Renamed **Max size** to **Effective size**. It now displays the smaller value between `max_size` and `storage_size`. The max size is still shown as a tooltip. - Changed usage calculation to use `effective_size` @@ -200,6 +211,7 @@ This section contains detailed release notes separated by areas. - Added support for Dart reports [#18897](https://github.com/apache/druid/pull/18897) - Changed what's considered an active worker: any nonzero rows, files, bytes, frames, or wall time is enough to consider a worker active [#19183](https://github.com/apache/druid/pull/19183) - Changed the **Cancel query** option to show only if a query is in an accepted or running state [#19182](https://github.com/apache/druid/pull/19182) +- Changed the ordering of the current Dart queries panel to show queries in the following order: RUNNING, ACCEPTED, and then COMPLETED. RUNNING and ACCEPTED queries are ordered by the most recent first (based on timestamp). COMPLETED queries are sorted by finish time [#19237](https://github.com/apache/druid/pull/19237) ### Ingestion @@ -210,6 +222,7 @@ This section contains detailed release notes separated by areas. - Added support for virtual storage fabric when performing SQL-based ingestion [#18873](https://github.com/apache/druid/pull/18873) - Added support for `StorageMonitor` so that MSQ task engine tasks always emit `taskId` and `groupId` [#19048](https://github.com/apache/druid/pull/19048) - Improved worker cancellation [#18931](https://github.com/apache/druid/pull/18931) +- Improved exception handling [#19234](https://github.com/apache/druid/pull/19234) #### Streaming ingestion @@ -228,6 +241,7 @@ Added `serverPriorityToReplicas` parameter to the streaming supervisor specs (Ka ##### Other streaming ingestion improvements - Improved cost-based autoscaler performance in high lag scenarios [#19045](https://github.com/apache/druid/pull/19045) +- Improved the performance of realtime task scheduling by ordering schedule requests by priority on the `TaskQueue` [#19203](https://github.com/apache/druid/pull/19203) #### Ingestion improvements @@ -322,6 +336,7 @@ The algorithm for cost-based autoscaling has been changed: #### Other cluster management improvements +- Added a `ReadOnly` authorizer that allows all READ operations but denies any other operation, such as WRITE [#19243](https://github.com/apache/druid/pull/19243) - Added `/status/ready` endpoint for service health so that external load balancers can handle a graceful shutdown better [#19148](https://github.com/apache/druid/pull/19148) - Added a configurable option to scale-down during task run time for cost-based autoscaler [#18958](https://github.com/apache/druid/pull/18958) - Added `storage_size` to `sys.servers` to facilitate retrieving disk cache size for Historicals when using the virtual storage fabric [#18979](https://github.com/apache/druid/pull/18979) @@ -344,10 +359,18 @@ The new configuration property `druid.msq.intermediate.storage.cleaner.durationT #### Other data management improvements +- Added the `druid.storage.transfer.asyncHttpClientType` config that specifies which async HTTP client to use for S3 transfers: `crt` for Amazon CRT or `netty` for Netty NIO [#19249](https://github.com/apache/druid/pull/19249) +- Added a mechanism to automatically clean up intermediary files on HDFS storage [#19187](https://github.com/apache/druid/pull/19187) - Changed the default value for `druid.indexing.formats.maxStringLength` to null from 0 [#19198](https://github.com/apache/druid/pull/19198) ### Metrics and monitoring +#### Version field + +All Druid metrics now include `version` and `buildRevision` fields to help you identify which version of Druid is emitting a metric. You can use this information to verify that all nodes in a cluster are running the intended revision during rolling deployments. + +[#19123](https://github.com/apache/druid/pull/19123) + #### Monitoring supervisor state Added a new `supervisor/count` metric when `SupervisorStatsMonitor` is enabled in `druid.monitoring.monitors`. The metric reports each supervisor’s state, such as `RUNNING` or `SUSPENDED`, for Prometheus, StatsD, and other metric systems. @@ -386,8 +409,8 @@ Added `segment/schemaCache/rowSignature/changed` and `segment/schemaCache/rowSig - Added `queries` and `totalQueries` counters, which reflect queries made to realtime servers to get realtime data - Added `tier/storage/capacity` metric for the Coordinator to emit that is guaranteed to reflect the total `StorageLocation` size configured across all Historicals in a tier [#18962](https://github.com/apache/druid/pull/18962) - Added new metrics for virtual storage fabric to the MSQ task engine `ChannelCounters`: `loadBytes`, `loadTime`, `loadWait`, and `loadFiles` [#18971](https://github.com/apache/druid/pull/18971) -- Added `storage/virtual/hit/bytes` metric to `StorageMonitor` -[#18895](https://github.com/apache/druid/pull/18895) +- Added `storage/virtual/hit/bytes`, `storage/virtual/hold/count` and `storage/virtual/hold/bytes` metric to `StorageMonitor` +[#18895](https://github.com/apache/druid/pull/18895) [#19217](https://github.com/apache/druid/pull/19217) - Added `supervisorId` dimension for streaming tasks to `TaskCountStatsMonitor` [#18920](https://github.com/apache/druid/pull/18920) - Changed `StorageMonitor` to always be on [#19048](https://github.com/apache/druid/pull/19048) - Improved the metrics for autoscalers, so that they all emit the same metrics: `supervisorId`, `dataSource`, and `stream` [#19097](https://github.com/apache/druid/pull/19097) @@ -402,6 +425,13 @@ The gRPC query extension now cancels in-flight queries when clients cancel or di #### Iceberg +##### GCS warehouse + +The Iceberg input source now supports GCS warehouses. To use this feature, you need to load the `druid-google-extensions` extension in addition to the Iceberg extension. +[#19137](https://github.com/apache/druid/pull/19137) + +##### Filters + You can now configure residual filters for non-partition columns when using the Iceberg input source. Set `residualFilterMode` in the Iceberg input source to one of the following: - `ignore`: (default) Ingest the residual rows with a warning log unless filtered by `transformSpec`. @@ -495,6 +525,15 @@ Segment locking and `NumberedOverwriteShardSpec` are deprecated and will be remo ### Incompatible changes +#### Removed `ParseSpec` and deprecated parsers + +The Parser for native batch tasks and streaming ingestion indexing services has been removed. Where possible, use the input format instead. Note that `JavascriptParseSpec` and `JSONLowercaseParseSpec` have no InputFormat equivalents. + +Druid supports custom text data formats and can use the Regex input format to parse them. However, be aware doing this to +parse data is less efficient than writing a native Java `InputFormat` extension, or using an external stream processor. We welcome contributions of new input formats. + +[#19239](https://github.com/apache/druid/pull/19239) + #### Removed `defaultProcessingRate` config This config allowed scaling actions to begin prior to the first metrics becoming available. @@ -509,6 +548,8 @@ Druid now defaults to v1 of the front-coded format instead of version 0 if enabl ### Developer notes +- Added `typecheck` to `npm run test-unit` to ensure TypeScript typechecking happens on calls to `test-unit` [#19251](https://github.com/apache/druid/pull/19251) +- Added a 14 day cooldown to dependabot updates to protect against not-yet-discovered regressions and security issues [#19241](https://github.com/apache/druid/pull/19241) - Added `AGENTS.md` [#19084](https://github.com/apache/druid/pull/19084) - Added a requirement to use [conventional commit syntax](https://www.conventionalcommits.org/en/v1.0.0/) [#19089](https://github.com/apache/druid/pull/19089) - Updated `checkstyle` from `3.0.0` to `3.6.0` [#19064](https://github.com/apache/druid/pull/19064) @@ -541,4 +582,26 @@ The following dependencies have been updated: - Added AWS SDK `2.40.0` [#18891](https://github.com/apache/druid/pull/18891) - `com.lmax.disruptor` from `3.3.6` to `3.4.4` [#19122](https://github.com/apache/druid/pull/19122) - `org.junit.junit-bom` from `5.13.3` to `5.14.3` [#19122](https://github.com/apache/druid/pull/19122) -- Several Kubernetes related dependencies have been updated. For more information, see [#19071](https://github.com/apache/druid/pull/19071) \ No newline at end of file +- Several Kubernetes related dependencies have been updated. For more information, see [#19071](https://github.com/apache/druid/pull/19071) +- `caffeine` from `2.8.0` to `2.9.3` [#19208](https://github.com/apache/druid/pull/19208) +- `commons-io` from `2.17.0` to `2.21.0` [#19208](https://github.com/apache/druid/pull/19208) +- `commons-collections4` from `4.2` to `4.5.0` [#19208](https://github.com/apache/druid/pull/19208) +- `commons-compress` from `1.27.0` to `1.28.0` [#19208](https://github.com/apache/druid/pull/19208) +- `zstd-jni` from `1.5.2-3` to `1.5.7-7` [#19208](https://github.com/apache/druid/pull/19208) +- `scala-library` from `2.12.7` to `2.13.16` [#19208](https://github.com/apache/druid/pull/19208) +- - `iceberg` from `1.7.2` to `1.10.0` [#19232](https://github.com/apache/druid/pull/19232) +- `parquet` from `1.15.2` to `1.16.0` [#19232](https://github.com/apache/druid/pull/19232) +- `avro` from `1.11.5` to `1.12.0` [#19232](https://github.com/apache/druid/pull/19232) +- - `jackson` from `2.19.2` to `2.20.2` [#19248](https://github.com/apache/druid/pull/19248) +- `netty4` from `4.2.6.Final` to `4.2.12.Final` [#19248](https://github.com/apache/druid/pull/19248) +- `errorprone` from `2.35.1` to `2.41.0` [#19248](https://github.com/apache/druid/pull/19248) +- `bcprov-jdk18on` / `bcpkix-jdk18on` from `1.81` to `1.82` [#19248](https://github.com/apache/druid/pull/19248) +- `RoaringBitmap` from `0.9.49` to `1.6.13` [#19248](https://github.com/apache/druid/pull/19248) +- `jedis` from `5.1.2` to `7.0.0` [#19248](https://github.com/apache/druid/pull/19248) +- `snakeyaml` from `2.4` to `2.5` [#19248](https://github.com/apache/druid/pull/19248) +- `aircompressor` from `0.21` to `2.0.2` [#19248](https://github.com/apache/druid/pull/19248) +- `reflections` from `0.9.12` to `0.10.2` [#19248](https://github.com/apache/druid/pull/19248) +- `httpclient5` from `5.5` to `5.5.1` [#19248](https://github.com/apache/druid/pull/19248) +- `jakarta.activation` from `1.2.2` to `2.0.1` [#19248](https://github.com/apache/druid/pull/19248) +- `netty-tcnative-boringssl-static` from `2.0.73.Final` to `2.0.75.Final` [#19248](https://github.com/apache/druid/pull/19248) +- `maven-compiler-plugin` from `3.11.0` to `3.14.1` [#19248](https://github.com/apache/druid/pull/19248) \ No newline at end of file diff --git a/docs/release-info/upgrade-notes.md b/docs/release-info/upgrade-notes.md index 371a14b7838d..dca396f60301 100644 --- a/docs/release-info/upgrade-notes.md +++ b/docs/release-info/upgrade-notes.md @@ -103,6 +103,15 @@ Segment locking and `NumberedOverwriteShardSpec` are deprecated and will be remo ### Incompatible changes +#### Removed `ParseSpec` and deprecated parsers + +The Parser for native batch tasks and streaming ingestion indexing services has been removed. Where possible, use the input format instead. Note that `JavascriptParseSpec` and `JSONLowercaseParseSpec` have no InputFormat equivalents. + +Druid supports custom text data formats and can use the Regex input format to parse them. However, be aware doing this to +parse data is less efficient than writing a native Java `InputFormat` extension, or using an external stream processor. We welcome contributions of new input formats. + +[#19239](https://github.com/apache/druid/pull/19239) + #### Removed `defaultProcessingRate` config This config allowed scaling actions to begin prior to the first metrics becoming available. From 3378d7801cc25ecec96cd86f6293435f3bb2799b Mon Sep 17 00:00:00 2001 From: brian Date: Fri, 10 Apr 2026 10:29:26 -0700 Subject: [PATCH 04/21] fix spelling --- docs/release-info/release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release-info/release-notes.md b/docs/release-info/release-notes.md index c53b00de1f70..ce55feaa2daa 100644 --- a/docs/release-info/release-notes.md +++ b/docs/release-info/release-notes.md @@ -549,7 +549,7 @@ Druid now defaults to v1 of the front-coded format instead of version 0 if enabl ### Developer notes - Added `typecheck` to `npm run test-unit` to ensure TypeScript typechecking happens on calls to `test-unit` [#19251](https://github.com/apache/druid/pull/19251) -- Added a 14 day cooldown to dependabot updates to protect against not-yet-discovered regressions and security issues [#19241](https://github.com/apache/druid/pull/19241) +- Added a 14 day cooldown to `dependabot` updates to protect against not-yet-discovered regressions and security issues [#19241](https://github.com/apache/druid/pull/19241) - Added `AGENTS.md` [#19084](https://github.com/apache/druid/pull/19084) - Added a requirement to use [conventional commit syntax](https://www.conventionalcommits.org/en/v1.0.0/) [#19089](https://github.com/apache/druid/pull/19089) - Updated `checkstyle` from `3.0.0` to `3.6.0` [#19064](https://github.com/apache/druid/pull/19064) From 87ff8d5037b4119a9f618360feb61f94fbf89de7 Mon Sep 17 00:00:00 2001 From: 317brian <53799971+317brian@users.noreply.github.com> Date: Mon, 13 Apr 2026 15:16:03 -0700 Subject: [PATCH 05/21] add suggested revisions Co-authored-by: Charles Smith --- docs/release-info/release-notes.md | 36 +++++++++++++++--------------- 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/docs/release-info/release-notes.md b/docs/release-info/release-notes.md index ce55feaa2daa..2960af2b64a9 100644 --- a/docs/release-info/release-notes.md +++ b/docs/release-info/release-notes.md @@ -67,7 +67,7 @@ Use one of Druid's other supported ingestion methods, such as SQL-based ingestio ### Query blocklist -You can now create a query blocklist for dynamically blocking queries without restarting your deployment. You can block queries by datasource, query type, or query context using the new `/druid/coordinator/v1/config/broker` API. Block rules use `AND` logic, which means all criteria must match. Druid stores the rules in the metadata database. +You can now use the using the `/druid/coordinator/v1/config/broker` API tp create a query blocklist to dynamically block queries by datasource, query type, or query context. The blocklist takes effect without a restarting Druid. Block rules use `AND` logic, which means all criteria must match. The following example blocks all groupBy queries on the `wikipedia` datasource with a query context parameter of `priority` equal to `0`: @@ -89,7 +89,7 @@ POST /druid/coordinator/v1/config/broker ### Minor compaction for Overlord-based compaction (experimental) -You can now configure minor compaction. With minor compaction, Druid only compacts newly ingested segments while upgrading existing compacted segments. When Druid upgrades segments, it updates the metadata instead of using resources to compact it again. You can use the native compaction engine or the MSQ task engine. +You can now configure minor compaction to compact only newly ingested segments while upgrading existing compacted segments. When Druid upgrades segments, it updates the metadata instead of using resources to compact it again. You can use the native compaction engine or the MSQ task engine. Use the `mostFragmentedFirst` compaction policy and set either a percentage of rows-based or byte-based threshold for minor compaction. @@ -117,7 +117,7 @@ Tombstones for JSON-based native batch ingestion (the `dropExisting` flag for `i ### Dynamic default query context -You can now add default query context parameters as a dynamic configuration to the Broker. This allows you to override static defaults set in your runtime properties without restarting your deployment or having to update multiple queries individually. Druid resolves which query context parameters to use based on the following priority: +You can now add default query context parameters as a dynamic configuration to the Broker. This allows you to override static defaults set in your runtime properties without restarting your deployment or having to update multiple queries individually. Druid applies query context parameters based on the following priority: 1. The query context included with the query 1. The query context set as a dynamic configuration on the Broker @@ -125,14 +125,14 @@ You can now add default query context parameters as a dynamic configuration to t 1. The defaults that ship with Druid Note that like other Broker dynamic configuration, this is best-effort. Settings may not be applied in certain -cases, such as when a Broker has recently started and hasn't received the config yet, or if the +cases, such as when a Broker has recently started and hasn't received the configuration yet, or if the Broker can't contact the Coordinator. If a query context parameter is critical for all your queries, set it in the runtime properties. [#19146](https://github.com/apache/druid/pull/19146) ### `sys.queries` table (experimental) -The new system queries table provides information about currently running and recently completed queries that use the Dart engine. This table is off by default. To enable the table, set the following config: +The new system queries table provides information about currently running and recently completed queries that use the Dart engine. This table is off by default. To enable the table, set the following: ``` druid.sql.planner.enableSysQueriesTable = true @@ -146,11 +146,11 @@ As part of this change, the `/druid/v2/sql/queries` API now supports an `include Auto-compaction using compaction supervisors has been improved and is now generally available. -As part of the improvement, instead of individual compacted segments having to store their full compaction state in the metadata store, states are now stored in a central location, a new `indexingStates` table, in the metadata store. Individual segments only need to store a unique reference (`indexing_state_fingerprint`) to their full compaction state. +As part of the improvement compaction states are now stored in a central location, a new `indexingStates` table. Individual segments only need to store a unique reference (`indexing_state_fingerprint`) to their full compaction state. Since many segments in a single datasource share the same underlying compaction state, this greatly reduces metadata storage requirements for automatic compaction. -For backwards compatibility, detailed compaction state will continue to be persisted in each segment for now. This will be removed in a future release. +For backwards compatibility, Druid continues to persist the detailed compaction state in each segment. This functionality will be removed in a future release. You can stop storing detailed compaction state by setting `storeCompactionStatePerSegment` to `false` in the cluster compaction config. If you turn it off and need to downgrade, Druid needs to re-compact any segments that have been compacted since you changed the config. @@ -209,7 +209,7 @@ This section contains detailed release notes separated by areas. - Added a detail dialog to the **Services** page [#18960](https://github.com/apache/druid/pull/18960) - Added icons to indicate when data is loaded into virtual storage, including a tooltip that shows all the counters for the data [#19010](https://github.com/apache/druid/pull/19010) - Added support for Dart reports [#18897](https://github.com/apache/druid/pull/18897) -- Changed what's considered an active worker: any nonzero rows, files, bytes, frames, or wall time is enough to consider a worker active [#19183](https://github.com/apache/druid/pull/19183) +- Changed the criteria for active workers: any nonzero rows, files, bytes, frames, or wall time is enough to consider a worker active [#19183](https://github.com/apache/druid/pull/19183) - Changed the **Cancel query** option to show only if a query is in an accepted or running state [#19182](https://github.com/apache/druid/pull/19182) - Changed the ordering of the current Dart queries panel to show queries in the following order: RUNNING, ACCEPTED, and then COMPLETED. RUNNING and ACCEPTED queries are ordered by the most recent first (based on timestamp). COMPLETED queries are sorted by finish time [#19237](https://github.com/apache/druid/pull/19237) @@ -312,9 +312,9 @@ GROUP BY 1, 2 Operators can now configure two new Broker `TierSelectorStrategy` implementations: - `strict` - Only selects servers whose priorities match the configured list. Example configuration: `druid.broker.select.tier=strict` and `druid.broker.select.tier.strict.priorities=[1]`. -- `pooled` - Pools servers across the configured priorities and selects among them, allowing queries to utilize multiple priority tiers for improved availability. Example configuration: `druid.broker.select.tier=pooled` and `druid.broker.select.tier.pooled.priorities=[2,1]`. +- `pooled` - Pools servers across the configured priorities and selects among them, allowing queries to use multiple priority tiers for improved availability. Example configuration: `druid.broker.select.tier=pooled` and `druid.broker.select.tier.pooled.priorities=[2,1]`. -These strategies can also be configured for realtime servers using `druid.broker.realtime.select.tier`. +You can also use `druid.broker.realtime.select.tier` to configure these strategies for realtime servers. [#19094](https://github.com/apache/druid/pull/19094) @@ -390,24 +390,24 @@ Added a new `supervisor/count` metric when `SupervisorStatsMonitor` is enabled i #### Filtering metrics -Operators can limit which metrics the logging emitter writes by setting `druid.emitter.logging.shouldFilterMetrics=true` and, if desired, `druid.emitter.logging.allowedMetricsPath` to a JSON object file where the keys are metric names. A missing custom file results in a warning and use of the bundled `defaultMetrics.json`. Alerts and other non-metric events are always logged. +Operators can set `druid.emitter.logging.shouldFilterMetrics=true` to limit which metrics the logging emitter writes. Optionally, they can set `druid.emitter.logging.allowedMetricsPath` to a JSON object file where the keys are metric names. A missing custom file results in a warning and use of the bundled `defaultMetrics.json`. Alerts and other non-metric events are always logged. [#19030](https://github.com/apache/druid/pull/19030) #### New Broker metrics -Added `segment/schemaCache/rowSignature/changed` and `segment/schemaCache/rowSignature/column/count` metrics to get visibility into when the Broker's segment metadata cache's row signature for each datasource is initialized and updated. +Added `segment/schemaCache/rowSignature/changed` and `segment/schemaCache/rowSignature/column/count` metrics to expose events when the Broker initializes and updates the row signature in the the segment metadata cache for each datasource. [#18966](https://github.com/apache/druid/pull/18966) #### Other metrics and monitoring improvements - Added the following metrics to the default for Prometheus: `mergeBuffer/bytesUsed` and `mergeBuffer/maxBytesUsed` [#19110](https://github.com/apache/druid/pull/19110) -- Added compaction mode to what gets emitted for the `compact/task/count` metric [#19151](https://github.com/apache/druid/pull/19151) +- Added compaction mode to the `compact/task/count` metric [#19151](https://github.com/apache/druid/pull/19151) - Added support for logging and emitting SQL dynamic parameter values [#19067](https://github.com/apache/druid/pull/19067) -- Added `ingest/rows/published`, which gets emitted from all task types to denote the total row count of successfully published segments [#19177](https://github.com/apache/druid/pull/19177) -- Added `queries` and `totalQueries` counters, which reflect queries made to realtime servers to get realtime data -- Added `tier/storage/capacity` metric for the Coordinator to emit that is guaranteed to reflect the total `StorageLocation` size configured across all Historicals in a tier [#18962](https://github.com/apache/druid/pull/18962) +- Added `ingest/rows/published`, which all task types emit to denote the total row count of successfully published segments [#19177](https://github.com/apache/druid/pull/19177) +- Added `queries` and `totalQueries` counters, which reflect queries made to realtime servers to retrieve realtime data +- Added `tier/storage/capacity` metric for the Coordinator. This metric is guaranteed to reflect the total `StorageLocation` size configured across all Historicals in a tier [#18962](https://github.com/apache/druid/pull/18962) - Added new metrics for virtual storage fabric to the MSQ task engine `ChannelCounters`: `loadBytes`, `loadTime`, `loadWait`, and `loadFiles` [#18971](https://github.com/apache/druid/pull/18971) - Added `storage/virtual/hit/bytes`, `storage/virtual/hold/count` and `storage/virtual/hold/bytes` metric to `StorageMonitor` [#18895](https://github.com/apache/druid/pull/18895) [#19217](https://github.com/apache/druid/pull/19217) @@ -427,7 +427,7 @@ The gRPC query extension now cancels in-flight queries when clients cancel or di ##### GCS warehouse -The Iceberg input source now supports GCS warehouses. To use this feature, you need to load the `druid-google-extensions` extension in addition to the Iceberg extension. +The Iceberg input source now supports GCS warehouses. To use this feature, you must load the `druid-google-extensions` extension in addition to the Iceberg extension. [#19137](https://github.com/apache/druid/pull/19137) ##### Filters @@ -484,7 +484,7 @@ You can't perform a rolling upgrade from versions earlier than Druid 0.23. #### Metadata storage for auto-compaction with compaction supervisors -Automatic compaction using compaction supervisors now requires incremental segment metadata caching to be enabled on the Overlord and Coordinator via runtime properties. Specifically, the `druid.manager.segments.useIncrementalCache` config must be set to `always` or `ifSynced`. For more information about the config, see [Segment metadata cache](https://druid.apache.org/docs/latest/configuration/#segment-metadata-cache-experimental). +Automatic compaction using compaction supervisors now requires incremental segment metadata caching to be enabled on the Overlord and Coordinator in the runtime properties. Specifically, the `druid.manager.segments.useIncrementalCache` config must be set to `always` or `ifSynced`. For more information about the config, see [Segment metadata cache](https://druid.apache.org/docs/latest/configuration/#segment-metadata-cache-experimental). Additionally, metadata store changes are required for this upgrade. From dd02c603aff5afddb06619e5288214520281fa3f Mon Sep 17 00:00:00 2001 From: brian Date: Mon, 13 Apr 2026 16:29:13 -0700 Subject: [PATCH 06/21] update 19071 --- docs/release-info/release-notes.md | 15 ++++++++++++++- 1 file changed, 14 insertions(+), 1 deletion(-) diff --git a/docs/release-info/release-notes.md b/docs/release-info/release-notes.md index 2960af2b64a9..b95aa8332fdd 100644 --- a/docs/release-info/release-notes.md +++ b/docs/release-info/release-notes.md @@ -417,6 +417,12 @@ Added `segment/schemaCache/rowSignature/changed` and `segment/schemaCache/rowSig ### Extensions +#### Kubernetes + +Added a new `WebClientOptions` pass-through for the Vert.x HTTP client in the `kubernetes-overlord-extensions`. Operators can now configure any property on the underlying Vert.x `WebClientOptions` object by using Druid runtime properties. Some of the options you can configure includeconnection pool size, keep-alive timeouts, and idle timeouts. This is particularly useful for environments with intermediate load balancers that close idle connections. Most Druid deployments will not need this configuration. + +[#19071](https://github.com/apache/druid/pull/19071) + #### gRPC The gRPC query extension now cancels in-flight queries when clients cancel or disconnect. @@ -582,7 +588,14 @@ The following dependencies have been updated: - Added AWS SDK `2.40.0` [#18891](https://github.com/apache/druid/pull/18891) - `com.lmax.disruptor` from `3.3.6` to `3.4.4` [#19122](https://github.com/apache/druid/pull/19122) - `org.junit.junit-bom` from `5.13.3` to `5.14.3` [#19122](https://github.com/apache/druid/pull/19122) -- Several Kubernetes related dependencies have been updated. For more information, see [#19071](https://github.com/apache/druid/pull/19071) +- - `io.fabric8:kubernetes-client` 7.5.2 → 7.6.0 [#19071](https://github.com/apache/druid/pull/19071) +- `io.kubernetes:client-java` 19.0.0 → 25.0.0-legacy [#19071](https://github.com/apache/druid/pull/19071) +- `com.squareup.okhttp3:okhttp` 4.12.0 → 5.3.2 [#19071](https://github.com/apache/druid/pull/19071) +- `org.jetbrains.kotlin:kotlin-stdlib` 1.9.25 → 2.2.21 [#19071](https://github.com/apache/druid/pull/19071) +- `commons-codec:commons-codec` 1.17.1 → 1.20.0 [#19071](https://github.com/apache/druid/pull/19071) +- `org.apache.commons:commons-lang3` 3.19.0 → 3.20.0 [#19071](https://github.com/apache/druid/pull/19071) +- `com.google.code.gson:gson` 2.12.0 → 2.13.2 [#19071](https://github.com/apache/druid/pull/19071) +- `com.amazonaws:aws-java-sdk` 1.12.784 → 1.12.793 [#19071](https://github.com/apache/druid/pull/19071) - `caffeine` from `2.8.0` to `2.9.3` [#19208](https://github.com/apache/druid/pull/19208) - `commons-io` from `2.17.0` to `2.21.0` [#19208](https://github.com/apache/druid/pull/19208) - `commons-collections4` from `4.2` to `4.5.0` [#19208](https://github.com/apache/druid/pull/19208) From 4cffd6117baab1c2fe6dabbf30f64a125e99f076 Mon Sep 17 00:00:00 2001 From: brian Date: Thu, 16 Apr 2026 10:46:15 -0700 Subject: [PATCH 07/21] add cascading reindexing --- docs/release-info/release-notes.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/release-info/release-notes.md b/docs/release-info/release-notes.md index b95aa8332fdd..25d222f1cd1c 100644 --- a/docs/release-info/release-notes.md +++ b/docs/release-info/release-notes.md @@ -95,6 +95,16 @@ Use the `mostFragmentedFirst` compaction policy and set either a percentage of r [#19059](https://github.com/apache/druid/pull/19059) [#19205](https://github.com/apache/druid/pull/19205) [#19016](https://github.com/apache/druid/pull/19016) +### Cascading reindexing (experimental) + +Using cascading reindexing, you can now define age-based rules to automatically apply different compaction configurations based on the age of your data. While standard auto-compaction applies a single flat configuration across an entire datasource, cascading reindexing lets you tailor your compaction settings to the characteristics of your data. + +For example, you can keep recent data in hourly segments while automatically rolling up to daily segments after 90 days to reduce segment count. You can also layer on age-based row deletion (such as dropping bot traffic from older data), change compression settings, or shift to rollup with coarser query granularity as data ages. Rules are defined inline in the supervisor spec. + +You must use compaction supervisors with the MSQ task engine to use cascading reindexing. + +[#18939](https://github.com/apache/druid/pull/18939) [#19213](https://github.com/apache/druid/pull/19213) [#19106](https://github.com/apache/druid/pull/19106) [#19078](https://github.com/apache/druid/pull/19078) + ### Thrift input format As part of the Thrift contributor extension, Druid now supports Thrift-encoded data for Kafka and Kinesis streaming ingestion. From 722d4d2ebc5909cd81f734105d0760d49c70246f Mon Sep 17 00:00:00 2001 From: brian Date: Thu, 16 Apr 2026 11:34:13 -0700 Subject: [PATCH 08/21] fix typos ; --- docs/release-info/release-notes.md | 8 ++++---- docs/release-info/upgrade-notes.md | 2 +- website/.spelling | 1 + 3 files changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/release-info/release-notes.md b/docs/release-info/release-notes.md index 25d222f1cd1c..5f72bf3decfc 100644 --- a/docs/release-info/release-notes.md +++ b/docs/release-info/release-notes.md @@ -67,7 +67,7 @@ Use one of Druid's other supported ingestion methods, such as SQL-based ingestio ### Query blocklist -You can now use the using the `/druid/coordinator/v1/config/broker` API tp create a query blocklist to dynamically block queries by datasource, query type, or query context. The blocklist takes effect without a restarting Druid. Block rules use `AND` logic, which means all criteria must match. +You can now use the using the `/druid/coordinator/v1/config/broker` API to create a query blocklist to dynamically block queries by datasource, query type, or query context. The blocklist takes effect without a restarting Druid. Block rules use `AND` logic, which means all criteria must match. The following example blocks all groupBy queries on the `wikipedia` datasource with a query context parameter of `priority` equal to `0`: @@ -268,7 +268,7 @@ Added a new groupBy query configuration property `druid.query.groupBy.maxSpillFi #### Improved handling of nested aggregates -Druid can now merge two Aggregates with a Projection between them. For example, the following query: +Druid can now merge two aggregates with a projection between them. For example, the following query: ```sql SELECT @@ -429,7 +429,7 @@ Added `segment/schemaCache/rowSignature/changed` and `segment/schemaCache/rowSig #### Kubernetes -Added a new `WebClientOptions` pass-through for the Vert.x HTTP client in the `kubernetes-overlord-extensions`. Operators can now configure any property on the underlying Vert.x `WebClientOptions` object by using Druid runtime properties. Some of the options you can configure includeconnection pool size, keep-alive timeouts, and idle timeouts. This is particularly useful for environments with intermediate load balancers that close idle connections. Most Druid deployments will not need this configuration. +Added a new `WebClientOptions` pass-through for the Vert.x HTTP client in the `kubernetes-overlord-extensions`. Operators can now configure any property on the underlying Vert.x `WebClientOptions` object by using Druid runtime properties. Some of the options you can configure include connection pool size, keep-alive timeouts, and idle timeouts. This is particularly useful for environments with intermediate load balancers that close idle connections. Most Druid deployments will not need this configuration. [#19071](https://github.com/apache/druid/pull/19071) @@ -564,7 +564,7 @@ Druid now defaults to v1 of the front-coded format instead of version 0 if enabl ### Developer notes -- Added `typecheck` to `npm run test-unit` to ensure TypeScript typechecking happens on calls to `test-unit` [#19251](https://github.com/apache/druid/pull/19251) +- Added `typecheck` to `npm run test-unit` to ensure TypeScript type checking happens on calls to `test-unit` [#19251](https://github.com/apache/druid/pull/19251) - Added a 14 day cooldown to `dependabot` updates to protect against not-yet-discovered regressions and security issues [#19241](https://github.com/apache/druid/pull/19241) - Added `AGENTS.md` [#19084](https://github.com/apache/druid/pull/19084) - Added a requirement to use [conventional commit syntax](https://www.conventionalcommits.org/en/v1.0.0/) [#19089](https://github.com/apache/druid/pull/19089) diff --git a/docs/release-info/upgrade-notes.md b/docs/release-info/upgrade-notes.md index dca396f60301..8e4e32acc8ce 100644 --- a/docs/release-info/upgrade-notes.md +++ b/docs/release-info/upgrade-notes.md @@ -62,7 +62,7 @@ You can't perform a rolling upgrade from versions earlier than Druid 0.23. #### Metadata storage for auto-compaction with compaction supervisors -Automatic compaction using compaction supervisors now requires incremental segment metadata caching to be enabled on the Overlord and Coordinator via runtime properties. Specifically, the `druid.manager.segments.useIncrementalCache` config must be set to `always` or `ifSynced`. For more information about the config, see [Segment metadata cache](https://druid.apache.org/docs/latest/configuration/#segment-metadata-cache-experimental). +Automatic compaction using compaction supervisors now requires incremental segment metadata caching to be enabled on the Overlord and Coordinator in the runtime properties. Specifically, the `druid.manager.segments.useIncrementalCache` config must be set to `always` or `ifSynced`. For more information about the config, see [Segment metadata cache](https://druid.apache.org/docs/latest/configuration/#segment-metadata-cache-experimental). Additionally, metadata store changes are required for this upgrade. diff --git a/website/.spelling b/website/.spelling index d4569666af32..4efcb676e645 100644 --- a/website/.spelling +++ b/website/.spelling @@ -1317,6 +1317,7 @@ pull-deps sqlserver-metadata-storage statsd-emitter coords +cooldown dimName maxCoords Mb From 78ef933b2fd5b886e11f12bea6c8098d8dbf9be9 Mon Sep 17 00:00:00 2001 From: brian Date: Thu, 16 Apr 2026 15:33:38 -0700 Subject: [PATCH 09/21] add 19258 --- docs/release-info/release-notes.md | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/docs/release-info/release-notes.md b/docs/release-info/release-notes.md index 5f72bf3decfc..c7be4496ec57 100644 --- a/docs/release-info/release-notes.md +++ b/docs/release-info/release-notes.md @@ -113,6 +113,12 @@ As part of the Thrift contributor extension, Druid now supports Thrift-encoded d To use this feature, you must add `druid-thrift-extensions` to your extension load list. +### Incremental cache + +Incremental segment metadata cache (`useIncrementalCache`) is now generally available and defaults to `ifSynced`. Druid blocks reads from the cache until it has synced with the metadata store at least once after becoming leader. + +[#19252](https://github.com/apache/druid/pull/19252) + ### Kubernetes-based task management This extension is now generally available. @@ -256,6 +262,7 @@ Added `serverPriorityToReplicas` parameter to the streaming supervisor specs (Ka #### Ingestion improvements +- Added `StringColumnFormatSpec` for string dimension configs [#19258](https://github.com/apache/druid/pull/19258) - Sped up task scheduling on the Overlord [#19199](https://github.com/apache/druid/pull/19199) ### Querying @@ -359,6 +366,24 @@ The algorithm for cost-based autoscaling has been changed: ### Data management +#### Per-segment timeout configuration + +You can now set a timeout for the segments in a specific datasource using a dynamic configuration: + +``` +POST /druid/coordinator/v1/config/broker + { + "perSegmentTimeoutConfig": { + "my_large_datasource": { "perSegmentTimeoutMs": 5000, "monitorOnly": false }, + "my_new_datasource": { "perSegmentTimeoutMs": 3000, "monitorOnly": true } + } + } +``` + +This is useful when different datasources have different performance characteristics — for example, allowing longer timeouts for larger datasets. + +[#19221](https://github.com/apache/druid/pull/19221) + #### Durable storage cleaner The durable storage cleaner now supports configurable time-based retention for MSQ query results. Previously, query results were retained for all known tasks, which was unreliable for completed tasks. With this change, query results are retained for a configurable time period based on the task creation time. From ad3f309d90fab6e30648b81538639fb0a756094e Mon Sep 17 00:00:00 2001 From: 317brian <53799971+317brian@users.noreply.github.com> Date: Mon, 20 Apr 2026 15:19:31 -0700 Subject: [PATCH 10/21] Apply suggestion from @317brian --- docs/release-info/release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release-info/release-notes.md b/docs/release-info/release-notes.md index c7be4496ec57..0bb1e8614472 100644 --- a/docs/release-info/release-notes.md +++ b/docs/release-info/release-notes.md @@ -144,7 +144,7 @@ Note that like other Broker dynamic configuration, this is best-effort. Settings cases, such as when a Broker has recently started and hasn't received the configuration yet, or if the Broker can't contact the Coordinator. If a query context parameter is critical for all your queries, set it in the runtime properties. -[#19146](https://github.com/apache/druid/pull/19146) +[#19144](https://github.com/apache/druid/pull/19144) ### `sys.queries` table (experimental) From 239fcf1bf9c28597b56c7337fbca5ace2b530dd2 Mon Sep 17 00:00:00 2001 From: brian Date: Mon, 20 Apr 2026 15:23:59 -0700 Subject: [PATCH 11/21] fix 19146 --- docs/release-info/release-notes.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/docs/release-info/release-notes.md b/docs/release-info/release-notes.md index 0bb1e8614472..366c378b2464 100644 --- a/docs/release-info/release-notes.md +++ b/docs/release-info/release-notes.md @@ -231,6 +231,10 @@ This section contains detailed release notes separated by areas. ### Ingestion +- Added the `maxStringLength` configuration for string dimensions that truncates values exceeding the specified length during ingestion. You can set the length globally using `druid.indexing.formats.maxStringLength` or per-dimension in the ingestion spec [#19146](https://github.com/apache/druid/pull/19146) +- Added `StringColumnFormatSpec` for string dimension configs [#19258](https://github.com/apache/druid/pull/19258) +- Sped up task scheduling on the Overlord [#19199](https://github.com/apache/druid/pull/19199) + #### SQL-based ingestion ##### Other SQL-based ingestion improvements @@ -259,12 +263,6 @@ Added `serverPriorityToReplicas` parameter to the streaming supervisor specs (Ka - Improved cost-based autoscaler performance in high lag scenarios [#19045](https://github.com/apache/druid/pull/19045) - Improved the performance of realtime task scheduling by ordering schedule requests by priority on the `TaskQueue` [#19203](https://github.com/apache/druid/pull/19203) - -#### Ingestion improvements - -- Added `StringColumnFormatSpec` for string dimension configs [#19258](https://github.com/apache/druid/pull/19258) -- Sped up task scheduling on the Overlord [#19199](https://github.com/apache/druid/pull/19199) - ### Querying #### groupBy query configuration From c691d1bb3026e879f6b1e5a1691254fc331c7fba Mon Sep 17 00:00:00 2001 From: brian Date: Mon, 20 Apr 2026 15:26:54 -0700 Subject: [PATCH 12/21] update 19050 --- docs/release-info/release-notes.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/release-info/release-notes.md b/docs/release-info/release-notes.md index 366c378b2464..93f46062edef 100644 --- a/docs/release-info/release-notes.md +++ b/docs/release-info/release-notes.md @@ -172,7 +172,7 @@ You can stop storing detailed compaction state by setting `storeCompactionStateP This change has upgrade impacts for metadata storage and metadata caching. For more information, see the [Metadata storage for auto-compaction with compaction supervisors](#metadata-storage-for-auto-compaction-with-compaction-supervisors) upgrade note. -[#19113](https://github.com/apache/druid/pull/19113) [#18844](https://github.com/apache/druid/pull/18844) +[#19113](https://github.com/apache/druid/pull/19113) [#18844](https://github.com/apache/druid/pull/18844) [#19252](https://github.com/apache/druid/pull/19252) ### Broker tier selection for realtime servers @@ -558,7 +558,7 @@ You may have to adapt the syntax to fit your table naming prefix and metadata st #### Segment locking -Segment locking and `NumberedOverwriteShardSpec` are deprecated and will be removed in a future release. Use time chunk locking instead. You can make sure only time chunk locking is used by setting `druid.indexer.tasklock.forceTimeChunkLock` to `true`. +Segment locking and `NumberedOverwriteShardSpec` are deprecated and will be removed in a future release. Use time chunk locking instead. You can make sure only time chunk locking is used by setting `druid.indexer.tasklock.forceTimeChunkLock` to `true`, which is the default. [#19050](https://github.com/apache/druid/pull/19050) From 69555c5d4a87321325ef092847a904c0297e6455 Mon Sep 17 00:00:00 2001 From: brian Date: Mon, 20 Apr 2026 15:51:44 -0700 Subject: [PATCH 13/21] update metadata storage upgrade note --- docs/release-info/release-notes.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/docs/release-info/release-notes.md b/docs/release-info/release-notes.md index 93f46062edef..f7eb6ea112da 100644 --- a/docs/release-info/release-notes.md +++ b/docs/release-info/release-notes.md @@ -523,13 +523,16 @@ You can't perform a rolling upgrade from versions earlier than Druid 0.23. #### Metadata storage for auto-compaction with compaction supervisors -Automatic compaction using compaction supervisors now requires incremental segment metadata caching to be enabled on the Overlord and Coordinator in the runtime properties. Specifically, the `druid.manager.segments.useIncrementalCache` config must be set to `always` or `ifSynced`. For more information about the config, see [Segment metadata cache](https://druid.apache.org/docs/latest/configuration/#segment-metadata-cache-experimental). +Automatic compaction using compaction supervisors requires incremental segment metadata caching to be enabled on the Overlord and Coordinator in the runtime properties. -Additionally, metadata store changes are required for this upgrade. +As part of this update, Druid requires the incremental cache to be enabled and a new table in the metadata store. No action is required if you are using the default settings for the following configs: -If you already have `druid.metadata.storage.connector.createTables` set to `true`, no action is needed. +- `druid.manager.segments.useIncrementalCache` +- `druid.metadata.storage.connector.createTables` -If you have this feature turned off, you will need to alter the segments table and create the `compactionStates` table. The Postgres DDL is provided below as a guide: +If `druid.manager.segments.useIncrementalCache` is set to `never`, update it to `ifSynced` or `always`. For more information about the config, see [Segment metadata cache](https://druid.apache.org/docs/latest/configuration/#segment-metadata-cache-experimental). + +If you set the `druid.metadata.storage.connector.createTables` config to `false`, you need to manually alter the segments table and create the `compactionStates` table. The Postgres DDL is provided below as a guide: ``` -- create the indexing states lookup table and associated indices From 22f07ec02164402031992dbe90debec37221c5dd Mon Sep 17 00:00:00 2001 From: brian Date: Tue, 21 Apr 2026 13:16:19 -0700 Subject: [PATCH 14/21] address comments --- docs/release-info/release-notes.md | 68 +++++++++++++++++++----------- 1 file changed, 43 insertions(+), 25 deletions(-) diff --git a/docs/release-info/release-notes.md b/docs/release-info/release-notes.md index f7eb6ea112da..3347f26b3899 100644 --- a/docs/release-info/release-notes.md +++ b/docs/release-info/release-notes.md @@ -67,7 +67,7 @@ Use one of Druid's other supported ingestion methods, such as SQL-based ingestio ### Query blocklist -You can now use the using the `/druid/coordinator/v1/config/broker` API to create a query blocklist to dynamically block queries by datasource, query type, or query context. The blocklist takes effect without a restarting Druid. Block rules use `AND` logic, which means all criteria must match. +You can now use the Broker API (`/druid/coordinator/v1/config/broker`) to create a query blocklist to dynamically block queries by datasource, query type, or query context. The blocklist takes effect without a restarting Druid. Block rules use `AND` logic, which means all criteria must match. The following example blocks all groupBy queries on the `wikipedia` datasource with a query context parameter of `priority` equal to `0`: @@ -107,7 +107,7 @@ You must use compaction supervisors with the MSQ task engine to use cascading re ### Thrift input format -As part of the Thrift contributor extension, Druid now supports Thrift-encoded data for Kafka and Kinesis streaming ingestion. +As part of the Thrift contributor extension, Druid now supports Thrift-encoded data for Kafka and Kinesis streaming ingestion using `InputFormat`. Previously, Druid supported this through parsers, which have been removed in Druid 37. [#19111](https://github.com/apache/druid/pull/19111) @@ -125,12 +125,6 @@ This extension is now generally available. [#19128](https://github.com/apache/druid/pull/19128) -### Tombstones - -Tombstones for JSON-based native batch ingestion (the `dropExisting` flag for `ioConfig`) are now generally available. - -[#19128](https://github.com/apache/druid/pull/19128) - ### Dynamic default query context You can now add default query context parameters as a dynamic configuration to the Broker. This allows you to override static defaults set in your runtime properties without restarting your deployment or having to update multiple queries individually. Druid applies query context parameters based on the following priority: @@ -210,6 +204,8 @@ This section contains detailed release notes separated by areas. #### Changed storage column displays +The following improvements have been made to how storage columns are displayed in the web console: + - Improved the compaction config view to - Renamed **Current size** to **Assigned size**. - Renamed **Max size** to **Effective size**. It now displays the smaller value between `max_size` and `storage_size`. The max size is still shown as a tooltip. @@ -231,8 +227,25 @@ This section contains detailed release notes separated by areas. ### Ingestion -- Added the `maxStringLength` configuration for string dimensions that truncates values exceeding the specified length during ingestion. You can set the length globally using `druid.indexing.formats.maxStringLength` or per-dimension in the ingestion spec [#19146](https://github.com/apache/druid/pull/19146) -- Added `StringColumnFormatSpec` for string dimension configs [#19258](https://github.com/apache/druid/pull/19258) +#### Truncate string columns + +Use the `StringColumnFormatSpec` config to set the maximum length for string dimension columns you ingest: + +- For a specific dimension: `dimensionSchema.columnFormatSpec.maxStringLength` +- For a specific job: `indexSpec.columnFormatSpec.maxStringLength` +- Cluster-wide: `druid.indexing.formats.maxStringLength` + +Druid truncates any string longer than the specified length. The default is to not truncate string values. + +[#19146](https://github.com/apache/druid/pull/19146) [#19258](https://github.com/apache/druid/pull/19258) (https://github.com/apache/druid/pull/19198) + + +- Added the `maxStringLength` configuration for string dimensions that truncates values exceeding the specified length during ingestion. You can set the length globally using `druid.indexing.formats.maxStringLength` or per-dimension in the ingestion spec +- Added `StringColumnFormatSpec` for string dimension configs + + +#### Other ingestion improvements + - Sped up task scheduling on the Overlord [#19199](https://github.com/apache/druid/pull/19199) #### SQL-based ingestion @@ -394,7 +407,6 @@ The new configuration property `druid.msq.intermediate.storage.cleaner.durationT - Added the `druid.storage.transfer.asyncHttpClientType` config that specifies which async HTTP client to use for S3 transfers: `crt` for Amazon CRT or `netty` for Netty NIO [#19249](https://github.com/apache/druid/pull/19249) - Added a mechanism to automatically clean up intermediary files on HDFS storage [#19187](https://github.com/apache/druid/pull/19187) -- Changed the default value for `druid.indexing.formats.maxStringLength` to null from 0 [#19198](https://github.com/apache/druid/pull/19198) ### Metrics and monitoring @@ -501,6 +513,12 @@ Use one of Druid's other supported ingestion methods, such as SQL-based ingestio [#19109](https://github.com/apache/druid/pull/19109) +#### AWS SDK v2 + +Druid now uses AWS SDK version `2.40.0` since v1 of the SDK is at end of life. + +[#18891](https://github.com/apache/druid/pull/18891) + #### Segment metadata cache on by default Starting in Druid 37, the segment metadata cache is on by default. This feature allows the Broker to cache segment metadata polled from the Coordinator, rather than having to fetch metadata for every query against the `sys.segments` table. This improves performance but increases memory usage on Brokers. @@ -509,12 +527,24 @@ The `druid.sql.planner.metadataSegmentCacheEnable` config controls this feature. [#19075](https://github.com/apache/druid/pull/19075) -#### Streaming ingestion `parser` +#### Parser changes + +##### Streaming ingestion `parser` Support for the deprecated `parser` has been removed for streaming ingest tasks such as Kafka and Kinesis. Operators must now specify `inputSource`/`inputFormat` on the `ioConfig` of the supervisor spec, and the `dataSchema` must not specify a parser. Do this before upgrading to Druid 37 or newer. [#19173](https://github.com/apache/druid/pull/19173) [#19166](https://github.com/apache/druid/pull/19166) +##### Removed `ParseSpec` and deprecated parsers + +The Parser for native batch tasks and streaming ingestion indexing services has been removed. Where possible, use the input format instead. Note that `JavascriptParseSpec` and `JSONLowercaseParseSpec` have no InputFormat equivalents. + +Druid supports custom text data formats and can use the Regex input format to parse them. However, be aware doing this to +parse data is less efficient than writing a native Java `InputFormat` extension, or using an external stream processor. We welcome contributions of new input formats. + +[#19239](https://github.com/apache/druid/pull/19239) + + #### Rolling upgrades from Druid versions prior to version 0.23 You can't perform a rolling upgrade from versions earlier than Druid 0.23. @@ -523,9 +553,7 @@ You can't perform a rolling upgrade from versions earlier than Druid 0.23. #### Metadata storage for auto-compaction with compaction supervisors -Automatic compaction using compaction supervisors requires incremental segment metadata caching to be enabled on the Overlord and Coordinator in the runtime properties. - -As part of this update, Druid requires the incremental cache to be enabled and a new table in the metadata store. No action is required if you are using the default settings for the following configs: +Automatic compaction with supervisors requires incremental segment metadata caching on the Overlord and a new metadata store table; no action is required if you are using the default settings for the following configs: - `druid.manager.segments.useIncrementalCache` - `druid.metadata.storage.connector.createTables` @@ -567,15 +595,6 @@ Segment locking and `NumberedOverwriteShardSpec` are deprecated and will be remo ### Incompatible changes -#### Removed `ParseSpec` and deprecated parsers - -The Parser for native batch tasks and streaming ingestion indexing services has been removed. Where possible, use the input format instead. Note that `JavascriptParseSpec` and `JSONLowercaseParseSpec` have no InputFormat equivalents. - -Druid supports custom text data formats and can use the Regex input format to parse them. However, be aware doing this to -parse data is less efficient than writing a native Java `InputFormat` extension, or using an external stream processor. We welcome contributions of new input formats. - -[#19239](https://github.com/apache/druid/pull/19239) - #### Removed `defaultProcessingRate` config This config allowed scaling actions to begin prior to the first metrics becoming available. @@ -621,7 +640,6 @@ The following dependencies have been updated: - `bytebuddy` from `1.18.3` to `1.18.5` [#19145](https://github.com/apache/druid/pull/19145) - Added `objenesis` `3.5` [#19145](https://github.com/apache/druid/pull/19145) - `org.apache.zookeeper` from 3.8.4 to 3.8.6 [#19135](https://github.com/apache/druid/pull/19135) -- Added AWS SDK `2.40.0` [#18891](https://github.com/apache/druid/pull/18891) - `com.lmax.disruptor` from `3.3.6` to `3.4.4` [#19122](https://github.com/apache/druid/pull/19122) - `org.junit.junit-bom` from `5.13.3` to `5.14.3` [#19122](https://github.com/apache/druid/pull/19122) - - `io.fabric8:kubernetes-client` 7.5.2 → 7.6.0 [#19071](https://github.com/apache/druid/pull/19071) From 3958f339415cb22e0864ded31f42833d8097a61b Mon Sep 17 00:00:00 2001 From: brian Date: Wed, 22 Apr 2026 14:08:50 -0700 Subject: [PATCH 15/21] add 18983 --- docs/release-info/release-notes.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/release-info/release-notes.md b/docs/release-info/release-notes.md index 3347f26b3899..edeffa0d3084 100644 --- a/docs/release-info/release-notes.md +++ b/docs/release-info/release-notes.md @@ -105,6 +105,12 @@ You must use compaction supervisors with the MSQ task engine to use cascading re [#18939](https://github.com/apache/druid/pull/18939) [#19213](https://github.com/apache/druid/pull/19213) [#19106](https://github.com/apache/druid/pull/19106) [#19078](https://github.com/apache/druid/pull/19078) +### Multi-supervisor ingestion + +Multi-supervisor ingestion is now generally available. You can run multiple stream supervisors that ingest into the same datasource. + +[#18983](https://github.com/apache/druid/pull/18983) + ### Thrift input format As part of the Thrift contributor extension, Druid now supports Thrift-encoded data for Kafka and Kinesis streaming ingestion using `InputFormat`. Previously, Druid supported this through parsers, which have been removed in Druid 37. From 8a0c99e861f515e7f4ccfd7ccb8a7af265c45099 Mon Sep 17 00:00:00 2001 From: 317brian <53799971+317brian@users.noreply.github.com> Date: Wed, 22 Apr 2026 14:31:03 -0700 Subject: [PATCH 16/21] Apply suggestion from @cecemei Co-authored-by: Cece Mei --- docs/release-info/release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release-info/release-notes.md b/docs/release-info/release-notes.md index edeffa0d3084..5195e812394a 100644 --- a/docs/release-info/release-notes.md +++ b/docs/release-info/release-notes.md @@ -160,7 +160,7 @@ As part of this change, the `/druid/v2/sql/queries` API now supports an `include ### Auto-compaction with compaction supervisors -Auto-compaction using compaction supervisors has been improved and is now generally available. + Auto-compaction using compaction supervisors has been improved, now generally available, and the recommended default. Automatic compaction tasks are now prefixed with `auto` instead of `coordinator-issued`. As part of the improvement compaction states are now stored in a central location, a new `indexingStates` table. Individual segments only need to store a unique reference (`indexing_state_fingerprint`) to their full compaction state. From 65fc1e1ca85c0fea58a9a9a2541c77b49b5edbcf Mon Sep 17 00:00:00 2001 From: brian Date: Wed, 22 Apr 2026 14:35:58 -0700 Subject: [PATCH 17/21] cleanup 19146 --- docs/release-info/release-notes.md | 5 ----- 1 file changed, 5 deletions(-) diff --git a/docs/release-info/release-notes.md b/docs/release-info/release-notes.md index 5195e812394a..d5499085779c 100644 --- a/docs/release-info/release-notes.md +++ b/docs/release-info/release-notes.md @@ -245,11 +245,6 @@ Druid truncates any string longer than the specified length. The default is to n [#19146](https://github.com/apache/druid/pull/19146) [#19258](https://github.com/apache/druid/pull/19258) (https://github.com/apache/druid/pull/19198) - -- Added the `maxStringLength` configuration for string dimensions that truncates values exceeding the specified length during ingestion. You can set the length globally using `druid.indexing.formats.maxStringLength` or per-dimension in the ingestion spec -- Added `StringColumnFormatSpec` for string dimension configs - - #### Other ingestion improvements - Sped up task scheduling on the Overlord [#19199](https://github.com/apache/druid/pull/19199) From 68d05a624c5a52188c29f284122f1109cb55665a Mon Sep 17 00:00:00 2001 From: 317brian <53799971+317brian@users.noreply.github.com> Date: Thu, 23 Apr 2026 09:43:45 -0700 Subject: [PATCH 18/21] Apply suggestion from @abhishekrb19 Co-authored-by: Abhishek Radhakrishnan --- docs/release-info/release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release-info/release-notes.md b/docs/release-info/release-notes.md index d5499085779c..c95a4c8a44b2 100644 --- a/docs/release-info/release-notes.md +++ b/docs/release-info/release-notes.md @@ -411,7 +411,7 @@ The new configuration property `druid.msq.intermediate.storage.cleaner.durationT ### Metrics and monitoring -#### Version field +#### BuildRevision field All Druid metrics now include `version` and `buildRevision` fields to help you identify which version of Druid is emitting a metric. You can use this information to verify that all nodes in a cluster are running the intended revision during rolling deployments. From 155e2e303b4e8cc5e4db9eb79fa1b7444c7e9b21 Mon Sep 17 00:00:00 2001 From: 317brian <53799971+317brian@users.noreply.github.com> Date: Thu, 23 Apr 2026 09:44:22 -0700 Subject: [PATCH 19/21] Apply suggestions from code review Co-authored-by: Abhishek Radhakrishnan --- docs/release-info/release-notes.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/release-info/release-notes.md b/docs/release-info/release-notes.md index c95a4c8a44b2..1cb6c02e0a77 100644 --- a/docs/release-info/release-notes.md +++ b/docs/release-info/release-notes.md @@ -413,7 +413,7 @@ The new configuration property `druid.msq.intermediate.storage.cleaner.durationT #### BuildRevision field -All Druid metrics now include `version` and `buildRevision` fields to help you identify which version of Druid is emitting a metric. You can use this information to verify that all nodes in a cluster are running the intended revision during rolling deployments. +All Druid metrics now include a `buildRevision` field to help identify the Git build revision of the Druid server emitting a metric. You can use this information to verify that all nodes in a cluster are running the intended revision. [#19123](https://github.com/apache/druid/pull/19123) @@ -428,6 +428,7 @@ Added a new `supervisor/count` metric when `SupervisorStatsMonitor` is enabled i `GroupByStatsMonitor` now provides the following metrics: - `mergeBuffer/bytesUsed` +- `mergeBuffer/maxBytesUsed` - `mergeBuffer/maxAcquisitionTimeNs` - `groupBy/maxSpilledBytes` - `groupBy/maxMergeDictionarySize` From aa5c6820c2eec26b472795708fe47097871d4a1d Mon Sep 17 00:00:00 2001 From: brian Date: Thu, 23 Apr 2026 10:28:21 -0700 Subject: [PATCH 20/21] typo --- docs/release-info/release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/release-info/release-notes.md b/docs/release-info/release-notes.md index 1cb6c02e0a77..54fb6aaea2e5 100644 --- a/docs/release-info/release-notes.md +++ b/docs/release-info/release-notes.md @@ -443,7 +443,7 @@ Operators can set `druid.emitter.logging.shouldFilterMetrics=true` to limit whic #### New Broker metrics -Added `segment/schemaCache/rowSignature/changed` and `segment/schemaCache/rowSignature/column/count` metrics to expose events when the Broker initializes and updates the row signature in the the segment metadata cache for each datasource. +Added `segment/schemaCache/rowSignature/changed` and `segment/schemaCache/rowSignature/column/count` metrics to expose events when the Broker initializes and updates the row signature in the segment metadata cache for each datasource. [#18966](https://github.com/apache/druid/pull/18966) From 3fa6979412688d8aedda864122275b8b607f368b Mon Sep 17 00:00:00 2001 From: brian Date: Thu, 23 Apr 2026 10:46:38 -0700 Subject: [PATCH 21/21] typo --- docs/release-info/release-notes.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/release-info/release-notes.md b/docs/release-info/release-notes.md index 54fb6aaea2e5..f98307ffe6eb 100644 --- a/docs/release-info/release-notes.md +++ b/docs/release-info/release-notes.md @@ -644,7 +644,7 @@ The following dependencies have been updated: - `org.apache.zookeeper` from 3.8.4 to 3.8.6 [#19135](https://github.com/apache/druid/pull/19135) - `com.lmax.disruptor` from `3.3.6` to `3.4.4` [#19122](https://github.com/apache/druid/pull/19122) - `org.junit.junit-bom` from `5.13.3` to `5.14.3` [#19122](https://github.com/apache/druid/pull/19122) -- - `io.fabric8:kubernetes-client` 7.5.2 → 7.6.0 [#19071](https://github.com/apache/druid/pull/19071) +- `io.fabric8:kubernetes-client` 7.5.2 → 7.6.0 [#19071](https://github.com/apache/druid/pull/19071) - `io.kubernetes:client-java` 19.0.0 → 25.0.0-legacy [#19071](https://github.com/apache/druid/pull/19071) - `com.squareup.okhttp3:okhttp` 4.12.0 → 5.3.2 [#19071](https://github.com/apache/druid/pull/19071) - `org.jetbrains.kotlin:kotlin-stdlib` 1.9.25 → 2.2.21 [#19071](https://github.com/apache/druid/pull/19071) @@ -658,10 +658,10 @@ The following dependencies have been updated: - `commons-compress` from `1.27.0` to `1.28.0` [#19208](https://github.com/apache/druid/pull/19208) - `zstd-jni` from `1.5.2-3` to `1.5.7-7` [#19208](https://github.com/apache/druid/pull/19208) - `scala-library` from `2.12.7` to `2.13.16` [#19208](https://github.com/apache/druid/pull/19208) -- - `iceberg` from `1.7.2` to `1.10.0` [#19232](https://github.com/apache/druid/pull/19232) +- `iceberg` from `1.7.2` to `1.10.0` [#19232](https://github.com/apache/druid/pull/19232) - `parquet` from `1.15.2` to `1.16.0` [#19232](https://github.com/apache/druid/pull/19232) - `avro` from `1.11.5` to `1.12.0` [#19232](https://github.com/apache/druid/pull/19232) -- - `jackson` from `2.19.2` to `2.20.2` [#19248](https://github.com/apache/druid/pull/19248) +- `jackson` from `2.19.2` to `2.20.2` [#19248](https://github.com/apache/druid/pull/19248) - `netty4` from `4.2.6.Final` to `4.2.12.Final` [#19248](https://github.com/apache/druid/pull/19248) - `errorprone` from `2.35.1` to `2.41.0` [#19248](https://github.com/apache/druid/pull/19248) - `bcprov-jdk18on` / `bcpkix-jdk18on` from `1.81` to `1.82` [#19248](https://github.com/apache/druid/pull/19248)