From 2270725290ed287084d36055308b2f3a76b09eab Mon Sep 17 00:00:00 2001 From: Jill Osborne Date: Mon, 31 Oct 2022 16:06:49 +0000 Subject: [PATCH 01/17] 25.0 release notes: batch 1 --- docs/next-release-notes.md | 273 +++++++++++++++++++++++++++++++++++++ 1 file changed, 273 insertions(+) create mode 100644 docs/next-release-notes.md diff --git a/docs/next-release-notes.md b/docs/next-release-notes.md new file mode 100644 index 000000000000..bd8505589b6b --- /dev/null +++ b/docs/next-release-notes.md @@ -0,0 +1,273 @@ +# New features + +## Updated Kafka support + +Updated the Apache Kafka core dependency to version 3.3.1. + +https://github.com/apache/druid/pull/13176 + + +## Query engine + +### BIG_SUM SQL function + +Added SQL function `BIG_SUM` that uses the [Compressed Big Decimal](https://github.com/apache/druid/pull/10705) Druid extension. + +https://github.com/apache/druid/pull/13102 + +### Add Compressed Big Decimal min and max functions + +Added min and max functions for Compressed Big Decimal and exposed these functions via SQL: BIG_MIN and BIG_MAX. + +https://github.com/apache/druid/pull/13141 + +### Metrics used to downsample bucket + +Changed the way the MSQ task engine determines whether or not to downsample data, to improve accuracy. The task engine now uses the number of bytes instead of number of keys. + +https://github.com/apache/druid/pull/12998 + +### MSQ Docker improvement + +Enabled MSQ query engine for Docker by default. + +https://github.com/apache/druid/pull/13069 + +### Improved MSQ warnings + +For disallowed MSQ warnings of certain types, the warning is now surfaced as the error. + +https://github.com/apache/druid/pull/13198 + + +## Querying + +### Improvements to querying user experience + +Exposed HTTP response headers for SQL queries. + +https://github.com/apache/druid/pull/13052 + +### Enable async reads for JDBC + +Prevented JDBC timeouts on long queries by returning empty batches when a batch fetch takes too long. Uses an async model to run the result fetch concurrently with JDBC requests. + +https://github.com/apache/druid/pull/13196 + +### Enabled composite approach for checking in-filter values set in column dictionary + +To accommodate large value sets arising from large in-filters or from joins pushed down as in-filters, Druid now uses sorted merge algorithm for merging the set and dictionary for larger values. + +https://github.com/apache/druid/pull/13133 + +### Add new configuration keys to query context security model + +Added the following configuration keys that refine the query context security model controlled by `druid.auth.authorizeQueryContextParams`: +* `druid.auth.unsecuredContextKeys`: The set of query context keys that do not require a security check. +* `druid.auth.securedContextKeys`: The set of query context keys that do require a security check. + + +## Nested columns + +### Refactor a data source before unnest + +When data requires "flattening" during processing, the operator now takes in an array and then flattens the array into N (N=number of elements in the array) rows where each row has one of the values from the array. + +https://github.com/apache/druid/pull/13085 + +### Fix JSON_VALUE bug + +Fixed JSON_VALUE SQL planning with decimal type, and fixed vectorized expression math null value handling in default mode. + +https://github.com/apache/druid/pull/13214 + +### Fix issue with nested columns NULL value index + +Fixed queries using IS NULL or IS NOT NULL filters on nested columns that do not contain a null value in every segment, which presented as having a single value per segment matched and returned in the results. + +https://github.com/apache/druid/pull/13211 + + +## Ingestion + +### Improved control for how Druid reads JSON data for streaming ingestion + +You can now better control how Druid reads JSON data for streaming ingestion by setting the following fields in the input format specification: + +* `assumedNewlineDelimited` to parse lines of JSON independently. +* `useJsonNodeReader` to retain valid JSON events when parsing multi-line JSON events when a parsing exception occurs. + +https://github.com/apache/druid/pull/13089 + +### Kafka Consumer improvement + +Allowed Kafka Consumer's custom deserializer to be configured after its instantiation. + +https://github.com/apache/druid/pull/13097 + +### Fix Overlord leader election + +Fixed a problem where Overlord leader election failed due to lock reacquisition issues. Druid now fails these tasks and clears all locks so that the Overlord leader election isn't blocked. + +https://github.com/apache/druid/pull/13172 + +### Add inline descriptor Protobuf bytes decoder + +Added a new `ProtobufBytesDecoder` implementation that allows a user to pass inline the contents of a Protobuf descriptor file, encoded as a Base64 string. + +https://github.com/apache/druid/pull/13192 + +### When a Kafka stream becomes inactive, prevent Supervisor from creating new indexing tasks + +Added Idle feature to `SeekableStreamSupervisor` for inactive stream. + +https://github.com/apache/druid/pull/13144 + + +## Operations + +### Segment discovery + +The default segment discovery method now uses HTTP instead of ZooKeeper. + +https://github.com/apache/druid/pull/13092 + +### Memory estimates + +The task context flag `useMaxMemoryEstimates` is now set to false by default to improve memory usage estimation. + +https://github.com/apache/druid/pull/13178 + +### Docker improvements + +Updated dependencies for the Druid image for Docker, including JRE 11. Docker BuildKit cache is enabled to speed up building. + +https://github.com/apache/druid/pull/13059 + +### Segment replication + +Improved the process of checking server inventory to prevent over-replication of segments during segment balancing. + +https://github.com/apache/druid/pull/13114 + +### Kill tasks do not include markAsUnuseddone + +When you kill a task, Druid no longer automatically marks segments as unused. You must explicitly mark them as unused with `POST /druid/coordinator/v1/datasources/{dataSourceName}/markUnused`. +For more information, see the [API reference](https://druid.apache.org/docs/latest/operations/api-reference.html#coordinator) + +https://github.com/apache/druid/pull/13104 + +### Nested columns performance improvement + +Improved `NestedDataColumnSerializer` to no longer explicitly write null values to the field writers for the missing values of every row. Instead, passing the row counter is moved to the field writers so that they can backfill null values in bulk. + +https://github.com/apache/druid/pull/13101 + +### Provide service specific log4j overrides in containerized deployments + +Provided an option to override log4j configs setup at the service level directories so that it works with Druid-operator based deployments. + +https://github.com/apache/druid/pull/13020 + +### Various Docker improvements + +* Updated Docker to run with JRE 11 by default. +* Updated Docker to use [`gcr.io/distroless/java11-debian11`](https://github.com/GoogleContainerTools/distroless) image as base by default. +* Enabled Docker buildkit cache to speed up building. +* Downloaded [`bash-static`](https://github.com/robxu9/bash-static) to the Docker image so that scripts that require bash can be executed. +* Bumped builder image from `3.8.4-jdk-11-slim` to `3.8.6-jdk-11-slim`. +* Switched busybox from `amd64/busybox:1.30.0-glibc` to `busybox:1.35.0-glibc`. +* Added support to build arm64-based image. + +https://github.com/apache/druid/pull/13059 + +### Improved supervisor termination + +Fixed issues with delayed supervisor termination during certain transient states. + +https://github.com/apache/druid/pull/13072 + +### Fixed problem when running Druid with JDK11+ + +Export `com.sun.management.internal` when running Druid under JRE11 and JRE17. + +https://github.com/apache/druid/pull/13068 + +### Enable cleaner JSON for various input sources and formats + +Added JsonInclude to various properties, to avoid population of default values in serialized JSON. + +https://github.com/apache/druid/pull/13064 + +### Improved metric reporting + +Improved global-cached-lookups metric reporting. + +https://github.com/apache/druid/pull/13219 + +### Fix a bug in HttpPostEmitter + +Fixed a bug in HttpPostEmitter where the emitting thread was prematurely stopped while there was data to be flushed. + +https://github.com/apache/druid/pull/13237 + +### Improve direct memory check on startup + +Improved direct memory check on startup by providing better support for Java 9+ in `RuntimeInfo`, and clearer log messages where validation fails. + +https://github.com/apache/druid/pull/13207 + +### Add a new way of storing STRING type columns + +Added support for 'front coded' string dictionaries for smaller string columns. + +https://github.com/apache/druid/pull/12277 + + +## Extensions + +### Extension optimization + +Optimized the `compareTo` function in `CompressedBigDecimal`. + +https://github.com/apache/druid/pull/13086 + +### CompressedBigDecimal cleanup and extension + +Removed unnecessary generic type from CompressedBigDecimal, added support for number input types, added support for reading aggregator input types directly (uningested data), and fixed scaling bug in buffer aggregator. + +https://github.com/apache/druid/pull/13048 + + +## Web console + +### Old query view is no longer available + +The old query view is removed. Use the new query view with tabs. +For more information, see [Web console](https://druid.apache.org/docs/latest/operations/web-console.html#query). + +https://github.com/apache/druid/pull/13169 + +### Filter column values in query results + +The web console now allows you to add to existing filters for a selected column. + +https://github.com/apache/druid/pull/13169 + +### Add issue comments + +You can now add an issue comment in SQL, for example `--:ISSUE: this is an issue` that is rendered in red and prevents the SQL from running. The comments are used by the spec-to-SQL converter to indicate that something could not be converted. + +https://github.com/apache/druid/pull/13136 + +### Support Kafka lookups + +Added support for Kafka-based lookups rendering and input in the web console. + +https://github.com/apache/druid/pull/13098 + +### Improved array detection + +Added better detection for arrays containing objects. + +https://github.com/apache/druid/pull/13077 From 3f7094cdfd5c8ec4c2e8947b25254e5754e33494 Mon Sep 17 00:00:00 2001 From: Jill Osborne Date: Mon, 7 Nov 2022 11:07:47 +0000 Subject: [PATCH 02/17] Update next-release-notes.md --- docs/next-release-notes.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/next-release-notes.md b/docs/next-release-notes.md index bd8505589b6b..4e16c4287995 100644 --- a/docs/next-release-notes.md +++ b/docs/next-release-notes.md @@ -111,9 +111,9 @@ Fixed a problem where Overlord leader election failed due to lock reacquisition https://github.com/apache/druid/pull/13172 -### Add inline descriptor Protobuf bytes decoder +### Support for inline protobuf descriptor -Added a new `ProtobufBytesDecoder` implementation that allows a user to pass inline the contents of a Protobuf descriptor file, encoded as a Base64 string. +Added a new `inline` type `protoBytesDecoder` that allows a user to pass inline the contents of a Protobuf descriptor file, encoded as a Base64 string. https://github.com/apache/druid/pull/13192 From 01b6e0a972615231438200c14421a5b9a496449d Mon Sep 17 00:00:00 2001 From: Jill Osborne Date: Tue, 8 Nov 2022 12:33:54 +0000 Subject: [PATCH 03/17] Update next-release-notes.md --- docs/next-release-notes.md | 14 +------------- 1 file changed, 1 insertion(+), 13 deletions(-) diff --git a/docs/next-release-notes.md b/docs/next-release-notes.md index 4e16c4287995..48e926148b28 100644 --- a/docs/next-release-notes.md +++ b/docs/next-release-notes.md @@ -75,18 +75,6 @@ When data requires "flattening" during processing, the operator now takes in an https://github.com/apache/druid/pull/13085 -### Fix JSON_VALUE bug - -Fixed JSON_VALUE SQL planning with decimal type, and fixed vectorized expression math null value handling in default mode. - -https://github.com/apache/druid/pull/13214 - -### Fix issue with nested columns NULL value index - -Fixed queries using IS NULL or IS NOT NULL filters on nested columns that do not contain a null value in every segment, which presented as having a single value per segment matched and returned in the results. - -https://github.com/apache/druid/pull/13211 - ## Ingestion @@ -254,7 +242,7 @@ The web console now allows you to add to existing filters for a selected column. https://github.com/apache/druid/pull/13169 -### Add issue comments +### Ability to add issue comments You can now add an issue comment in SQL, for example `--:ISSUE: this is an issue` that is rendered in red and prevents the SQL from running. The comments are used by the spec-to-SQL converter to indicate that something could not be converted. From a5a7749e191a8a3ae240e3b077e4a67f30f405d4 Mon Sep 17 00:00:00 2001 From: Katya Macedo Date: Fri, 11 Nov 2022 14:22:04 -0600 Subject: [PATCH 04/17] 25.0 release notes: batch 2 --- docs/next-release-notes.md | 120 ++++++++++++++++++++++++++++++------- 1 file changed, 100 insertions(+), 20 deletions(-) diff --git a/docs/next-release-notes.md b/docs/next-release-notes.md index 48e926148b28..ddfb85be1e98 100644 --- a/docs/next-release-notes.md +++ b/docs/next-release-notes.md @@ -6,7 +6,6 @@ Updated the Apache Kafka core dependency to version 3.3.1. https://github.com/apache/druid/pull/13176 - ## Query engine ### BIG_SUM SQL function @@ -15,7 +14,7 @@ Added SQL function `BIG_SUM` that uses the [Compressed Big Decimal](https://gith https://github.com/apache/druid/pull/13102 -### Add Compressed Big Decimal min and max functions +### Added Compressed Big Decimal min and max functions Added min and max functions for Compressed Big Decimal and exposed these functions via SQL: BIG_MIN and BIG_MAX. @@ -39,16 +38,46 @@ For disallowed MSQ warnings of certain types, the warning is now surfaced as the https://github.com/apache/druid/pull/13198 +### Added support for indexSpec + +The MSQ task engine now supports the `indexSpec` context parameter. + +https://github.com/apache/druid/pull/13275 + +### Added task start status to the worker report + +Added `pendingTasks` and `runningTasks` fields to the worker report for the MSQ task engine. + +https://github.com/apache/druid/pull/13263 + +### Improved handling of secrets + +When MSQ submits tasks containing SQL with sensitive keys, the keys can get logged in the file. +Druid now masks the sensitive keys in the log files using regular expressions. + +https://github.com/apache/druid/pull/13231 + +### Use worker number to communicate between tasks + +Changed the way WorkerClient communicates between the worker tasks, to abstract away the complexity of resolving the `workerNumber` to the `taskId` from the callers. +Once the WorkerClient writes it's outputs to the durable storage, it adds a file with `__success` in the `workerNumber` output directory for that stage and with its `taskId`. This allows you to determine the worker, which has successfully written its outputs to the durable storage, and differentiate from the partial outputs by orphan or failed worker tasks. + +https://github.com/apache/druid/pull/13062 ## Querying ### Improvements to querying user experience -Exposed HTTP response headers for SQL queries. +This release includes several improvements for querying: + +* Exposed HTTP response headers for SQL queries (https://github.com/apache/druid/pull/13052) +* Added the `shouldFinalize` feature for HLL and quantiles sketches. Druid will no longer finalize aggregators when: + - aggregators appear in the outer level of a query + - aggregators are used as input to an expression or finalizing-field-access post-aggregator -https://github.com/apache/druid/pull/13052 + To provide backwards compatibility, we added a `sqlFinalizeOuterSketches` query context parameter that restores the old behavior (https://github.com/apache/druid/pull/13247) -### Enable async reads for JDBC +### Enabled async reads for JDBC Prevented JDBC timeouts on long queries by returning empty batches when a batch fetch takes too long. Uses an async model to run the result fetch concurrently with JDBC requests. @@ -60,22 +89,20 @@ To accommodate large value sets arising from large in-filters or from joins push https://github.com/apache/druid/pull/13133 -### Add new configuration keys to query context security model +### Added new configuration keys to query context security model Added the following configuration keys that refine the query context security model controlled by `druid.auth.authorizeQueryContextParams`: * `druid.auth.unsecuredContextKeys`: The set of query context keys that do not require a security check. * `druid.auth.securedContextKeys`: The set of query context keys that do require a security check. - ## Nested columns -### Refactor a data source before unnest +### Refactored a data source before unnest When data requires "flattening" during processing, the operator now takes in an array and then flattens the array into N (N=number of elements in the array) rows where each row has one of the values from the array. https://github.com/apache/druid/pull/13085 - ## Ingestion ### Improved control for how Druid reads JSON data for streaming ingestion @@ -93,7 +120,7 @@ Allowed Kafka Consumer's custom deserializer to be configured after its instanti https://github.com/apache/druid/pull/13097 -### Fix Overlord leader election +### Fixed Overlord leader election Fixed a problem where Overlord leader election failed due to lock reacquisition issues. Druid now fails these tasks and clears all locks so that the Overlord leader election isn't blocked. @@ -111,6 +138,29 @@ Added Idle feature to `SeekableStreamSupervisor` for inactive stream. https://github.com/apache/druid/pull/13144 +### Sampling from stream input now respects the configured timeout + +Fixed a problem where sampling from a stream input, such as Kafka or Kinesis, failed to respect the configured timeout when the stream had no records available. You can now set the maximum amount of time in which the entry iterator will return results. + +https://github.com/apache/druid/pull/13296 + +### Streaming tasks resume on Overlord switch + +Fixed a problem where streaming ingestion tasks continued to run until their duration elapsed after the Overlord leader had issued a pause to the tasks. Now, when the Overlord switch occurs right after it has issued a pause to the task, the task remains in a paused state even after the Overlord re-election. + +https://github.com/apache/druid/pull/13223 + +### Fixed Parquet list conversion + +Fixes an issue with Parquet list conversion, where lists of complex objects could unexpectedly be wrapped in an extra object, appearing as `[{"element":},{"element":}...]` instead of the direct list. This changes the behavior of the parquet reader for lists of structured objects to be consistent with other parquet logical list conversions. The data is now fetched directly, more closely matching its expected structure. + +https://github.com/apache/druid/pull/13294 + +### Introduced a tree type to flattenSpec + +Introduced a `tree` type to `flattenSpec`. In the event that a simple hierarchical lookup is required, the `tree` type allows for faster JSON parsing than `jq` and `path` parsing types. + +https://github.com/apache/druid/pull/12177 ## Operations @@ -151,7 +201,7 @@ Improved `NestedDataColumnSerializer` to no longer explicitly write null values https://github.com/apache/druid/pull/13101 -### Provide service specific log4j overrides in containerized deployments +### Provided service specific log4j overrides in containerized deployments Provided an option to override log4j configs setup at the service level directories so that it works with Druid-operator based deployments. @@ -175,15 +225,15 @@ Fixed issues with delayed supervisor termination during certain transient states https://github.com/apache/druid/pull/13072 -### Fixed problem when running Druid with JDK11+ +### Fixed a problem when running Druid with JDK11+ Export `com.sun.management.internal` when running Druid under JRE11 and JRE17. https://github.com/apache/druid/pull/13068 -### Enable cleaner JSON for various input sources and formats +### Enabled cleaner JSON for various input sources and formats -Added JsonInclude to various properties, to avoid population of default values in serialized JSON. +Added `JsonInclude` to various properties, to avoid population of default values in serialized JSON. https://github.com/apache/druid/pull/13064 @@ -193,24 +243,29 @@ Improved global-cached-lookups metric reporting. https://github.com/apache/druid/pull/13219 -### Fix a bug in HttpPostEmitter +### Fixed a bug in HttpPostEmitter Fixed a bug in HttpPostEmitter where the emitting thread was prematurely stopped while there was data to be flushed. https://github.com/apache/druid/pull/13237 -### Improve direct memory check on startup +### Improved direct memory check on startup Improved direct memory check on startup by providing better support for Java 9+ in `RuntimeInfo`, and clearer log messages where validation fails. https://github.com/apache/druid/pull/13207 -### Add a new way of storing STRING type columns +### Added a new way of storing STRING type columns Added support for 'front coded' string dictionaries for smaller string columns. https://github.com/apache/druid/pull/12277 +### Improved the run time of the MarkAsUnusedOvershadowedSegments duty + +Improved the run time of the MarkAsUnusedOvershadowedSegments duty by iterating over all overshadowed segments and marking segments as unused in batches. + +https://github.com/apache/druid/pull/13287 ## Extensions @@ -226,10 +281,23 @@ Removed unnecessary generic type from CompressedBigDecimal, added support for nu https://github.com/apache/druid/pull/13048 +### Support for running tasks as Kubernetes jobs + +Added an extension that allows Druid to use Kubernetes for launching and managing tasks, eliminating the need for MiddleManagers. +To use this extension, [include](../extensions.md#loading-extensions) `druid-kubernetes-overlord-extensions` in the extensions load list for your Overlord process. + +https://github.com/apache/druid/pull/13156 + +### Support for Kubernetes discovery + +Added `POD_NAME` and `POD_NAMESPACE` env variables to all Kubernetes Deployments and StatefulSets. +Helm deployment is now compatible with `druid-kubernetes-extension`. + +https://github.com/apache/druid/pull/13262 ## Web console -### Old query view is no longer available +### Removed the old query view The old query view is removed. Use the new query view with tabs. For more information, see [Web console](https://druid.apache.org/docs/latest/operations/web-console.html#query). @@ -244,11 +312,11 @@ https://github.com/apache/druid/pull/13169 ### Ability to add issue comments -You can now add an issue comment in SQL, for example `--:ISSUE: this is an issue` that is rendered in red and prevents the SQL from running. The comments are used by the spec-to-SQL converter to indicate that something could not be converted. +You can now add an issue comment in SQL, for example `--:ISSUE: this is an issue` that is rendered in red and prevents the SQL from running. The comments are used by the spec-to-SQL converter to indicate that something could not be converted. https://github.com/apache/druid/pull/13136 -### Support Kafka lookups +### Support for Kafka lookups Added support for Kafka-based lookups rendering and input in the web console. @@ -259,3 +327,15 @@ https://github.com/apache/druid/pull/13098 Added better detection for arrays containing objects. https://github.com/apache/druid/pull/13077 + +### Updated Druid Query Toolkit version + +[Druid Query Toolkit](https://www.npmjs.com/package/druid-query-toolkit) version 0.16.1 adds quotes to references in auto generated queries by default. + +https://github.com/apache/druid/pull/13243 + +### Query task status information + +The web console now exposes a textual indication about running and pending tasks when a query is stuck due to lack of task slots. + +https://github.com/apache/druid/pull/13291 \ No newline at end of file From 4d578be000fd3160d4731b03a7d3ca0ef157ecb0 Mon Sep 17 00:00:00 2001 From: Katya Macedo Date: Mon, 21 Nov 2022 10:40:48 -0600 Subject: [PATCH 05/17] Add link to related changes --- docs/next-release-notes.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/next-release-notes.md b/docs/next-release-notes.md index ddfb85be1e98..cb0173826cc0 100644 --- a/docs/next-release-notes.md +++ b/docs/next-release-notes.md @@ -47,6 +47,7 @@ https://github.com/apache/druid/pull/13275 ### Added task start status to the worker report Added `pendingTasks` and `runningTasks` fields to the worker report for the MSQ task engine. +See [Query task status information](#query-task-status-information) for related web console changes. https://github.com/apache/druid/pull/13263 From 6b2d3de26b815641d0c160d1fbad173afda1ea16 Mon Sep 17 00:00:00 2001 From: "brian.le" Date: Wed, 23 Nov 2022 17:47:03 -0800 Subject: [PATCH 06/17] next set of release notes after 25.0 branch creation --- docs/next-release-notes.md | 160 ++++++++++++++++++++++++++++++++++++- 1 file changed, 157 insertions(+), 3 deletions(-) diff --git a/docs/next-release-notes.md b/docs/next-release-notes.md index cb0173826cc0..68d5052295b7 100644 --- a/docs/next-release-notes.md +++ b/docs/next-release-notes.md @@ -26,9 +26,15 @@ Changed the way the MSQ task engine determines whether or not to downsample data https://github.com/apache/druid/pull/12998 +### MSQ heap footprint + +When determining partition boundaries, the heap footprint of the sketches that MSQ uses is capped at 10% of available memory or 300 MB, whichever is lower. Previously, the cap was strictly 300 MB. + +https://github.com/apache/druid/pull/13274 + ### MSQ Docker improvement -Enabled MSQ query engine for Docker by default. +Enabled MSQ task query engine for Docker by default. https://github.com/apache/druid/pull/13069 @@ -40,7 +46,7 @@ https://github.com/apache/druid/pull/13198 ### Added support for indexSpec -The MSQ task engine now supports the `indexSpec` context parameter. +The MSQ task engine now supports the `indexSpec` context parameter. This context parameter can also be configured through the web console. https://github.com/apache/druid/pull/13275 @@ -65,6 +71,12 @@ Once the WorkerClient writes it's outputs to the durable storage, it adds a file https://github.com/apache/druid/pull/13062 +### Sketch merging mode + +When a query requires key statistics to generate partition boundaries, key statistics are gathered by the workers while reading rows from the datasource.You can now configure whether the MSQ task engine does this task in parallel or sequentially. Configure the behavior using `clusterStatisticsMergeMode` context parameter. For more information, see [Sketch merging mode](https://druid.apache.org/docs/latest/multi-stage-query/reference.html#sketch-merging-mode). + +https://github.com/apache/druid/pull/13205 + ## Querying ### Improvements to querying user experience @@ -98,6 +110,14 @@ Added the following configuration keys that refine the query context security mo ## Nested columns +### Support for more formats + +Druid nested columns and associated JSON transform functions now supports Avro, ORC, and Parquet. + +https://github.com/apache/druid/pull/13325 + +https://github.com/apache/druid/pull/13375 + ### Refactored a data source before unnest When data requires "flattening" during processing, the operator now takes in an array and then flattens the array into N (N=number of elements in the array) rows where each row has one of the values from the array. @@ -106,6 +126,24 @@ https://github.com/apache/druid/pull/13085 ## Ingestion +### Improved filtering for cloud objects + +You can now stop at arbitrary subfolders using glob syntax in the `ioConfig.inputSource.filter` field for native batch ingestion from cloud storage, such as S3. + +https://github.com/apache/druid/pull/13027 + +### CLUSTERED BY limit + +When using the MSQ task engine to ingest data, there is now a 1,500 column limit to the number of columns that can be passed in the CLUSTERED BY clause. + +https://github.com/apache/druid/pull/13352 + +### Async task client for streaming ingestion + +You can now use asynchronous communication with indexing tasks by setting `chatAsync` to true in the `tuningConfig`. Enabling asynchronous communication means that the `chatThreads` property is ignored. + +https://github.com/apache/druid/pull/13354 + ### Improved control for how Druid reads JSON data for streaming ingestion You can now better control how Druid reads JSON data for streaming ingestion by setting the following fields in the input format specification: @@ -113,14 +151,24 @@ You can now better control how Druid reads JSON data for streaming ingestion by * `assumedNewlineDelimited` to parse lines of JSON independently. * `useJsonNodeReader` to retain valid JSON events when parsing multi-line JSON events when a parsing exception occurs. +The web console has been updated to include these options. + https://github.com/apache/druid/pull/13089 + + ### Kafka Consumer improvement Allowed Kafka Consumer's custom deserializer to be configured after its instantiation. https://github.com/apache/druid/pull/13097 +### Kafka supervisor logging + +Kafka supervisor logs are now less noisy. The supervisors now log events at the DEBUG level instead of INFO. + +https://github.com/apache/druid/pull/13392 + ### Fixed Overlord leader election Fixed a problem where Overlord leader election failed due to lock reacquisition issues. Druid now fails these tasks and clears all locks so that the Overlord leader election isn't blocked. @@ -133,12 +181,20 @@ Added a new `inline` type `protoBytesDecoder` that allows a user to pass inline https://github.com/apache/druid/pull/13192 +### Duplicate notices + +For streaming ingestion, notices that are the same as one already in queue won't be enqueued. This will help reduce notice queue size. + +https://github.com/apache/druid/pull/13334 + ### When a Kafka stream becomes inactive, prevent Supervisor from creating new indexing tasks Added Idle feature to `SeekableStreamSupervisor` for inactive stream. https://github.com/apache/druid/pull/13144 +### + ### Sampling from stream input now respects the configured timeout Fixed a problem where sampling from a stream input, such as Kafka or Kinesis, failed to respect the configured timeout when the stream had no records available. You can now set the maximum amount of time in which the entry iterator will return results. @@ -165,6 +221,100 @@ https://github.com/apache/druid/pull/12177 ## Operations +### Compaction + +Compaction behavior has changed to improve the amount of time it takes and disk space it takes: + +- When segments need to be fetched, download them one at a time and delete them when Druid is done with them. This still takes time but minimizes the required disk space. +- Don't fetch segments on the main compact task when they aren't needed. If the user provides a full `granularitySpec`, `dimensionsSpec`, and `metricsSpec`, Druid skips fetching segments. + +For more information, see the documentation on [Compaction](https://druid.apache.org/docs/latest/data-management/compaction.html) and [Automatic compaction](https://druid.apache.org/docs/latest/data-management/automatic-compaction.html). + +https://github.com/apache/druid/pull/13280 + +### New metric for segments + +`segment/handoff/time` captures the total time taken for handoff for a given set of published segments. + +https://github.com/apache/druid/pull/13238 + +### Idle configs for the Supervisor + +You can now configure the following properties: + +| Property | Description | Default | +| - | - | -| +|`druid.supervisor.idleConfig.enabled`| (Cluster wide) If `true`, supervisor can become idle if there is no data on input stream/topic for some time.|false| +|`druid.supervisor.idleConfig.inactiveAfterMillis`| (Cluster wide) Supervisor is marked as idle if all existing data has been read from input topic and no new data has been published for `inactiveAfterMillis` milliseconds.|`600_000`| +| `inactiveAfterMillis` | (Individual Supervisor) Supervisor is marked as idle if all existing data has been read from input topic and no new data has been published for `inactiveAfterMillis` milliseconds. | no (default == `600_000`) | + +https://github.com/apache/druid/pull/13311 + +### cachingCost balancer strategy + +The `cachingCost` balancer strategy now behaves more similarly to cost strategy. When computing the cost of moving a segment to a server, the following calculations are performed: + +- Subtract the self cost of a segment if it is being served by the target server +- Subtract the cost of segments that are marked to be dropped + +https://github.com/apache/druid/pull/13321 + +### New metrics for streaming ingestion + +The following metrics related to streaming ingestion have been added: + +- `ingest/kafka/partitionLag`: Partition-wise lag between the offsets consumed by the Kafka indexing tasks and latest offsets in Kafka brokers. +- `ingest/kinesis/partitionLag/time`: Partition-wise lag time in milliseconds between the current message sequence number consumed by the Kinesis indexing tasks and latest sequence number in Kinesis. +- `ingest/pause/time`: Milliseconds spent by a task in a paused state without ingesting.|dataSource, taskId| < 10 seconds.| + +https://github.com/apache/druid/pull/13331 +https://github.com/apache/druid/pull/13313 + +### Backoff for HttpPostEmitter + +The `HttpPostEmitter` option now has a backoff. This means that there should be less noise in the logs and lower CPU usage if you use this option for logging. + +https://github.com/apache/druid/pull/12102 + +### taskActionType dimension for task/action/run/time metric + +The `task/action/run/time` metric for the Indexing service now includes the `taskActionType` dimension. + +https://github.com/apache/druid/pull/13333 + +### DumpSegment tool for nested columns + +The DumpSegment tool can now be used on nested columns with the `--dump nested` option. + +For more information, see [dump-segment tool](https://druid.apache.org/docs/latest/operations/dump-segment). + +https://github.com/apache/druid/pull/13356 + +### Segment assignment + +You can now use a round-robin segment strategy to speed up initial segment assignments. + +Set `useRoundRobinSegmentAssigment` to `true` in the Coordinator dynamic config to enable this feature. + +https://github.com/apache/druid/pull/13367 + +### Segment balancing + +Batch sampling is now the default method for sampling segments during balancing as it performs significantly better than the alternative when there is a large number of used segments in the cluster. + +As part of this change, the following properties have been deprecated and will be removed in future releases: + +- `coordinator dynamic config useBatchedSegmentSampler` +- `coordinator dynamic config percentOfSegmentsToConsiderPerMove` +- non-batch method of sampling segments used during coordinator duty `BalanceSegments` + +The unused coordinator property `druid.coordinator.loadqueuepeon.repeatDelay` has been removed. + +Use only `druid.coordinator.loadqueuepeon.http.repeatDelay` to configure repeat delay for the HTTP-based segment loading queue. + +https://github.com/apache/druid/pull/13391 + + ### Segment discovery The default segment discovery method now uses HTTP instead of ZooKeeper. @@ -339,4 +489,8 @@ https://github.com/apache/druid/pull/13243 The web console now exposes a textual indication about running and pending tasks when a query is stuck due to lack of task slots. -https://github.com/apache/druid/pull/13291 \ No newline at end of file +https://github.com/apache/druid/pull/13291 + +### Query history + +Multi-stage queries no longer show up in the Query history dialog. They are still available in the **Recent query tasks** panel. \ No newline at end of file From b91e41801001105619d6bd03dcdbc0c1531cded5 Mon Sep 17 00:00:00 2001 From: "brian.le" Date: Tue, 29 Nov 2022 12:17:47 -0800 Subject: [PATCH 07/17] address feedback from kfaraz --- docs/next-release-notes.md | 67 +++++++++++++++++++++++++------------- 1 file changed, 44 insertions(+), 23 deletions(-) diff --git a/docs/next-release-notes.md b/docs/next-release-notes.md index 68d5052295b7..3d5289f7e451 100644 --- a/docs/next-release-notes.md +++ b/docs/next-release-notes.md @@ -1,10 +1,25 @@ -# New features - -## Updated Kafka support - -Updated the Apache Kafka core dependency to version 3.3.1. - -https://github.com/apache/druid/pull/13176 +--- +title: "WIP release notes for 25.0" +--- + + ## Query engine @@ -155,7 +170,11 @@ The web console has been updated to include these options. https://github.com/apache/druid/pull/13089 +## Updated Kafka version + +Updated the Apache Kafka core dependency to version 3.3.1. +https://github.com/apache/druid/pull/13176 ### Kafka Consumer improvement @@ -193,7 +212,6 @@ Added Idle feature to `SeekableStreamSupervisor` for inactive stream. https://github.com/apache/druid/pull/13144 -### ### Sampling from stream input now respects the configured timeout @@ -250,13 +268,6 @@ You can now configure the following properties: https://github.com/apache/druid/pull/13311 -### cachingCost balancer strategy - -The `cachingCost` balancer strategy now behaves more similarly to cost strategy. When computing the cost of moving a segment to a server, the following calculations are performed: - -- Subtract the self cost of a segment if it is being served by the target server -- Subtract the cost of segments that are marked to be dropped - https://github.com/apache/druid/pull/13321 ### New metrics for streaming ingestion @@ -290,7 +301,16 @@ For more information, see [dump-segment tool](https://druid.apache.org/docs/late https://github.com/apache/druid/pull/13356 -### Segment assignment +### Segmen loading and balancings + +#### cachingCost balancer strategy + +The `cachingCost` balancer strategy now behaves more similarly to cost strategy. When computing the cost of moving a segment to a server, the following calculations are performed: + +- Subtract the self cost of a segment if it is being served by the target server +- Subtract the cost of segments that are marked to be dropped + +#### Segment assignment You can now use a round-robin segment strategy to speed up initial segment assignments. @@ -298,7 +318,7 @@ Set `useRoundRobinSegmentAssigment` to `true` in the Coordinator dynamic config https://github.com/apache/druid/pull/13367 -### Segment balancing +#### Segment batch balancing Batch sampling is now the default method for sampling segments during balancing as it performs significantly better than the alternative when there is a large number of used segments in the cluster. @@ -314,13 +334,18 @@ Use only `druid.coordinator.loadqueuepeon.http.repeatDelay` to configure repeat https://github.com/apache/druid/pull/13391 - -### Segment discovery +#### Segment discovery The default segment discovery method now uses HTTP instead of ZooKeeper. https://github.com/apache/druid/pull/13092 +#### Segment replication + +Improved the process of checking server inventory to prevent over-replication of segments during segment balancing. + +https://github.com/apache/druid/pull/13114 + ### Memory estimates The task context flag `useMaxMemoryEstimates` is now set to false by default to improve memory usage estimation. @@ -333,11 +358,7 @@ Updated dependencies for the Druid image for Docker, including JRE 11. Docker Bu https://github.com/apache/druid/pull/13059 -### Segment replication -Improved the process of checking server inventory to prevent over-replication of segments during segment balancing. - -https://github.com/apache/druid/pull/13114 ### Kill tasks do not include markAsUnuseddone From 5e5c6a8f24325286887bda0bb2cc97a1de92b0ad Mon Sep 17 00:00:00 2001 From: 317brian <53799971+317brian@users.noreply.github.com> Date: Mon, 5 Dec 2022 14:18:54 -0800 Subject: [PATCH 08/17] Apply suggestions from code review Co-authored-by: Kashif Faraz --- docs/next-release-notes.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/next-release-notes.md b/docs/next-release-notes.md index 3d5289f7e451..84b97e5c7019 100644 --- a/docs/next-release-notes.md +++ b/docs/next-release-notes.md @@ -301,7 +301,7 @@ For more information, see [dump-segment tool](https://druid.apache.org/docs/late https://github.com/apache/druid/pull/13356 -### Segmen loading and balancings +### Segment loading and balancing #### cachingCost balancer strategy @@ -318,15 +318,15 @@ Set `useRoundRobinSegmentAssigment` to `true` in the Coordinator dynamic config https://github.com/apache/druid/pull/13367 -#### Segment batch balancing +#### Sampling segments for balancing Batch sampling is now the default method for sampling segments during balancing as it performs significantly better than the alternative when there is a large number of used segments in the cluster. -As part of this change, the following properties have been deprecated and will be removed in future releases: +As part of this change, the following have been deprecated and will be removed in future releases: -- `coordinator dynamic config useBatchedSegmentSampler` -- `coordinator dynamic config percentOfSegmentsToConsiderPerMove` -- non-batch method of sampling segments used during coordinator duty `BalanceSegments` +- coordinator dynamic config `useBatchedSegmentSampler` +- coordinator dynamic config `percentOfSegmentsToConsiderPerMove` +- non-batch method of sampling segments used by coordinator duty `BalanceSegments` The unused coordinator property `druid.coordinator.loadqueuepeon.repeatDelay` has been removed. From 5f18d1686946d15baf591bf4a50fe677d9fe59c3 Mon Sep 17 00:00:00 2001 From: "brian.le" Date: Tue, 6 Dec 2022 14:02:08 -0800 Subject: [PATCH 09/17] address feedback from kashif and will --- docs/next-release-notes.md | 304 +++++++++++++++++++++---------------- 1 file changed, 177 insertions(+), 127 deletions(-) diff --git a/docs/next-release-notes.md b/docs/next-release-notes.md index 84b97e5c7019..835bb6bc2369 100644 --- a/docs/next-release-notes.md +++ b/docs/next-release-notes.md @@ -21,19 +21,82 @@ title: "WIP release notes for 25.0" ~ under the License. --> -## Query engine +## Highlights -### BIG_SUM SQL function +### Multi-stage query -Added SQL function `BIG_SUM` that uses the [Compressed Big Decimal](https://github.com/apache/druid/pull/10705) Druid extension. +The multi-stage query (MSQ) task engine used for SQL-based ingestion is now production ready. Use it for any supported workloads. For more information, see the following pages: -https://github.com/apache/druid/pull/13102 +- [Ingestion](https://druid.apache.org/docs/latest/ingestion/index.html) +- [SQL-based ingestion](https://druid.apache.org/docs/latest/multi-stage-query/index.html) -### Added Compressed Big Decimal min and max functions +### String dictionary compression (experimental) -Added min and max functions for Compressed Big Decimal and exposed these functions via SQL: BIG_MIN and BIG_MAX. +> Any segment written using string dictionary compression is not readable by older versions of Druid. -https://github.com/apache/druid/pull/13141 +Added support for front coded string dictionaries for smaller string columns, leading to reduced segment sizes with only minor performance penalties for most Druid queries. + +This functionality can be utilized by a new property to `IndexSpec.stringDictionaryEncoding`, which can be set to {"type":"frontCoded", "bucketSize": 4}, {"type":"frontCoded", "bucketSize": 16}, or any power of 2 that is 128 or lower. This property instructs indexing tasks to write segments with the compressed dictionaries with the specific bucket size specified. (`{"type":"utf8"}` is the default). + +For more information, see [Front coding](https://druid.apache.org/docs/latest/ingestion/ingestion-spec.html#front-coding). + +https://github.com/apache/druid/pull/12277 + +### Kubernetes native tasks + +Druid can now use Kubernetes to launch and manage tasks, eliminating the need for MiddleManagers. + +To use this feature, enable the [`druid-kubernetes-overlord-extensions`]((../extensions.md#loading-extensions) in the extensions load list for your Overlord process. + +https://github.com/apache/druid/pull/13156 + +## Behavior changes + +### HLL and quantiles sketches + +The aggregation functions for HLL and quantiles sketches returned sketches or numbers when they are finalized depending on where they were in the native query plan. + +Druid no longer finalizes aggregators in the following two cases: + + - aggregators appear in the outer level of a query + - aggregators are used as input to an expression or finalizing-field-access post-aggregator + +This change aligns the behavior of HLL and quantiles sketches with theta sketches. + +To provide backwards compatibility, you can use the `sqlFinalizeOuterSketches` query context parameter that restores the old behavior. + +https://github.com/apache/druid/pull/13247 + +### Kill tasks do not include markAsUnuseddone + +When you kill a task, Druid no longer automatically marks segments as unused. You must explicitly mark them as unused with `POST /druid/coordinator/v1/datasources/{dataSourceName}/markUnused`. +For more information, see the [API reference](https://druid.apache.org/docs/latest/operations/api-reference.html#coordinator) + +https://github.com/apache/druid/pull/13104 + +### Segment discovery + +The default segment discovery method now uses HTTP instead of ZooKeeper. + +This update changes the defaults for the following properties: + +| Property | New default | Previous default | +| - | - | - | +| `druid.serverview.type` for segment management | http | batch | +| `druid.coordinator.loadqueuepeon.type` for segment management | http | curator | +| `druid.indexer.runner.type` for the Overlord | httpRemote | local | + +To use ZooKeeper instead of HTTP, change the values for the properties back to the previous defaults. + +https://github.com/apache/druid/pull/13092 + +## Multi-stage query task engine + +### CLUSTERED BY limit + +When using the MSQ task engine to ingest data, there is now a 1,500 column limit to the number of columns that can be passed in the CLUSTERED BY clause. + +https://github.com/apache/druid/pull/13352 ### Metrics used to downsample bucket @@ -53,6 +116,7 @@ Enabled MSQ task query engine for Docker by default. https://github.com/apache/druid/pull/13069 + ### Improved MSQ warnings For disallowed MSQ warnings of certain types, the warning is now surfaced as the error. @@ -79,6 +143,12 @@ Druid now masks the sensitive keys in the log files using regular expressions. https://github.com/apache/druid/pull/13231 +### Query history + +Multi-stage queries no longer show up in the Query history dialog. They are still available in the **Recent query tasks** panel. + +## Query engine + ### Use worker number to communicate between tasks Changed the way WorkerClient communicates between the worker tasks, to abstract away the complexity of resolving the `workerNumber` to the `taskId` from the callers. @@ -94,16 +164,11 @@ https://github.com/apache/druid/pull/13205 ## Querying -### Improvements to querying user experience - -This release includes several improvements for querying: +### HTTP response headers -* Exposed HTTP response headers for SQL queries (https://github.com/apache/druid/pull/13052) -* Added the `shouldFinalize` feature for HLL and quantiles sketches. Druid will no longer finalize aggregators when: - - aggregators appear in the outer level of a query - - aggregators are used as input to an expression or finalizing-field-access post-aggregator +Exposed HTTP response headers for SQL queries. - To provide backwards compatibility, we added a `sqlFinalizeOuterSketches` query context parameter that restores the old behavior (https://github.com/apache/druid/pull/13247) +https://github.com/apache/druid/pull/13052 ### Enabled async reads for JDBC @@ -123,11 +188,48 @@ Added the following configuration keys that refine the query context security mo * `druid.auth.unsecuredContextKeys`: The set of query context keys that do not require a security check. * `druid.auth.securedContextKeys`: The set of query context keys that do require a security check. +## Metrics + +### Improved metric reporting + +Improved global-cached-lookups metric reporting. + +https://github.com/apache/druid/pull/13219 + + +### New metric for segments + +`segment/handoff/time` captures the total time taken for handoff for a given set of published segments. + +https://github.com/apache/druid/pull/13238 + +### New metrics for streaming ingestion + +The following metrics related to streaming ingestion have been added: + +- `ingest/kafka/partitionLag`: Partition-wise lag between the offsets consumed by the Kafka indexing tasks and latest offsets in Kafka brokers. +- `ingest/kinesis/partitionLag/time`: Partition-wise lag time in milliseconds between the current message sequence number consumed by the Kinesis indexing tasks and latest sequence number in Kinesis. +- `ingest/pause/time`: Milliseconds spent by a task in a paused state without ingesting.|dataSource, taskId| < 10 seconds.| + +https://github.com/apache/druid/pull/13331 +https://github.com/apache/druid/pull/13313 + +### taskActionType dimension for task/action/run/time metric + +The `task/action/run/time` metric for the Indexing service now includes the `taskActionType` dimension. + +https://github.com/apache/druid/pull/13333 + + ## Nested columns +### Nested columns performance improvement + +Improved `NestedDataColumnSerializer` to no longer explicitly write null values to the field writers for the missing values of every row. Instead, passing the row counter is moved to the field writers so that they can backfill null values in bulk. + ### Support for more formats -Druid nested columns and associated JSON transform functions now supports Avro, ORC, and Parquet. +Druid nested columns and the associated JSON transform functions now support Avro, ORC, and Parquet. https://github.com/apache/druid/pull/13325 @@ -147,11 +249,6 @@ You can now stop at arbitrary subfolders using glob syntax in the `ioConfig.inpu https://github.com/apache/druid/pull/13027 -### CLUSTERED BY limit - -When using the MSQ task engine to ingest data, there is now a 1,500 column limit to the number of columns that can be passed in the CLUSTERED BY clause. - -https://github.com/apache/druid/pull/13352 ### Async task client for streaming ingestion @@ -170,11 +267,11 @@ The web console has been updated to include these options. https://github.com/apache/druid/pull/13089 -## Updated Kafka version +### When a Kafka stream becomes inactive, prevent Supervisor from creating new indexing tasks -Updated the Apache Kafka core dependency to version 3.3.1. +Added Idle feature to `SeekableStreamSupervisor` for inactive stream. -https://github.com/apache/druid/pull/13176 +https://github.com/apache/druid/pull/13144 ### Kafka Consumer improvement @@ -206,12 +303,6 @@ For streaming ingestion, notices that are the same as one already in queue won't https://github.com/apache/druid/pull/13334 -### When a Kafka stream becomes inactive, prevent Supervisor from creating new indexing tasks - -Added Idle feature to `SeekableStreamSupervisor` for inactive stream. - -https://github.com/apache/druid/pull/13144 - ### Sampling from stream input now respects the configured timeout @@ -250,15 +341,11 @@ For more information, see the documentation on [Compaction](https://druid.apache https://github.com/apache/druid/pull/13280 -### New metric for segments - -`segment/handoff/time` captures the total time taken for handoff for a given set of published segments. - -https://github.com/apache/druid/pull/13238 - ### Idle configs for the Supervisor -You can now configure the following properties: +You can now set the Supervisor to idle, which is useful in cases where freeing up slots so that autoscaling can be more effective. + +To configure the idle behavior, use the following properties: | Property | Description | Default | | - | - | -| @@ -270,16 +357,12 @@ https://github.com/apache/druid/pull/13311 https://github.com/apache/druid/pull/13321 -### New metrics for streaming ingestion +### Improved supervisor termination -The following metrics related to streaming ingestion have been added: +Fixed issues with delayed supervisor termination during certain transient states. -- `ingest/kafka/partitionLag`: Partition-wise lag between the offsets consumed by the Kafka indexing tasks and latest offsets in Kafka brokers. -- `ingest/kinesis/partitionLag/time`: Partition-wise lag time in milliseconds between the current message sequence number consumed by the Kinesis indexing tasks and latest sequence number in Kinesis. -- `ingest/pause/time`: Milliseconds spent by a task in a paused state without ingesting.|dataSource, taskId| < 10 seconds.| +https://github.com/apache/druid/pull/13072 -https://github.com/apache/druid/pull/13331 -https://github.com/apache/druid/pull/13313 ### Backoff for HttpPostEmitter @@ -287,12 +370,6 @@ The `HttpPostEmitter` option now has a backoff. This means that there should be https://github.com/apache/druid/pull/12102 -### taskActionType dimension for task/action/run/time metric - -The `task/action/run/time` metric for the Indexing service now includes the `taskActionType` dimension. - -https://github.com/apache/druid/pull/13333 - ### DumpSegment tool for nested columns The DumpSegment tool can now be used on nested columns with the `--dump nested` option. @@ -318,7 +395,7 @@ Set `useRoundRobinSegmentAssigment` to `true` in the Coordinator dynamic config https://github.com/apache/druid/pull/13367 -#### Sampling segments for balancing +#### Segment load queue peon Batch sampling is now the default method for sampling segments during balancing as it performs significantly better than the alternative when there is a large number of used segments in the cluster. @@ -334,11 +411,7 @@ Use only `druid.coordinator.loadqueuepeon.http.repeatDelay` to configure repeat https://github.com/apache/druid/pull/13391 -#### Segment discovery - -The default segment discovery method now uses HTTP instead of ZooKeeper. -https://github.com/apache/druid/pull/13092 #### Segment replication @@ -352,24 +425,8 @@ The task context flag `useMaxMemoryEstimates` is now set to false by default to https://github.com/apache/druid/pull/13178 -### Docker improvements -Updated dependencies for the Druid image for Docker, including JRE 11. Docker BuildKit cache is enabled to speed up building. -https://github.com/apache/druid/pull/13059 - - - -### Kill tasks do not include markAsUnuseddone - -When you kill a task, Druid no longer automatically marks segments as unused. You must explicitly mark them as unused with `POST /druid/coordinator/v1/datasources/{dataSourceName}/markUnused`. -For more information, see the [API reference](https://druid.apache.org/docs/latest/operations/api-reference.html#coordinator) - -https://github.com/apache/druid/pull/13104 - -### Nested columns performance improvement - -Improved `NestedDataColumnSerializer` to no longer explicitly write null values to the field writers for the missing values of every row. Instead, passing the row counter is moved to the field writers so that they can backfill null values in bulk. https://github.com/apache/druid/pull/13101 @@ -391,17 +448,7 @@ https://github.com/apache/druid/pull/13020 https://github.com/apache/druid/pull/13059 -### Improved supervisor termination - -Fixed issues with delayed supervisor termination during certain transient states. - -https://github.com/apache/druid/pull/13072 - -### Fixed a problem when running Druid with JDK11+ - -Export `com.sun.management.internal` when running Druid under JRE11 and JRE17. -https://github.com/apache/druid/pull/13068 ### Enabled cleaner JSON for various input sources and formats @@ -409,29 +456,12 @@ Added `JsonInclude` to various properties, to avoid population of default values https://github.com/apache/druid/pull/13064 -### Improved metric reporting - -Improved global-cached-lookups metric reporting. - -https://github.com/apache/druid/pull/13219 - -### Fixed a bug in HttpPostEmitter - -Fixed a bug in HttpPostEmitter where the emitting thread was prematurely stopped while there was data to be flushed. - -https://github.com/apache/druid/pull/13237 - ### Improved direct memory check on startup Improved direct memory check on startup by providing better support for Java 9+ in `RuntimeInfo`, and clearer log messages where validation fails. https://github.com/apache/druid/pull/13207 -### Added a new way of storing STRING type columns - -Added support for 'front coded' string dictionaries for smaller string columns. - -https://github.com/apache/druid/pull/12277 ### Improved the run time of the MarkAsUnusedOvershadowedSegments duty @@ -439,33 +469,6 @@ Improved the run time of the MarkAsUnusedOvershadowedSegments duty by iterating https://github.com/apache/druid/pull/13287 -## Extensions - -### Extension optimization - -Optimized the `compareTo` function in `CompressedBigDecimal`. - -https://github.com/apache/druid/pull/13086 - -### CompressedBigDecimal cleanup and extension - -Removed unnecessary generic type from CompressedBigDecimal, added support for number input types, added support for reading aggregator input types directly (uningested data), and fixed scaling bug in buffer aggregator. - -https://github.com/apache/druid/pull/13048 - -### Support for running tasks as Kubernetes jobs - -Added an extension that allows Druid to use Kubernetes for launching and managing tasks, eliminating the need for MiddleManagers. -To use this extension, [include](../extensions.md#loading-extensions) `druid-kubernetes-overlord-extensions` in the extensions load list for your Overlord process. - -https://github.com/apache/druid/pull/13156 - -### Support for Kubernetes discovery - -Added `POD_NAME` and `POD_NAMESPACE` env variables to all Kubernetes Deployments and StatefulSets. -Helm deployment is now compatible with `druid-kubernetes-extension`. - -https://github.com/apache/druid/pull/13262 ## Web console @@ -512,6 +515,53 @@ The web console now exposes a textual indication about running and pending tasks https://github.com/apache/druid/pull/13291 -### Query history +## Extensions + +### BIG_SUM SQL + +#### BIG_SUM SQL function + +Added SQL function `BIG_SUM` that uses the [Compressed Big Decimal](https://github.com/apache/druid/pull/10705) Druid extension. + +https://github.com/apache/druid/pull/13102 + +#### Added Compressed Big Decimal min and max functions + +Added min and max functions for Compressed Big Decimal and exposed these functions via SQL: BIG_MIN and BIG_MAX. + +https://github.com/apache/druid/pull/13141 + + +### Extension optimization + +Optimized the `compareTo` function in `CompressedBigDecimal`. + +https://github.com/apache/druid/pull/13086 + +### CompressedBigDecimal cleanup and extension + +Removed unnecessary generic type from CompressedBigDecimal, added support for number input types, added support for reading aggregator input types directly (uningested data), and fixed scaling bug in buffer aggregator. + +https://github.com/apache/druid/pull/13048 + + +### Support for Kubernetes discovery + +Added `POD_NAME` and `POD_NAMESPACE` env variables to all Kubernetes Deployments and StatefulSets. +Helm deployment is now compatible with `druid-kubernetes-extension`. + +https://github.com/apache/druid/pull/13262 + +## Dependency updates + +### Updated Kafka version + +Updated the Apache Kafka core dependency to version 3.3.1. + +https://github.com/apache/druid/pull/13176 + +### Docker improvements + +Updated dependencies for the Druid image for Docker, including JRE 11. Docker BuildKit cache is enabled to speed up building. -Multi-stage queries no longer show up in the Query history dialog. They are still available in the **Recent query tasks** panel. \ No newline at end of file +https://github.com/apache/druid/pull/13059 \ No newline at end of file From ad845238ee1fe45baa0afe901da015bc9c3a8a6f Mon Sep 17 00:00:00 2001 From: "brian.le" Date: Wed, 7 Dec 2022 12:12:28 -0800 Subject: [PATCH 10/17] add latest batch from milestone 48 and earlier --- docs/next-release-notes.md | 57 +++++++++++++++++++++++++++++++++++++- 1 file changed, 56 insertions(+), 1 deletion(-) diff --git a/docs/next-release-notes.md b/docs/next-release-notes.md index 835bb6bc2369..b9c302529b15 100644 --- a/docs/next-release-notes.md +++ b/docs/next-release-notes.md @@ -92,6 +92,12 @@ https://github.com/apache/druid/pull/13092 ## Multi-stage query task engine +### Worker retry + +The MSQ controller task now retries worker tasks if they fail up to two times. The maximum number of total retries for a job across all workers is 30. + +https://github.com/apache/druid/pull/13353 + ### CLUSTERED BY limit When using the MSQ task engine to ingest data, there is now a 1,500 column limit to the number of columns that can be passed in the CLUSTERED BY clause. @@ -197,12 +203,31 @@ Improved global-cached-lookups metric reporting. https://github.com/apache/druid/pull/13219 -### New metric for segments +### New metric for segment handoff `segment/handoff/time` captures the total time taken for handoff for a given set of published segments. https://github.com/apache/druid/pull/13238 +### New metrics for segment allocation + +Segment allocation can now be performed in batches, which can improve performance and decrease ingestion lag. For more information about this change, see [Segment batch allocation](#segment-batch-allocation). The following metrics correspond to those changes. + +Metrics for batch segment allocation (dims: dataSource, taskActionType=segmentAllocate): + +- `task/action/batch/runTime`: Milliseconds taken to execute a batch of task actions. Currently only being emitted for [batched `segmentAllocate` actions +- task/action/batch/queueTime: Milliseconds spent by a batch of task actions in queue. Currently only being emitted for [batched `segmentAllocate` actions](#segment-batch-allocation). +- `task/action/batch/size`: Number of task actions in a batch that was executed during the emission period. Currently only being emitted for [batched `segmentAllocate` actions](#segment-batch-allocation). +- `task/action/batch/attempts`: Number of execution attempts for a single batch of task actions. Currently only being emitted for [batched `segmentAllocate` actions](#segment-batch-allocation). + +Metrics for a request (dims: taskId, taskType, dataSource, taskActionType=segmentAllocate): + +- `task/action/failed/count`: Number of task actions that failed during the emission period. Currently only being emitted for [batched `segmentAllocate` actions](#segment-batch-allocation). +- `task/action/success/count`: Number of task actions that were executed successfully during the emission period. Currently only being emitted for [batched `segmentAllocate` actions](#segment-batch-allocation). + +https://github.com/apache/druid/pull/13369 +https://github.com/apache/druid/pull/13503 + ### New metrics for streaming ingestion The following metrics related to streaming ingestion have been added: @@ -380,6 +405,26 @@ https://github.com/apache/druid/pull/13356 ### Segment loading and balancing +#### Segment batch allocation + +Segment allocation on the Overlord can take some time to finish, which can cause ingestion lag while a task waits for segments to be allocated. Performing segment allocation in batches can help improve performance. + +There are two new properties that affect how Druid performs segment allocation: + +| Property | Description | Default | +| - | - | - | +|`druid.indexer.tasklock.batchSegmentAllocation`| If set to true, Druid performs segment allocate actions in batches to improve throughput and reduce the average `task/action/run/time`. See [batching `segmentAllocate` actions](https://druid.apache.org/docs/latest/ingestion/tasks.html#batching-segmentallocate-actions) for details.|false| +|`druid.indexer.tasklock.batchAllocationWaitTime`|Number of milliseconds after Druid adds the first segment allocate action to a batch, until it executes the batch. Allows the batch to add more requests and improve the average segment allocation run time. This configuration takes effect only if `batchSegmentAllocation` is enabled.|500| + +In addition to these properties, there are new metrics to track batch segment allocation. For more information, see [New metrics for segment allocation](#new-metrics-for-segment-allocation). + +For more information, see the following: +- [Overlord operations](https://druid.apache.org/docs/latest/configuration/index.html#overlord-operations) +- [Task actions and Batching `segmentAllocate` actions](https://druid.apache.org/docs/latest/ingestion/tasks.html#task-actions) + +https://github.com/apache/druid/pull/13369 +https://github.com/apache/druid/pull/13503 + #### cachingCost balancer strategy The `cachingCost` balancer strategy now behaves more similarly to cost strategy. When computing the cost of moving a segment to a server, the following calculations are performed: @@ -387,6 +432,8 @@ The `cachingCost` balancer strategy now behaves more similarly to cost strategy. - Subtract the self cost of a segment if it is being served by the target server - Subtract the cost of segments that are marked to be dropped +https://github.com/apache/druid/pull/13321 + #### Segment assignment You can now use a round-robin segment strategy to speed up initial segment assignments. @@ -472,6 +519,14 @@ https://github.com/apache/druid/pull/13287 ## Web console +### + +### Delete an interval + +You can now pick an interval to delete from a dropdown in the kill task dialog. + +https://github.com/apache/druid/pull/13431 + ### Removed the old query view The old query view is removed. Use the new query view with tabs. From d3c2f66626164c110b405703a32e783c97ccfb0d Mon Sep 17 00:00:00 2001 From: "brian.le" Date: Wed, 7 Dec 2022 13:26:31 -0800 Subject: [PATCH 11/17] delete empty heading --- docs/next-release-notes.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/next-release-notes.md b/docs/next-release-notes.md index b9c302529b15..8485eba06bd8 100644 --- a/docs/next-release-notes.md +++ b/docs/next-release-notes.md @@ -519,8 +519,6 @@ https://github.com/apache/druid/pull/13287 ## Web console -### - ### Delete an interval You can now pick an interval to delete from a dropdown in the kill task dialog. From 0a305a36a1d8fd6a84b914ef900138d710fdcaac Mon Sep 17 00:00:00 2001 From: "brian.le" Date: Wed, 7 Dec 2022 15:36:00 -0800 Subject: [PATCH 12/17] clean up from moving things around --- docs/next-release-notes.md | 40 ++++++++++++++++++-------------------- 1 file changed, 19 insertions(+), 21 deletions(-) diff --git a/docs/next-release-notes.md b/docs/next-release-notes.md index 8485eba06bd8..60ed7ccc735c 100644 --- a/docs/next-release-notes.md +++ b/docs/next-release-notes.md @@ -52,6 +52,12 @@ https://github.com/apache/druid/pull/13156 ## Behavior changes +### Memory estimates + +The task context flag `useMaxMemoryEstimates` is now set to false by default to improve memory usage estimation. + +https://github.com/apache/druid/pull/13178 + ### HLL and quantiles sketches The aggregation functions for HLL and quantiles sketches returned sketches or numbers when they are finalized depending on where they were in the native query plan. @@ -190,9 +196,14 @@ https://github.com/apache/druid/pull/13133 ### Added new configuration keys to query context security model -Added the following configuration keys that refine the query context security model controlled by `druid.auth.authorizeQueryContextParams`: -* `druid.auth.unsecuredContextKeys`: The set of query context keys that do not require a security check. -* `druid.auth.securedContextKeys`: The set of query context keys that do require a security check. +Added the following configuration properties that refine the query context security model controlled by `druid.auth.authorizeQueryContextParams`: + +* `druid.auth.unsecuredContextKeys`: A JSON list of query context keys that do not require a security check. +* `druid.auth.securedContextKeys`: A JSON list of query context keys that do require a security check. + +If both are set, `unsecuredContextKeys` acts as exceptions to `securedContextKeys`.. + +https://github.com/apache/druid/pull/13071 ## Metrics @@ -252,6 +263,8 @@ https://github.com/apache/druid/pull/13333 Improved `NestedDataColumnSerializer` to no longer explicitly write null values to the field writers for the missing values of every row. Instead, passing the row counter is moved to the field writers so that they can backfill null values in bulk. +https://github.com/apache/druid/pull/13101 + ### Support for more formats Druid nested columns and the associated JSON transform functions now support Avro, ORC, and Parquet. @@ -260,7 +273,7 @@ https://github.com/apache/druid/pull/13325 https://github.com/apache/druid/pull/13375 -### Refactored a data source before unnest +### Refactored a datasource before unnest When data requires "flattening" during processing, the operator now takes in an array and then flattens the array into N (N=number of elements in the array) rows where each row has one of the values from the array. @@ -300,7 +313,7 @@ https://github.com/apache/druid/pull/13144 ### Kafka Consumer improvement -Allowed Kafka Consumer's custom deserializer to be configured after its instantiation. +You can now configure the Kafka Consumer's custom deserializer after its instantiation. https://github.com/apache/druid/pull/13097 @@ -380,8 +393,6 @@ To configure the idle behavior, use the following properties: https://github.com/apache/druid/pull/13311 -https://github.com/apache/druid/pull/13321 - ### Improved supervisor termination Fixed issues with delayed supervisor termination during certain transient states. @@ -458,25 +469,12 @@ Use only `druid.coordinator.loadqueuepeon.http.repeatDelay` to configure repeat https://github.com/apache/druid/pull/13391 - - #### Segment replication Improved the process of checking server inventory to prevent over-replication of segments during segment balancing. https://github.com/apache/druid/pull/13114 -### Memory estimates - -The task context flag `useMaxMemoryEstimates` is now set to false by default to improve memory usage estimation. - -https://github.com/apache/druid/pull/13178 - - - - -https://github.com/apache/druid/pull/13101 - ### Provided service specific log4j overrides in containerized deployments Provided an option to override log4j configs setup at the service level directories so that it works with Druid-operator based deployments. @@ -512,7 +510,7 @@ https://github.com/apache/druid/pull/13207 ### Improved the run time of the MarkAsUnusedOvershadowedSegments duty -Improved the run time of the MarkAsUnusedOvershadowedSegments duty by iterating over all overshadowed segments and marking segments as unused in batches. +Improved the run time of the `MarkAsUnusedOvershadowedSegments` duty by iterating over all overshadowed segments and marking segments as unused in batches. https://github.com/apache/druid/pull/13287 From 126c33f91a716b2fd43a1d0162ddad63891bb5c0 Mon Sep 17 00:00:00 2001 From: 317brian <53799971+317brian@users.noreply.github.com> Date: Wed, 14 Dec 2022 08:28:34 -0800 Subject: [PATCH 13/17] Remove 13353 - it might not make it into 25 Co-authored-by: Karan Kumar --- docs/next-release-notes.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/docs/next-release-notes.md b/docs/next-release-notes.md index 60ed7ccc735c..95fb027df509 100644 --- a/docs/next-release-notes.md +++ b/docs/next-release-notes.md @@ -98,12 +98,6 @@ https://github.com/apache/druid/pull/13092 ## Multi-stage query task engine -### Worker retry - -The MSQ controller task now retries worker tasks if they fail up to two times. The maximum number of total retries for a job across all workers is 30. - -https://github.com/apache/druid/pull/13353 - ### CLUSTERED BY limit When using the MSQ task engine to ingest data, there is now a 1,500 column limit to the number of columns that can be passed in the CLUSTERED BY clause. From 740d2d85f23de1a7cf6b7153a4b30ba6b540e456 Mon Sep 17 00:00:00 2001 From: "brian.le" Date: Thu, 15 Dec 2022 14:28:58 -0800 Subject: [PATCH 14/17] add start druid script to operations section --- docs/next-release-notes.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/docs/next-release-notes.md b/docs/next-release-notes.md index 95fb027df509..9a824ab563ff 100644 --- a/docs/next-release-notes.md +++ b/docs/next-release-notes.md @@ -362,6 +362,10 @@ https://github.com/apache/druid/pull/12177 ## Operations +### Deploying Druid + +There is now a Python installation script available for Druid that simplifies deployments that don't fit the parameters of the singel-server profiles, such as the nano or micro-quickstart profiles. This `start-druid` script lets you override Druid settings to fit your needs. For more information, see [Single server deployment](https://druid.apache.org/docs/latest/operations/single-server.html). + ### Compaction Compaction behavior has changed to improve the amount of time it takes and disk space it takes: From c587b435f73f31b455f9d271b3a0c398f6f78e52 Mon Sep 17 00:00:00 2001 From: Victoria Lim Date: Fri, 16 Dec 2022 14:50:35 -0800 Subject: [PATCH 15/17] add RN for Jupyter tutorials --- docs/next-release-notes.md | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/docs/next-release-notes.md b/docs/next-release-notes.md index 9a824ab563ff..d3d2384d6ce6 100644 --- a/docs/next-release-notes.md +++ b/docs/next-release-notes.md @@ -601,6 +601,16 @@ Helm deployment is now compatible with `druid-kubernetes-extension`. https://github.com/apache/druid/pull/13262 +## Docs + +### Jupyter Notebook tutorials + +We released our first Jupyter Notebook-based tutorial to learn the basics of the Druid API. Download the notebook and follow along with the tutorial to learn how to get basic cluster information, ingest data, and query data. +For more information, see [Jupyter Notebook tutorials](https://druid.apache.org/docs/latest/tutorials/tutorial-jupyter-index.html). + +https://github.com/apache/druid/pull/13342 +https://github.com/apache/druid/pull/13345 + ## Dependency updates ### Updated Kafka version @@ -613,4 +623,4 @@ https://github.com/apache/druid/pull/13176 Updated dependencies for the Druid image for Docker, including JRE 11. Docker BuildKit cache is enabled to speed up building. -https://github.com/apache/druid/pull/13059 \ No newline at end of file +https://github.com/apache/druid/pull/13059 From dec5494e9fad435a240f977d7a151df98bc797c9 Mon Sep 17 00:00:00 2001 From: Victoria Lim Date: Fri, 16 Dec 2022 15:03:29 -0800 Subject: [PATCH 16/17] Apply suggestions from code review Co-authored-by: Kashif Faraz --- docs/next-release-notes.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/next-release-notes.md b/docs/next-release-notes.md index d3d2384d6ce6..0535da2006a6 100644 --- a/docs/next-release-notes.md +++ b/docs/next-release-notes.md @@ -218,14 +218,14 @@ https://github.com/apache/druid/pull/13238 Segment allocation can now be performed in batches, which can improve performance and decrease ingestion lag. For more information about this change, see [Segment batch allocation](#segment-batch-allocation). The following metrics correspond to those changes. -Metrics for batch segment allocation (dims: dataSource, taskActionType=segmentAllocate): +Metrics emitted for a batch segment allocate request (dims: dataSource, taskActionType=segmentAllocate): - `task/action/batch/runTime`: Milliseconds taken to execute a batch of task actions. Currently only being emitted for [batched `segmentAllocate` actions -- task/action/batch/queueTime: Milliseconds spent by a batch of task actions in queue. Currently only being emitted for [batched `segmentAllocate` actions](#segment-batch-allocation). +- `task/action/batch/queueTime`: Milliseconds spent by a batch of task actions in queue. Currently only being emitted for [batched `segmentAllocate` actions](#segment-batch-allocation). - `task/action/batch/size`: Number of task actions in a batch that was executed during the emission period. Currently only being emitted for [batched `segmentAllocate` actions](#segment-batch-allocation). - `task/action/batch/attempts`: Number of execution attempts for a single batch of task actions. Currently only being emitted for [batched `segmentAllocate` actions](#segment-batch-allocation). -Metrics for a request (dims: taskId, taskType, dataSource, taskActionType=segmentAllocate): +Metrics emitted for a single segment allocate request (dims: taskId, taskType, dataSource, taskActionType=segmentAllocate): - `task/action/failed/count`: Number of task actions that failed during the emission period. Currently only being emitted for [batched `segmentAllocate` actions](#segment-batch-allocation). - `task/action/success/count`: Number of task actions that were executed successfully during the emission period. Currently only being emitted for [batched `segmentAllocate` actions](#segment-batch-allocation). From 139dd61833fe8ddd99370ceb19348b5131ac767f Mon Sep 17 00:00:00 2001 From: Victoria Lim Date: Fri, 16 Dec 2022 15:11:44 -0800 Subject: [PATCH 17/17] Comments from Kashif --- docs/next-release-notes.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/docs/next-release-notes.md b/docs/next-release-notes.md index 0535da2006a6..a7dddb3a3203 100644 --- a/docs/next-release-notes.md +++ b/docs/next-release-notes.md @@ -30,6 +30,12 @@ The multi-stage query (MSQ) task engine used for SQL-based ingestion is now prod - [Ingestion](https://druid.apache.org/docs/latest/ingestion/index.html) - [SQL-based ingestion](https://druid.apache.org/docs/latest/multi-stage-query/index.html) +### Deploying Druid + +The new `start-druid` script greatly simplifies deploying any combination of Druid services on a single-server. It comes pre-packaged with the required configs and can be used to launch a fully functional Druid cluster simply by invoking `./start-druid`. For the experienced Druids, it also gives you complete control over the runtime properties and JVM arguments to have a cluster that exactly fits your needs. + +The `start-druid` script deprecates the existing profiles such as `start-micro-quickstart` and `start-nano-quickstart`. These profiles may be removed in future releases. For more information, see [Single server deployment](https://druid.apache.org/docs/latest/operations/single-server.html). + ### String dictionary compression (experimental) > Any segment written using string dictionary compression is not readable by older versions of Druid. @@ -93,6 +99,7 @@ This update changes the defaults for the following properties: | `druid.indexer.runner.type` for the Overlord | httpRemote | local | To use ZooKeeper instead of HTTP, change the values for the properties back to the previous defaults. +ZooKeeper-based implementations for these properties are deprecated and will be removed in a subsequent release. https://github.com/apache/druid/pull/13092 @@ -362,10 +369,6 @@ https://github.com/apache/druid/pull/12177 ## Operations -### Deploying Druid - -There is now a Python installation script available for Druid that simplifies deployments that don't fit the parameters of the singel-server profiles, such as the nano or micro-quickstart profiles. This `start-druid` script lets you override Druid settings to fit your needs. For more information, see [Single server deployment](https://druid.apache.org/docs/latest/operations/single-server.html). - ### Compaction Compaction behavior has changed to improve the amount of time it takes and disk space it takes: