25.0 release notes: batch 3 by 317brian · Pull Request #13429 · apache/druid

317brian · 2022-11-24T01:49:18Z

Replaces #13357

This takes us up to November 23 (after the 25.0.0 branch was created)

2bethere

Made some comments, mostly about shuffling content around. Thanks for pulling this together.

I think the new format with links are great!

2bethere · 2022-12-03T04:16:25Z

+  ~ under the License.
+  -->
+
+## Query engine


I think we should have a highlight section here. The top 3 features I think we should highlight are:

MSQ production ready

String dictionary compression (Experimental)

K8S native tasks (i.e. Run tasks on K8S without middle managers)

2bethere · 2022-12-03T04:16:41Z

+
+## Query engine
+
+### BIG_SUM SQL function


This should be moved to the BIG_SUM SQL extension section.

2bethere · 2022-12-03T04:17:08Z

+
+https://github.com/apache/druid/pull/13102
+
+### Added Compressed Big Decimal min and max functions


Same as this one. I think moving this along with the extension makes it clear they are all part of the same functionality.

2bethere · 2022-12-03T04:17:39Z

+
+### Metrics used to downsample bucket
+
+Changed the way the MSQ task engine determines whether or not to downsample data, to improve accuracy. The task engine now uses the number of bytes instead of number of keys.


I suggest create a MSQ section to group all MSQ things together so that it's easier for people to skip they want to see other stuff.

2bethere · 2022-12-03T04:21:05Z

+This release includes several improvements for querying:
+
+* Exposed HTTP response headers for SQL queries (https://github.com/apache/druid/pull/13052)
+* Added the `shouldFinalize` feature for HLL and quantiles sketches. Druid will no longer finalize aggregators when:


I think we need to update the query docs on sketches to describe this behavior and why we made this change.

For example, "Druid will not finalize if aggregator is on the outer level because ......."

@vtlim will work on the doc updates

2bethere · 2022-12-03T05:39:58Z

+
+
+
+### Kill tasks do not include markAsUnuseddone


This is also a behavior change.

2bethere · 2022-12-03T05:42:21Z

+
+https://github.com/apache/druid/pull/13059
+
+### Improved supervisor termination


This should be moved together with other supervisor idle stuff.

2bethere · 2022-12-03T05:42:41Z

+
+https://github.com/apache/druid/pull/13064
+
+### Improved metric reporting


This should go into metrics section.

2bethere · 2022-12-03T05:43:54Z

+
+https://github.com/apache/druid/pull/13262
+
+## Web console


Web console should be moved above extensions

2bethere · 2022-12-03T05:44:12Z

+
+https://github.com/apache/druid/pull/13291
+
+### Query history


This should move to MSQ section.

kfaraz · 2022-12-05T09:01:11Z

+The unused coordinator property `druid.coordinator.loadqueuepeon.repeatDelay` has been removed.
+
+Use only `druid.coordinator.loadqueuepeon.http.repeatDelay` to configure repeat delay for the HTTP-based segment loading queue.


These two items should either be under the heading Segment assignment or a separate heading titled Segment load queue peon.

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

317brian · 2022-12-06T22:04:53Z

@2bethere the latest commit addresses the comments from your review.

@kfaraz there's 1 more commit planned to capture the PRs from the last milestone. After that, we should be done once it's reviewed.

317brian · 2022-12-07T20:13:42Z

ad84523 (#13429) contains what should be the last batch from git log and milestone 48.

The only one I haven't added is #13365 from the milestone since the original PR is still open.

Co-authored-by: Karan Kumar <karankumar1100@gmail.com>

317brian · 2022-12-15T22:29:57Z

@kfaraz I've added the start-druid script entry to the release notes, so we should be all caught up.

kfaraz · 2022-12-16T14:16:15Z

+| `druid.coordinator.loadqueuepeon.type` for segment management | http | curator |
+| `druid.indexer.runner.type` for the Overlord | httpRemote | local |
+
+To use ZooKeeper instead of HTTP, change the values for the properties back to the previous defaults.


We should also mention that the Zookeeper based implementations for the above items are now deprecated and will be removed in subsequent releases.

kfaraz · 2022-12-16T14:22:11Z


 ## Operations

+### Deploying Druid


This item should be in the highlights.

kfaraz · 2022-12-16T14:35:55Z


+### Deploying Druid
+
+There is now a Python installation script available for Druid that simplifies deployments that don't fit the parameters of the singel-server profiles, such as the nano or micro-quickstart profiles. This `start-druid` script lets you override Druid settings to fit your needs. For more information, see [Single server deployment](https://druid.apache.org/docs/latest/operations/single-server.html).


The new script simplifies deployments for all cases, not just the ones that don't fit the existing single-server profiles. We could probably rephrase this section as follows:

Suggested change

There is now a Python installation script available for Druid that simplifies deployments that don't fit the parameters of the singel-server profiles, such as the nano or micro-quickstart profiles. This `start-druid` script lets you override Druid settings to fit your needs. For more information, see [Single server deployment](https://druid.apache.org/docs/latest/operations/single-server.html).

The new `start-druid` script greatly simplifies deploying any combination of Druid services on a single-server. It comes pre-packaged with the required configs and can be used to launch a fully functional Druid cluster simply by invoking `./start-druid`. For the experienced Druids, it also gives you complete control over the runtime properties and JVM arguments to have a cluster that exactly fits your needs.

The `start-druid` script deprecates the existing profiles such as `start-micro-quickstart` and `start-nano-quickstart`. These profiles may be removed in future releases. For more information, see [Single server deployment](https://druid.apache.org/docs/latest/operations/single-server.html).

@vogievetsky , does this capture our sentiments adequately?

I moved this section to the highlights and also used the text from this comment.

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

kfaraz · 2023-01-14T05:10:00Z

Closing this as Druid 25.0 is released.

writer-jill and others added 6 commits November 10, 2022 16:03

25.0 release notes: batch 1

2270725

Update next-release-notes.md

3f7094c

Update next-release-notes.md

01b6e0a

25.0 release notes: batch 2

a5a7749

Add link to related changes

4d578be

next set of release notes after 25.0 branch creation

6b2d3de

kfaraz reviewed Nov 24, 2022

View reviewed changes

Comment thread docs/next-release-notes.md Outdated

kfaraz reviewed Nov 24, 2022

View reviewed changes

Comment thread docs/next-release-notes.md Outdated

kfaraz reviewed Nov 24, 2022

View reviewed changes

Comment thread docs/next-release-notes.md Outdated

address feedback from kfaraz

b91e418

2bethere reviewed Dec 3, 2022

View reviewed changes

kfaraz reviewed Dec 5, 2022

View reviewed changes

Comment thread docs/next-release-notes.md Outdated

kfaraz reviewed Dec 5, 2022

View reviewed changes

Comment thread docs/next-release-notes.md Outdated

kfaraz reviewed Dec 5, 2022

View reviewed changes

Comment thread docs/next-release-notes.md Outdated

317brian and others added 2 commits December 5, 2022 14:18

Apply suggestions from code review

5e5c6a8

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

address feedback from kashif and will

5f18d16

add latest batch from milestone 48 and earlier

ad84523

317brian requested review from 2bethere and kfaraz and removed request for 2bethere and kfaraz December 7, 2022 20:13

317brian added 2 commits December 7, 2022 13:26

delete empty heading

d3c2f66

clean up from moving things around

0a305a3

cryptoe reviewed Dec 13, 2022

View reviewed changes

Comment thread docs/next-release-notes.md Outdated

317brian and others added 2 commits December 14, 2022 08:28

Remove 13353 - it might not make it into 25

126c33f

Co-authored-by: Karan Kumar <karankumar1100@gmail.com>

add start druid script to operations section

740d2d8

kfaraz reviewed Dec 16, 2022

View reviewed changes

Comment thread docs/next-release-notes.md Outdated

kfaraz reviewed Dec 16, 2022

View reviewed changes

Comment thread docs/next-release-notes.md Outdated

kfaraz reviewed Dec 16, 2022

View reviewed changes

Comment thread docs/next-release-notes.md Outdated

kfaraz reviewed Dec 16, 2022

View reviewed changes

vtlim and others added 3 commits December 16, 2022 14:50

add RN for Jupyter tutorials

c587b43

Apply suggestions from code review

dec5494

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

Comments from Kashif

139dd61

kfaraz closed this Jan 14, 2023


		https://github.com/apache/druid/pull/13102

		### Added Compressed Big Decimal min and max functions


		### Metrics used to downsample bucket

		Changed the way the MSQ task engine determines whether or not to downsample data, to improve accuracy. The task engine now uses the number of bytes instead of number of keys.


		https://github.com/apache/druid/pull/13059

		### Improved supervisor termination


		https://github.com/apache/druid/pull/13064

		### Improved metric reporting


		https://github.com/apache/druid/pull/13262

		## Web console


		https://github.com/apache/druid/pull/13291

		### Query history

		The unused coordinator property `druid.coordinator.loadqueuepeon.repeatDelay` has been removed.

		Use only `druid.coordinator.loadqueuepeon.http.repeatDelay` to configure repeat delay for the HTTP-based segment loading queue.


		### Deploying Druid

		There is now a Python installation script available for Druid that simplifies deployments that don't fit the parameters of the singel-server profiles, such as the nano or micro-quickstart profiles. This `start-druid` script lets you override Druid settings to fit your needs. For more information, see [Single server deployment](https://druid.apache.org/docs/latest/operations/single-server.html).

-There is now a Python installation script available for Druid that simplifies deployments that don't fit the parameters of the singel-server profiles, such as the nano or micro-quickstart profiles. This `start-druid` script lets you override Druid settings to fit your needs. For more information, see [Single server deployment](https://druid.apache.org/docs/latest/operations/single-server.html).
+The new `start-druid` script greatly simplifies deploying any combination of Druid services on a single-server. It comes pre-packaged with the required configs and can be used to launch a fully functional Druid cluster simply by invoking `./start-druid`. For the experienced Druids, it also gives you complete control over the runtime properties and JVM arguments to have a cluster that exactly fits your needs.
+The `start-druid` script deprecates the existing profiles such as `start-micro-quickstart` and `start-nano-quickstart`. These profiles may be removed in future releases. For more information, see [Single server deployment](https://druid.apache.org/docs/latest/operations/single-server.html).

Conversation

317brian commented Nov 24, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

2bethere left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

317brian commented Dec 6, 2022

Uh oh!

317brian commented Dec 7, 2022

Uh oh!

Uh oh!

317brian commented Dec 15, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kfaraz Dec 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kfaraz commented Jan 14, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

kfaraz Dec 16, 2022 •

edited

Loading