DO NOT MERGE - 27.0.0 WIP release notes by 317brian · Pull Request #14600 · apache/druid

317brian · 2023-07-17T19:37:51Z

Draft PR used to collaborate on the release notes so that we can use the comment/suggestion system.

abhishekrb19 · 2023-07-18T05:12:50Z

+
+[14281](https://github.com/apache/druid/pull/14281)
+
+## Dependency updates


Similar to the 26.0 release, should we include a "Bug fixes" section with a pointer to the list:
https://github.com/apache/druid/issues?q=is%3Aclosed+milestone%3A27.0+-label%3ABackport+label%3ABug+

For the 27.0.0 release, though, the bugs list looks relatively small - from a cursory scan of the merged PRs in 27.0, it seems like some of them weren't tagged "Bug".

Yeah, several of them might have been bug fixes but we might have just forgotten to label them appropriately.

gianm · 2023-07-20T02:17:03Z

+
+[13197](https://github.com/apache/druid/pull/13197)
+
+## Hadoop 2 support dropped


It is not being dropped in this release, but we did decide to deprecate it.

Co-authored-by: Abhishek Radhakrishnan <abhishek.rb19@gmail.com>

317brian · 2023-07-24T20:38:37Z

+| `segment/loadQueue/success` |Number of segment assignments that completed successfully.|`dataSource`, `server`|Varies|
+| `segment/loadQueue/cancelled` |Number of segment assignments that were canceled before completion. |`dataSource`,`server` | 0 |
+| `segment/loadQueue/failed` |Number of segment assignments that failed to complete.|`dataSource`, `server`|0|
+| `segment/assigned/broadcast` | |


@kfaraz It looks like the 2 broadcast metrics from 13197 didn't get added to https://github.com/apache/druid/blob/master/docs/operations/metrics.md

Yes, @317brian , these metrics were originally a part of the PR but were later removed.

kfaraz

Added some items that were left out because I had forgotten to update/add release notes in respective PRs.

kfaraz · 2023-07-27T05:35:05Z

+
+[14281](https://github.com/apache/druid/pull/14281)
+
+## Dependency updates


Yeah, several of them might have been bug fixes but we might have just forgotten to label them appropriately.

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

kfaraz · 2023-07-28T05:27:41Z

 ## Smart segment loading

-Revamped Coordinator to make it more stable and user-friendly. This is accompanied by several bug fixes, logging and metric improvements, and a whole new range of capabilities.
+The Coordinator is now more stable and user-friendly. This is accompanied by several bug fixes, logging and metric improvements, and a whole new range of capabilities.


"The Coordinator" seems to have been repeated several times in this section, can we reduce it?

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

kfaraz · 2023-08-01T05:26:41Z

+
 ## Smart segment loading

+INSTEAD OF PRIORITIZING THINGS EQUALLY; WE PRIORITIZE NEW SEGMENTS THAT ARE UNDERREPLICATED


Suggested content for this section:

The Coordinator is now much more stable and user-friendly. In the new smartSegmentLoading mode, it dynamically computes values for several configs which maximize performance.

The Coordinator can now prioritize load of more recent segments and segments that are completely unavailable over load of segments that already have some replicas loaded in the cluster. It can also re-evaluate decisions taken in previous runs and cancel operations that are not needed anymore. Moreoever, move operations started by segment balancing do not compete with the load of unavailable segments thus reducing the reaction time for changes in the cluster and speeding up segment assignment decisions.

317brian · 2023-08-02T01:44:53Z

+* What to do about input source security. There are no docs for it.
+* New query filters 
+* Confirm the correct naming in https://github.com/apache/druid/pull/14359 (OSHI sys mon change). The changed files say one thing and the PR description says another
+* Community extensions. What do we want to say about iceberg and the other one (what was the other one?)


Also SQL compatability mdoe mention somewhere?

vogievetsky · 2023-08-03T06:22:28Z

+* Caching cost for upgrade notes - is it this? [#14484](https://github.com/apache/druid/pull/14484)
+* User impact of https://github.com/apache/druid/pull/14048 for highlights
+* https://github.com/apache/druid/pull/14476 seems to only be documented here in release notes and the PR?
+* Vad's input for web console section


How about:

Web console

Console now uses the new statements API for all MSQ interaction

The console uses the new async statements API for all sql-msq-task engine queries.
While this has relatively few impacts on the UX of the query view you are invited to peek under the hood and check out the new network requests being sent as a working example of the new API.

There are some fun UX benefits: you can now specify durableStorage as the result destination for SELECT queries (when durable storage is configured).

After running a SELECT query that wrote its resuls to durableStorage you can download the full, unlimited, result set directly from the broker.

#14540
#14669
#14712

Added UI around data source with zero replica segments

This release of Druid supports having datasources with segments that are not replicated on any historicals, the console now had special handling for datasources like that.

[https://github.com//pull/14540]

Added a Explore view

The Explore view is a simple, stateless, SQL backed, data exploration view to the web console. It lets users explore data in Druid with point-and-click interaction and visualizations (instead of writing SQL and looking at a table). This can provide faster time-to-value for a user new to Druid and can allow a Druid veteran to quickly chart some data that they care about.

The Explore view is accessible from the more button in the header.

#14602

Added a dialog for viewing and setting the dynamic compaction config

[https://github.com//pull/14540]

Other changes

Replaced the Ingestion view with two views: Supervisors and Tasks #14395

Added 'Start time' column to the Services tab [https://github.com/Add start_time column to sys.servers #13358]

Added 'Replication factor' column to the Segments view #14403

Added stateful filter URLs for all views [https://github.com/Web console: split the Ingestion view into two views: Supervisors and Tasks #14395]

abhishekagarwal87 · 2023-08-08T05:20:38Z

+|`segment/serverview/sync/healthy`|Sync status of the Coordinator/Broker with a segment-loading server such as a Historical or Peon. Emitted by the Coordinator and Brokers only when HTTP-based server view is enabled. This metric can be used in conjunction with `segment/serverview/sync/unstableTime` to debug slow startup of the Coordinator.|`server`, `tier`|1 for fully synced servers, 0 otherwise|
+|`segment/serverview/sync/unstableTime`|Time in milliseconds for which the Coordinator/Broker has been failing to sync with a segment-loading server. Emitted by the Coordinator and Brokers only when HTTP-based server view is enabled.|`server`, `tier`|Not emitted for synced servers.|



Suggested change

|`segment/serverview/sync/healthy`|Sync status of the Coordinator/Broker with a segment-loading server such as a Historical or Peon. Emitted by the Coordinator and Brokers only when HTTP-based server view is enabled. This metric can be used in conjunction with `segment/serverview/sync/unstableTime` to debug slow startup of the Coordinator.|`server`, `tier`|1 for fully synced servers, 0 otherwise|

|`segment/serverview/sync/unstableTime`|Time in milliseconds for which the Coordinator/Broker has been failing to sync with a segment-loading server. Emitted by the Coordinator and Brokers only when HTTP-based server view is enabled.|`server`, `tier`|Not emitted for synced servers.|

|`serverview/sync/healthy`|Sync status of the Coordinator/Broker with a segment-loading server such as a Historical or Peon. Emitted by the Coordinator and Brokers only when HTTP-based server view is enabled. This metric can be used in conjunction with `serverview/sync/unstableTime` to debug slow startup of the Coordinator.|`server`, `tier`|1 for fully synced servers, 0 otherwise|

|`serverview/sync/unstableTime`|Time in milliseconds for which the Coordinator/Broker has been failing to sync with a segment-loading server. Emitted by the Coordinator and Brokers only when HTTP-based server view is enabled.|`server`, `tier`|Not emitted for synced servers.|

Manually updated in 0959b28

317brian · 2023-08-08T22:21:19Z

I've cleaned up all my notes and included more information as discussed. @abhishekagarwal87, @AmatyaAvadhanula , and @2bethere can y'all double check the deprecation list to make sure I have everything.

I ran the spellchecker locally, and the errors I received back were expected. (The RAT test will continue to fail since we intentionally did not add a license snippet to this file)

AmatyaAvadhanula · 2023-08-09T08:41:40Z

 |------|-----------|-----------|----------|
-|`segment/serverview/sync/healthy`|Sync status of the Coordinator/Broker with a segment-loading server such as a Historical or Peon. Emitted by the Coordinator and Brokers only when HTTP-based server view is enabled. This metric can be used in conjunction with `segment/serverview/sync/unstableTime` to debug slow startup of the Coordinator.|`server`, `tier`|1 for fully synced servers, 0 otherwise|
-|`segment/serverview/sync/unstableTime`|Time in milliseconds for which the Coordinator/Broker has been failing to sync with a segment-loading server. Emitted by the Coordinator and Brokers only when HTTP-based server view is enabled.|`server`, `tier`|Not emitted for synced servers.|
+|`serverview/sync/healthy`|Sync status of the Coordinator/Broker with a segment-loading server such as a Historical or Peon. Emitted by the Coordinator and Brokers only when HTTP-based server view is enabled. This metric can be used in conjunction with `segment/serverview/sync/unstableTime` to debug slow startup of the Coordinator.|`server`, `tier`|1 for fully synced servers, 0 otherwise|


segment/serverview/sync/unstableTime -> serverview/sync/unstableTime

27.0.0 WIP release notes

5e26014

github-actions Bot added the Area - Documentation label Jul 17, 2023

clintropolis added this to the 27.0 milestone Jul 17, 2023

abhishekrb19 reviewed Jul 18, 2023

View reviewed changes

Comment thread docs/do-not-merge.md Outdated

abhishekrb19 reviewed Jul 18, 2023

View reviewed changes

Comment thread docs/do-not-merge.md Outdated

gianm reviewed Jul 20, 2023

View reviewed changes

317brian and others added 3 commits July 24, 2023 13:23

Apply suggestions from code review

3f641b7

Co-authored-by: Abhishek Radhakrishnan <abhishek.rb19@gmail.com>

partial commit for new segment metrics

7308e1d

partial commit for new segment metrics

6059520

317brian commented Jul 24, 2023

View reviewed changes

cleanup

8cf85d2

kfaraz requested changes Jul 27, 2023

View reviewed changes

317brian and others added 2 commits July 27, 2023 15:53

Apply suggestions from code review

1f89f35

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

copyedits

6bac9ef

kfaraz reviewed Jul 28, 2023

View reviewed changes

Comment thread docs/do-not-merge.md Outdated

317brian and others added 3 commits July 28, 2023 10:01

Apply suggestions from code review

efcce93

Co-authored-by: Kashif Faraz <kashif.faraz@gmail.com>

copyedit smart segment load

d737513

notes from review session

8d577f2

kfaraz reviewed Aug 1, 2023

View reviewed changes

update based on discussoin

b746f65

317brian commented Aug 2, 2023

View reviewed changes

317brian added 2 commits August 1, 2023 18:47

add link to release milestone

9096159

add 13023

2d08c1e

2bethere reviewed Aug 2, 2023

View reviewed changes

Comment thread docs/do-not-merge.md

add hadoop 2 info

7b69c7c

vogievetsky reviewed Aug 3, 2023

View reviewed changes

317brian added 2 commits August 3, 2023 12:57

udpate web cosole stuff and add ice berg

76373f4

updates

5073a7b

abhishekagarwal87 reviewed Aug 8, 2023

View reviewed changes

AmatyaAvadhanula self-requested a review August 8, 2023 07:02

317brian added 3 commits August 8, 2023 14:52

update metric names

0959b28

cleanup

7657a9b

cleanup

dfd1236

formatting

bc9d49f

AmatyaAvadhanula reviewed Aug 9, 2023

View reviewed changes

fix metric name

f9b5dff

317brian closed this Aug 11, 2023


		[14281](https://github.com/apache/druid/pull/14281)

		## Dependency updates


		[13197](https://github.com/apache/druid/pull/13197)

		## Hadoop 2 support dropped


		## Smart segment loading

		INSTEAD OF PRIORITIZING THINGS EQUALLY; WE PRIORITIZE NEW SEGMENTS THAT ARE UNDERREPLICATED

		\|`segment/serverview/sync/healthy`\|Sync status of the Coordinator/Broker with a segment-loading server such as a Historical or Peon. Emitted by the Coordinator and Brokers only when HTTP-based server view is enabled. This metric can be used in conjunction with `segment/serverview/sync/unstableTime` to debug slow startup of the Coordinator.\|`server`, `tier`\|1 for fully synced servers, 0 otherwise\|
		\|`segment/serverview/sync/unstableTime`\|Time in milliseconds for which the Coordinator/Broker has been failing to sync with a segment-loading server. Emitted by the Coordinator and Brokers only when HTTP-based server view is enabled.\|`server`, `tier`\|Not emitted for synced servers.\|

Conversation

317brian commented Jul 17, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kfaraz left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Web console

Console now uses the new statements API for all MSQ interaction

Added UI around data source with zero replica segments

Added a Explore view

Added a dialog for viewing and setting the dynamic compaction config

Other changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

317brian commented Aug 8, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

317brian commented Aug 8, 2023 •

edited

Loading