remove select query by clintropolis · Pull Request #8739 · apache/druid

clintropolis · 2019-10-25T05:39:58Z

Description

This PR removes the Select query type, in favor of the Scan query. Since the introduction of time ordered scan query with #7133, our documentation has already been recommending people use scan queries instead of select due to the heavy memory impact of the select query, and Druid SQL has not generated select queries at all since #7373. Functionally, scan is equivalent, less the paging functionality, however the more efficient nature of the scan query makes the paging functionality un-necessary.

Additionally, I feel that removing the choice between 2 very similar query types should be more user friendly, and for code hygiene and lower maintenance for the query processing layer on our end.

This is obviously incompatible with previous releases so will need to be called out prominently in the release notes should we merge this PR.

This PR has:

been self-reviewed.
added documentation for new or modified features or behaviors.

vogievetsky · 2019-10-25T06:31:52Z

Very supportive of this in general!
I can not tell if you already do this from the diff but I think not: would it make sense to have a useful error message if someone does try to issue a select query explaining that it has been removed?

fjy · 2019-10-27T16:44:15Z

Closing/opening to restart teamcity

…eath-destroyer-of-select

clintropolis · 2019-10-28T01:06:21Z

Not sure why it is failing teamcity consistently with the same errors in files I didn't modify, merged latest master in to see if it changes things.

clintropolis · 2019-10-28T01:55:48Z

I can not tell if you already do this from the diff but I think not: would it make sense to have a useful error message if someone does try to issue a select query explaining that it has been removed?

I did not add any special handling for this, so it will show the default error:

Could not resolve type id 'select' into a subtype of [simple type, class org.apache.druid.query.Query]: known type ids = [Query, dataSourceMetadata, groupBy, scan, search, segmentMetadata, timeBoundary, timeseries, topN] at [Source: HttpInputOverHTTP@60001e07[c=336,q=0,[0]=null,s=STREAM]; line: 1, column: 2]

I guess I could add a RemovedQuery class or something to that effect, with an implementation explodes into a more specific error about a query type previously existing but is obsolete and has been removed, and bind it to select and anything else we ever remove. However, that also means I have to add some code and tests, and that makes me sad since I was just trying to nuke a bunch of code in this PR... but will add it if we really think it is necessary.

clintropolis · 2019-10-28T07:14:12Z

Ah, the teamcity failures are actually legitimate. The two errors are methods that only the select query engine was using: a variant of DefaultDimensionSpec.toSpec and Cursor.advanceTo.

… docs

clintropolis · 2019-10-29T08:37:23Z

I can not tell if you already do this from the diff but I think not: would it make sense to have a useful error message if someone does try to issue a select query explaining that it has been removed?

Ok, added back a SelectQuery class that implements Query and only knows how to make RuntimeException and a test to make sure that it happens on deserialization. Trying to use a select query will result in an error message of this form:

com.fasterxml.jackson.databind.JsonMappingException: Instantiation of [simple type, class org.apache.druid.query.select.SelectQuery] value failed: The 'select' query has been removed, use 'scan' instead. See https://druid.apache.org/docs/latest/querying/select-query.html for more details.

All it cost was adding 235 lines of code 😞

gianm

+1 on the idea of removing the select query; it's been deprecated for a while, causes clusters to crash (ouch), and the scan query is good enough now to mostly replace it. Suggested some wording changes.

gianm · 2019-10-29T21:59:48Z


-> We encourage you to use the [Scan query](../querying/scan-query.md) type rather than Select whenever possible.
-> In situations involving larger numbers of segments, the Select query can have very high memory and performance overhead.
+> The native Druid Select query has been removed in favor of the [Scan query](../querying/scan-query.md), which should


The > make this look weird when rendered (a note/quote block). Also the sentences read strangely together. I'd suggest:

Older versions of Apache Druid (incubating) included a Select query type. Since Druid 0.17.0, it has been removed and replaced by the Scan query, which offers improved memory usage and performance. This solves issues that users had with Select queries causing Druid to run out of memory or slow down.

The Scan query has a different syntax, but supports many of the features of the Select query, including time ordering and limiting. Scan does not include the Select query's pagination feature; however, in many cases pagination is unnecessary with Scan due to its ability to return a virtually unlimited number of results in one call.

Updated with your suggestion 👍

gianm · 2019-10-29T22:00:17Z

 |`druid.broker.cache.useResultLevelCache`|true, false|Enable result level caching on the Broker.|false|
 |`druid.broker.cache.populateResultLevelCache`|true, false|Populate the result level cache on the Broker.|false|
 |`druid.broker.cache.resultLevelCacheLimit`|positive integer|Maximum size of query response that can be cached.|`Integer.MAX_VALUE`|
-|`druid.broker.cache.unCacheable`|All druid query types|All query types to not cache.|`["groupBy", "select"]`|


…eath-destroyer-of-select

gianm

LGTM. Leaving open for the next committer to merge, as this needs one more design review.

When writing the release notes, make sure to mention that users of Druid SQL that do select queries must upgrade first to 0.15.0 or 0.16.0, then to 0.17.0, because 0.15.0 is the version that switched SQL selects to Scan. Upgrading directly from 0.14 to 0.17 will cause SQL selects to not work right during the rolling upgrade.

jihoonson

@clintropolis nice work! I like removing stuffs.

I can not tell if you already do this from the diff but I think not: would it make sense to have a useful error message if someone does try to issue a select query explaining that it has been removed?

I'm wondering whether we really need this. Should we remove SelectQuery at some point anyway because we cannot keep all legacy classes forever? A nice error message is always good for Druid users, but I guess this may not be needed since we will call out in the release notes and keep the doc for removed select query.

jihoonson · 2019-10-30T18:53:09Z

- */
-@JsonTypeName("select")
-public class SelectQuery extends BaseQuery<Result<SelectResultValue>>
+public class SelectQuery implements Query<Object>


Please add @Deprecated.

clintropolis · 2019-10-30T19:56:09Z

I'm wondering whether we really need this. Should we remove SelectQuery at some point anyway because we cannot keep all legacy classes forever? A nice error message is always good for Druid users, but I guess this may not be needed since we will call out in the release notes and keep the doc for removed select query.

I am in this camp as well, originally the PR just removed it all, but I guess this is a nicer user experience. I do think we could remove this placeholder in a release or two after the next one.

gianm · 2019-10-30T20:01:22Z

I'm wondering whether we really need this. Should we remove SelectQuery at some point anyway because we cannot keep all legacy classes forever? A nice error message is always good for Druid users, but I guess this may not be needed since we will call out in the release notes and keep the doc for removed select query.

I would suggest keeping the error message for at least a few releases. The Select query, even though flawed, was somewhat popular and I think this message will reduce confusion in the user base.

jihoonson

+1 after CI.

remove select query

9615968

clintropolis added Area - Querying Incompatible Release Notes Design Review labels Oct 25, 2019

fjy closed this Oct 27, 2019

fjy reopened this Oct 27, 2019

Merge remote-tracking branch 'upstream/master' into now-i-am-become-d…

cbe9640

…eath-destroyer-of-select

clintropolis added 5 commits October 28, 2019 00:16

thanks teamcity

48e359d

oops

1709621

oops

839b162

add back a SelectQuery class that throws RuntimeExceptions linking to…

451414d

… docs

adjust text

8a09b31

gianm reviewed Oct 29, 2019

View reviewed changes

clintropolis added 2 commits October 29, 2019 22:51

update docs per review

19c9ec8

Merge remote-tracking branch 'upstream/master' into now-i-am-become-d…

ec5338f

…eath-destroyer-of-select

gianm approved these changes Oct 30, 2019

View reviewed changes

jihoonson reviewed Oct 30, 2019

View reviewed changes

deprecated

cefcd47

jihoonson approved these changes Oct 30, 2019

View reviewed changes

gianm merged commit 3ff5e02 into apache:master Oct 31, 2019

gianm added this to the 0.17.0 milestone Oct 31, 2019

clintropolis deleted the now-i-am-become-death-destroyer-of-select branch October 31, 2019 02:42

jon-wei mentioned this pull request Dec 28, 2019

0.17.0 release notes #9066

Closed

Conversation

clintropolis commented Oct 25, 2019

Description

Uh oh!

vogievetsky commented Oct 25, 2019

Uh oh!

fjy commented Oct 27, 2019

Uh oh!

clintropolis commented Oct 28, 2019

Uh oh!

clintropolis commented Oct 28, 2019

Uh oh!

clintropolis commented Oct 28, 2019

Uh oh!

clintropolis commented Oct 29, 2019

Uh oh!

gianm left a comment

Choose a reason for hiding this comment

Uh oh!

gianm Oct 29, 2019

Choose a reason for hiding this comment

Uh oh!

clintropolis Oct 30, 2019

Choose a reason for hiding this comment

Uh oh!

gianm Oct 29, 2019

Choose a reason for hiding this comment

Uh oh!

gianm left a comment

Choose a reason for hiding this comment

Uh oh!

jihoonson left a comment

Choose a reason for hiding this comment

Uh oh!

jihoonson Oct 30, 2019

Choose a reason for hiding this comment

Uh oh!

clintropolis Oct 30, 2019

Choose a reason for hiding this comment

Uh oh!

clintropolis commented Oct 30, 2019

Uh oh!

gianm commented Oct 30, 2019

Uh oh!

jihoonson left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants