streaming version of select query #3307
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,157 @@ | ||
| --- | ||
| layout: doc_page | ||
| --- | ||
|
|
||
| # Scan query | ||
| Scan query returns raw Druid rows in streaming mode. | ||
|
|
||
| ```json | ||
| { | ||
| "queryType": "scan", | ||
| "dataSource": "wikipedia", | ||
| "resultFormat": "list", | ||
| "columns":[], | ||
| "intervals": [ | ||
| "2013-01-01/2013-01-02" | ||
| ], | ||
| "batchSize":20480, | ||
| "limit":5 | ||
| } | ||
| ``` | ||
|
|
||
| There are several main parts to a scan query: | ||
|
|
||
| |property|description|required?| | ||
| |--------|-----------|---------| | ||
| |queryType|This String should always be "scan"; this is the first thing Druid looks at to figure out how to interpret the query|yes| | ||
| |dataSource|A String or Object defining the data source to query, very similar to a table in a relational database. See [DataSource](../querying/datasource.html) for more information.|yes| | ||
| |intervals|A JSON Object representing ISO-8601 Intervals. This defines the time ranges to run the query over.|yes| | ||
| |resultFormat|How result represented, list or compactedList or valueVector. Currently only `list` and `compactedList` are supported. Default is `list`|no| | ||
| |filter|See [Filters](../querying/filters.html)|no| | ||
| |columns|A String array of dimensions and metrics to scan. If left empty, all dimensions and metrics are returned.|no| | ||
| |batchSize|How many rows buffered before return to client. Default is `20480`|no| | ||
| |limit|How many rows to return. If not specified, all rows will be returned.|no| | ||
| |context|An additional JSON Object which can be used to specify certain flags.|no| | ||
|
|
||
| The format of the result when resultFormat equals to `list`: | ||
|
|
||
| ```json | ||
| [{ | ||
| "segmentId" : "wikipedia_editstream_2012-12-29T00:00:00.000Z_2013-01-10T08:00:00.000Z_2013-01-10T08:13:47.830Z_v9", | ||
| "columns" : [ | ||
| "timestamp", | ||
| "robot", | ||
| "namespace", | ||
| "anonymous", | ||
| "unpatrolled", | ||
| "page", | ||
| "language", | ||
| "newpage", | ||
| "user", | ||
| "count", | ||
| "added", | ||
| "delta", | ||
| "variation", | ||
| "deleted" | ||
| ], | ||
| "events" : [ { | ||
| "timestamp" : "2013-01-01T00:00:00.000Z", | ||
| "robot" : "1", | ||
| "namespace" : "article", | ||
| "anonymous" : "0", | ||
| "unpatrolled" : "0", | ||
| "page" : "11._korpus_(NOVJ)", | ||
| "language" : "sl", | ||
| "newpage" : "0", | ||
| "user" : "EmausBot", | ||
| "count" : 1.0, | ||
| "added" : 39.0, | ||
| "delta" : 39.0, | ||
| "variation" : 39.0, | ||
| "deleted" : 0.0 | ||
| }, { | ||
| "timestamp" : "2013-01-01T00:00:00.000Z", | ||
| "robot" : "0", | ||
| "namespace" : "article", | ||
| "anonymous" : "0", | ||
| "unpatrolled" : "0", | ||
| "page" : "112_U.S._580", | ||
| "language" : "en", | ||
| "newpage" : "1", | ||
| "user" : "MZMcBride", | ||
| "count" : 1.0, | ||
| "added" : 70.0, | ||
| "delta" : 70.0, | ||
| "variation" : 70.0, | ||
| "deleted" : 0.0 | ||
| }, { | ||
| "timestamp" : "2013-01-01T00:00:00.000Z", | ||
| "robot" : "0", | ||
| "namespace" : "article", | ||
| "anonymous" : "0", | ||
| "unpatrolled" : "0", | ||
| "page" : "113_U.S._243", | ||
| "language" : "en", | ||
| "newpage" : "1", | ||
| "user" : "MZMcBride", | ||
| "count" : 1.0, | ||
| "added" : 77.0, | ||
| "delta" : 77.0, | ||
| "variation" : 77.0, | ||
| "deleted" : 0.0 | ||
| }, { | ||
| "timestamp" : "2013-01-01T00:00:00.000Z", | ||
| "robot" : "0", | ||
| "namespace" : "article", | ||
| "anonymous" : "0", | ||
| "unpatrolled" : "0", | ||
| "page" : "113_U.S._73", | ||
| "language" : "en", | ||
| "newpage" : "1", | ||
| "user" : "MZMcBride", | ||
| "count" : 1.0, | ||
| "added" : 70.0, | ||
| "delta" : 70.0, | ||
| "variation" : 70.0, | ||
| "deleted" : 0.0 | ||
| }, { | ||
| "timestamp" : "2013-01-01T00:00:00.000Z", | ||
| "robot" : "0", | ||
| "namespace" : "article", | ||
| "anonymous" : "0", | ||
| "unpatrolled" : "0", | ||
| "page" : "113_U.S._756", | ||
| "language" : "en", | ||
| "newpage" : "1", | ||
| "user" : "MZMcBride", | ||
| "count" : 1.0, | ||
| "added" : 68.0, | ||
| "delta" : 68.0, | ||
| "variation" : 68.0, | ||
| "deleted" : 0.0 | ||
| } ] | ||
| } ] | ||
| ``` | ||
|
|
||
| The format of the result when resultFormat equals to `compactedList`: | ||
|
|
||
| ```json | ||
| [{ | ||
| "segmentId" : "wikipedia_editstream_2012-12-29T00:00:00.000Z_2013-01-10T08:00:00.000Z_2013-01-10T08:13:47.830Z_v9", | ||
| "columns" : [ | ||
| "timestamp", "robot", "namespace", "anonymous", "unpatrolled", "page", "language", "newpage", "user", "count", "added", "delta", "variation", "deleted" | ||
| ], | ||
| "events" : [ | ||
| ["2013-01-01T00:00:00.000Z", "1", "article", "0", "0", "11._korpus_(NOVJ)", "sl", "0", "EmausBot", 1.0, 39.0, 39.0, 39.0, 0.0], | ||
| ["2013-01-01T00:00:00.000Z", "0", "article", "0", "0", "112_U.S._580", "en", "1", "MZMcBride", 1.0, 70.0, 70.0, 70.0, 0.0], | ||
| ["2013-01-01T00:00:00.000Z", "0", "article", "0", "0", "113_U.S._243", "en", "1", "MZMcBride", 1.0, 77.0, 77.0, 77.0, 0.0], | ||
| ["2013-01-01T00:00:00.000Z", "0", "article", "0", "0", "113_U.S._73", "en", "1", "MZMcBride", 1.0, 70.0, 70.0, 70.0, 0.0], | ||
| ["2013-01-01T00:00:00.000Z", "0", "article", "0", "0", "113_U.S._756", "en", "1", "MZMcBride", 1.0, 68.0, 68.0, 68.0, 0.0] | ||
| ] | ||
| } ] | ||
| ``` | ||
|
|
||
| The biggest difference between select query and scan query is that, scan query doesn't retain all rows in memory before rows can be returned to client. | ||
| It will cause memory pressure if too many rows required by select query. | ||
| Scan query doesn't have this issue. | ||
| Scan query can return all rows without issuing another pagination query, which is extremely useful when query against historical or realtime node directly. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,58 @@ | ||
| <?xml version="1.0" encoding="UTF-8"?> | ||
|
|
||
| <!-- | ||
| ~ Licensed to Metamarkets Group Inc. (Metamarkets) under one | ||
| ~ or more contributor license agreements. See the NOTICE file | ||
| ~ distributed with this work for additional information | ||
| ~ regarding copyright ownership. Metamarkets licenses this file | ||
| ~ to you under the Apache License, Version 2.0 (the | ||
| ~ "License"); you may not use this file except in compliance | ||
| ~ with the License. You may obtain a copy of the License at | ||
| ~ | ||
| ~ http://www.apache.org/licenses/LICENSE-2.0 | ||
| ~ | ||
| ~ Unless required by applicable law or agreed to in writing, | ||
| ~ software distributed under the License is distributed on an | ||
| ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| ~ KIND, either express or implied. See the License for the | ||
| ~ specific language governing permissions and limitations | ||
| ~ under the License. | ||
| --> | ||
|
|
||
| <project xmlns="http://maven.apache.org/POM/4.0.0" | ||
| xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" | ||
| xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> | ||
|
|
||
| <parent> | ||
| <groupId>io.druid</groupId> | ||
| <artifactId>druid</artifactId> | ||
| <version>0.9.3-SNAPSHOT</version> | ||
| <relativePath>../../pom.xml</relativePath> | ||
| </parent> | ||
| <modelVersion>4.0.0</modelVersion> | ||
|
|
||
| <groupId>io.druid.extensions.contrib</groupId> | ||
| <artifactId>scan-query</artifactId> | ||
| <name>scan-query</name> | ||
| <description>streaming version of select query</description> | ||
|
|
||
| <dependencies> | ||
| <dependency> | ||
| <groupId>io.druid</groupId> | ||
| <artifactId>druid-server</artifactId> | ||
| <version>${project.parent.version}</version> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>junit</groupId> | ||
| <artifactId>junit</artifactId> | ||
| <scope>test</scope> | ||
| </dependency> | ||
| <dependency> | ||
| <groupId>io.druid</groupId> | ||
| <artifactId>druid-processing</artifactId> | ||
| <version>${project.parent.version}</version> | ||
| <classifier>tests</classifier> | ||
| <scope>test</scope> | ||
| </dependency> | ||
| </dependencies> | ||
|
|
||
| </project> | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
looking through this review, I think the scan query is better than select in almost every way and I think maybe this should actually be a part of core, and be select query v2, with the goal to eventually replace the select query
Alternatively, we can keep this as an extension until we figure out the API
@gianm thoughts?