Skip to content

SQL support for t-digest based sketch aggregators #8100

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jon-wei merged 6 commits into from
Aug 5, 2019
Merged

SQL support for t-digest based sketch aggregators #8100

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
7f42f84
SQL support for t-digest based sketch aggregators
samarthjain Jul 17, 2019
4184db0
Fix teamcity errors
samarthjain Jul 26, 2019
bd643b9
Add missing dependencies
samarthjain Jul 27, 2019
4b9caac
Remove unused dependency
samarthjain Jul 28, 2019
c54bb5a
Address code review comments
samarthjain Jul 30, 2019
d0d9043
Add checks for compression param
samarthjain Aug 2, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
86 changes: 37 additions & 49 deletions docs/content/development/extensions-contrib/tdigestsketch-quantiles.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,16 +29,12 @@ T-Digest (https://github.com/tdunning/t-digest) is a popular datastructure for a
rank-based statistics such as quantiles and trimmed means.
The datastructure is also designed for parallel programming use cases like distributed aggregations or map reduce jobs by making combining two intermediate t-digests easy and efficient.

There are three flavors of T-Digest sketch aggregator available in Apache Druid (incubating):

1. buildTDigestSketch - used for building T-Digest sketches from raw numeric values. It generally makes sense to
use this aggregator when ingesting raw data into Druid. One can also use this aggregator during query time too to
generate sketches, just that one would be building these sketches on every query execution instead of building them
once during ingestion.
2. mergeTDigestSketch - used for merging pre-built T-Digest sketches. This aggregator is generally used during
query time to combine sketches generated by buildTDigestSketch aggregator.
3. quantilesFromTDigestSketch - used for generating quantiles from T-Digest sketches. This aggregator is generally used
during query time to generate quantiles from sketches built using the above two sketch generating aggregators.
The tDigestSketch aggregator is capable of generating sketches from raw numeric values as well as
aggregating/combining pre-generated T-Digest sketches generated using the tDigestSketch aggregator itself.
While one can generate sketches on the fly during the query time itself, it generally is more performant
to generate sketches during ingestion time itself and then combining them during query time.
The module also provides a postAggregator, quantilesFromTDigestSketch, that can be used to compute approximate
quantiles from T-Digest sketches generated by the tDigestSketch aggreator.

To use this aggregator, make sure you [include](../../operations/including-extensions.html) the extension in your config file:

Expand All @@ -48,11 +44,12 @@ druid.extensions.loadList=["druid-tdigestsketch"]

### Aggregator

The result of the aggregation is a T-Digest sketch that is built ingesting numeric values from the raw data.
The result of the aggregation is a T-Digest sketch that is built ingesting numeric values from the raw data or from
combining pre-generated T-Digest sketches.

```json
{
"type" : "buildTDigestSketch",
"type" : "tDigestSketch",
"name" : <output_name>,
"fieldName" : <metric_name>,
"compression": <parameter that controls size and accuracy>
Expand All @@ -63,56 +60,29 @@ Example:

```json
{
"type": "buildTDigestSketch",
"type": "tDigestSketch",
"name": "sketch",
"fieldName": "session_duration",
"compression": 200
}
```

|property|description|required?|
|--------|-----------|---------|
|type|This String should always be "buildTDigestSketch"|yes|
|name|A String for the output (result) name of the calculation.|yes|
|fieldName|A String for the name of the input field containing raw numeric values.|yes|
|compression|Parameter that determines the accuracy and size of the sketch. Higher compression means higher accuracy but more space to store sketches.|no, defaults to 100|


The result of the aggregation is a T-Digest sketch that is built by merging pre-built T-Digest sketches.

```json
{
"type" : "mergeTDigestSketch",
"name" : <output_name>,
"fieldName" : <metric_name>,
"compression": <parameter that controls size and accuracy>
}
"type": "tDigestSketch",
"name": "combined_sketch",
"fieldName": <input-column>,
"compression": 200
}
```

|property|description|required?|
|--------|-----------|---------|
|type|This String should always be "mergeTDigestSketch"|yes|
|type|This String should always be "tDigestSketch"|yes|
|name|A String for the output (result) name of the calculation.|yes|
|fieldName|A String for the name of the input field containing raw numeric values.|yes|
|fieldName|A String for the name of the input field containing raw numeric values or pre-generated T-Digest sketches.|yes|
|compression|Parameter that determines the accuracy and size of the sketch. Higher compression means higher accuracy but more space to store sketches.|no, defaults to 100|

Example:

```json
{
"queryType": "groupBy",
"dataSource": "test_datasource",
"granularity": "ALL",
"dimensions": [],
"aggregations": [{
"type": "mergeTDigestSketch",
"name": "merged_sketch",
"fieldName": "ingested_sketch",
"compression": 200
}],
"intervals": ["2016-01-01T00:00:00.000Z/2016-01-31T00:00:00.000Z"]
}
```

### Post Aggregators

Expand All @@ -133,7 +103,7 @@ This returns an array of quantiles corresponding to a given array of fractions.
|--------|-----------|---------|
|type|This String should always be "quantilesFromTDigestSketch"|yes|
|name|A String for the output (result) name of the calculation.|yes|
|fieldName|A String for the name of the input field containing raw numeric values.|yes|
|field|A field reference pointing to the field aggregated/combined T-Digest sketch.|yes|
Copy link
Copy Markdown
Contributor

@jon-wei jon-wei Jul 30, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggest rewriting to:

"The input field reference, which must be a PostAggregator that outputs T-Digest sketches. This can be a fieldAccess PostAggregator, which simply returns the value of the referenced input column."

|fractions|Non-empty array of fractions between 0 and 1|yes|

Example:
Expand All @@ -145,7 +115,7 @@ Example:
"granularity": "ALL",
"dimensions": [],
"aggregations": [{
"type": "mergeTDigestSketch",
"type": "tDigestSketch",
"name": "merged_sketch",
"fieldName": "ingested_sketch",
"compression": 200
Expand All @@ -162,3 +132,21 @@ Example:
"intervals": ["2016-01-01T00:00:00.000Z/2016-01-31T00:00:00.000Z"]
}
```

Similar to quantilesFromTDigestSketch except it takes in a single fraction for computing quantile.

```json
{
"type" : "quantileFromTDigestSketch",
"name": <output name>,
"field" : <post aggregator that refers to a TDigestSketch (fieldAccess or another post aggregator)>,
"fraction" : <value>
}
```

|property|description|required?|
|--------|-----------|---------|
|type|This String should always be "quantileFromTDigestSketch"|yes|
|name|A String for the output (result) name of the calculation.|yes|
|field|A field reference pointing to the field aggregated/combined T-Digest sketch.|yes|
|fraction|Decimal value between 0 and 1|yes|
2 changes: 2 additions & 0 deletions docs/content/querying/sql.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -184,6 +184,8 @@ Only the COUNT aggregation can accept DISTINCT.
|`APPROX_QUANTILE_DS(expr, probability, [k])`|Computes approximate quantiles on numeric or [Quantiles sketch](../development/extensions-core/datasketches-quantiles.html) exprs. The "probability" should be between 0 and 1 (exclusive). The `k` parameter is described in the Quantiles sketch documentation. The [DataSketches extension](../development/extensions-core/datasketches-extension.html) must be loaded to use this function.|
|`APPROX_QUANTILE_FIXED_BUCKETS(expr, probability, numBuckets, lowerLimit, upperLimit, [outlierHandlingMode])`|Computes approximate quantiles on numeric or [fixed buckets histogram](../development/extensions-core/approximate-histograms.html#fixed-buckets-histogram) exprs. The "probability" should be between 0 and 1 (exclusive). The `numBuckets`, `lowerLimit`, `upperLimit`, and `outlierHandlingMode` parameters are described in the fixed buckets histogram documentation. The [approximate histogram extension](../development/extensions-core/approximate-histograms.html) must be loaded to use this function.|
|`BLOOM_FILTER(expr, numEntries)`|Computes a bloom filter from values produced by `expr`, with `numEntries` maximum number of distinct values before false positive rate increases. See [bloom filter extension](../development/extensions-core/bloom-filter.html) documentation for additional details.|
|`TDIGEST_QUANTILE(expr, quantileFraction, [compression])`|Builds a T-Digest sketch on values produced by `expr` and returns the value for the quantile. Compression parameter (default value 100) determines the accuracy and size of the sketch. Higher compression means higher accuracy but more space to store sketches. See [t-digest extension](../development/extensions-contrib/tdigestsketch-quantiles.html) documentation for additional details.|
|`TDIGEST_GENERATE_SKETCH(expr, [compression])`|Builds a T-Digest sketch on values produced by `expr`. Compression parameter (default value 100) determines the accuracy and size of the sketch Higher compression means higher accuracy but more space to store sketches. See [t-digest extension](../development/extensions-contrib/tdigestsketch-quantiles.html) documentation for additional details.|
|`VAR_POP(expr)`|Computes variance population of `expr`. See [stats extension](../development/extensions-core/stats.html) documentation for additional details.|
|`VAR_SAMP(expr)`|Computes variance sample of `expr`. See [stats extension](../development/extensions-core/stats.html) documentation for additional details.|
|`VARIANCE(expr)`|Computes variance sample of `expr`. See [stats extension](../development/extensions-core/stats.html) documentation for additional details.|
Expand Down
84 changes: 83 additions & 1 deletion extensions-contrib/tdigestsketch/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,80 @@
<version>${project.parent.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.google.code.findbugs</groupId>
<artifactId>jsr305</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.google.inject</groupId>
<artifactId>guice</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.datatype</groupId>
<artifactId>jackson-datatype-guava</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.datatype</groupId>
<artifactId>jackson-datatype-joda</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.dataformat</groupId>
<artifactId>jackson-dataformat-smile</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.jaxrs</groupId>
<artifactId>jackson-jaxrs-json-provider</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.jaxrs</groupId>
<artifactId>jackson-jaxrs-smile-provider</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>it.unimi.dsi</groupId>
<artifactId>fastutil</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-core</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>com.google.errorprone</groupId>
<artifactId>error_prone_annotations</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.druid</groupId>
<artifactId>druid-sql</artifactId>
<version>${project.parent.version}</version>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.calcite</groupId>
<artifactId>calcite-core</artifactId>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.apache.druid</groupId>
<artifactId>druid-server</artifactId>
<scope>provided</scope>
<version>${project.parent.version}</version>
</dependency>

<!-- Test Dependencies -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
Expand Down Expand Up @@ -86,9 +160,17 @@
<groupId>org.apache.druid</groupId>
<artifactId>druid-server</artifactId>
<version>${project.parent.version}</version>
<type>test-jar</type>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.apache.druid</groupId>
<artifactId>druid-sql</artifactId>
<version>${project.parent.version}</version>
<type>test-jar</type>
<scope>test</scope>
</dependency>

</dependencies>

</project>
</project>
83 changes: 0 additions & 83 deletions ...n/java/org/apache/druid/query/aggregation/tdigestsketch/TDigestMergeSketchAggregator.java
View file
Open in desktop

This file was deleted.

Loading