document new filters and stuff by clintropolis · Pull Request #14760 · apache/druid

clintropolis · 2023-08-05T00:51:45Z

Description

Adds documentation to the new filters and SQL query context added in #14542, and also re-arranges some of the native filter documentation and makes things consistently use tables to specify their grammar similar to as I did in #14497.

clintropolis · 2023-08-05T00:58:57Z

-.getAPI {
-  color: #0073e6; 
-  font-weight: bold;
-}
-.postAPI {
-  color: #00bf7d; 
-  font-weight: bold;
-}
-.deleteAPI {
-  color: #f49200; 
-  font-weight: bold;
-}


hmm... idk why this happened, must've been when running mvn commands to spellcheck before I committed

do you want to just restore the changes to this file?

writer-jill

Made some suggestions!

writer-jill · 2023-08-07T13:02:12Z

 the [array functions](../querying/sql-array-functions.md) or [UNNEST](../querying/sql-functions.md#unnest). Nested
 columns can be queried with the [JSON functions](../querying/sql-json-functions.md).

+We also highly recommend setting `druid.generic.useDefaultValueForNull=false` when using these columns since it also enables out of the box `ARRAY` type filtering. If this is not set to true, setting `sqlUseBoundsAndSelectors` to `false` on the [SQL query context](../querying/sql-query-context.md) can enable `ARRAY` filtering.


I don't understand this. First it says set useDefaultValueForNull to false to enable ARRAY filtering. Then it says if this is set to false (not set to true) you can set something else to enable ARRAY filtering.

oops yeah, this was a mistake

writer-jill · 2023-08-07T13:02:32Z

+| -------- | ----------- | -------- |
+| `type` | Must be "selector".| Yes |
+| `dimension` | Input column or virtual column name to filter. | Yes |
+| `value` | String value to match. | No, if not specified the filter will match NULL values. |


writer-jill · 2023-08-07T13:02:56Z

-
-This is the equivalent of `WHERE <dimension_string> = '<dimension_value_string>'` or `WHERE <dimension_string> IS NULL`
-(if the `value` is `null`).
+The selector filter is limited to only being able to match against `STRING` (single and multi-valued), `LONG`, `FLOAT`,


Suggested change

The selector filter is limited to only being able to match against `STRING` (single and multi-valued), `LONG`, `FLOAT`,

The selector filter can only match against `STRING` (single and multi-valued), `LONG`, `FLOAT`,

writer-jill · 2023-08-07T13:03:16Z

+`DOUBLE` types. Use the newer null and equality filters to match against `ARRAY` or `COMPLEX` types.

-The selector filter supports the use of extraction functions, see [Filtering with Extraction Functions](#filtering-with-extraction-functions) for details.
+When the selector filter matches against numeric inputs, the string `value` will be best effort coerced into a numeric value.


Suggested change

When the selector filter matches against numeric inputs, the string `value` will be best effort coerced into a numeric value.

When the selector filter matches against numeric inputs, the string `value` will be best-effort coerced into a numeric value.

writer-jill · 2023-08-07T13:04:43Z

+## Equality Filter

-## Logical expression filters
+The equality filter is a replacement for the selector filter with the ability to match against any type of column. The equality filter intends to have more SQL compatible behavior than the selector filter and so cannot match NULL values, use the null filter instead.


Suggested change

The equality filter is a replacement for the selector filter with the ability to match against any type of column. The equality filter intends to have more SQL compatible behavior than the selector filter and so cannot match NULL values, use the null filter instead.

The equality filter is a replacement for the selector filter with the ability to match against any type of column. The equality filter is designed to include more SQL-compatible behavior than the selector filter and so can't match null values. To match null values, use the null filter.

writer-jill · 2023-08-07T13:40:35Z

 Druid supports filtering on timestamp, string, long, and float columns.

-Note that only string columns have bitmap indexes. Therefore, queries that filter on other column types will need to
+Note that only string columns and columns produced with the ['auto' ingestion spec](../ingestion/ingestion-spec.md#dimension-objects) also used by [type aware schema discovery](../ingestion/schema-design.md#type-aware-schema-discovery) have bitmap indexes. Therefore, queries that filter on other column types will need to


Suggested change

Note that only string columns and columns produced with the ['auto' ingestion spec](../ingestion/ingestion-spec.md#dimension-objects) also used by [type aware schema discovery](../ingestion/schema-design.md#type-aware-schema-discovery) have bitmap indexes. Therefore, queries that filter on other column types will need to

Note that only string columns and columns produced with the ['auto' ingestion spec](../ingestion/ingestion-spec.md#dimension-objects) also used by [type aware schema discovery](../ingestion/schema-design.md#type-aware-schema-discovery) have bitmap indexes. Queries that filter on other column types must

writer-jill · 2023-08-07T13:41:01Z


+### Filtering on multi-value string columns
+
+All filters will return true if any one of the dimension values is satisfies the filter.


Suggested change

All filters will return true if any one of the dimension values is satisfies the filter.

All filters return true if any one of the dimension values satisfies the filter.

writer-jill · 2023-08-07T13:44:52Z

 |`enableTimeBoundaryPlanning`|If true, SQL queries will get converted to TimeBoundary queries wherever possible. TimeBoundary queries are very efficient for min-max calculation on `__time` column in a datasource |`druid.query.default.context.enableTimeBoundaryPlanning` on the Broker (default: false)|
 |`useNativeQueryExplain`|If true, `EXPLAIN PLAN FOR` will return the explain plan as a JSON representation of equivalent native query(s), else it will return the original version of explain plan generated by Calcite.<br /><br />This property is provided for backwards compatibility. It is not recommended to use this parameter unless you were depending on the older behavior.|`druid.sql.planner.useNativeQueryExplain` on the Broker (default: true)|
 |`sqlFinalizeOuterSketches`|If false (default behavior in Druid 25.0.0 and later), `DS_HLL`, `DS_THETA`, and `DS_QUANTILES_SKETCH` return sketches in query results, as documented. If true (default behavior in Druid 24.0.1 and earlier), sketches from these functions are finalized when they appear in query results.<br /><br />This property is provided for backwards compatibility with behavior in Druid 24.0.1 and earlier. It is not recommended to use this parameter unless you were depending on the older behavior. Instead, use a function that does not return a sketch, such as `APPROX_COUNT_DISTINCT_DS_HLL`, `APPROX_COUNT_DISTINCT_DS_THETA`, `APPROX_QUANTILE_DS`, `DS_THETA_ESTIMATE`, or `DS_GET_QUANTILE`.|`druid.query.default.context.sqlFinalizeOuterSketches` on the Broker (default: false)|
+|`sqlUseBoundAndSelectors`|If false (default behavior if `druid.generic.useDefaultValueForNull=false` in Druid 27 and later), the SQL planner will use [equality](./filters.md#equality-filter), [null](./filters.md#null-filter), and [range](./filters.md#range-filter) filters instead of [selector](./filters.md#selector-filter) and [bounds](./filters.md#bound-filter). This value must be set to `false` for correct behavior for filtering `ARRAY` typed values. | Defaults to same value as `druid.generic.useDefaultValueForNull`. |


Suggested change

|`sqlUseBoundAndSelectors`|If false (default behavior if `druid.generic.useDefaultValueForNull=false` in Druid 27 and later), the SQL planner will use [equality](./filters.md#equality-filter), [null](./filters.md#null-filter), and [range](./filters.md#range-filter) filters instead of [selector](./filters.md#selector-filter) and [bounds](./filters.md#bound-filter). This value must be set to `false` for correct behavior for filtering `ARRAY` typed values. | Defaults to same value as `druid.generic.useDefaultValueForNull`. |

|`sqlUseBoundAndSelectors`|If false (default behavior if `druid.generic.useDefaultValueForNull=false` in Druid 27.0.0 and later), the SQL planner will use [equality](./filters.md#equality-filter), [null](./filters.md#null-filter), and [range](./filters.md#range-filter) filters instead of [selector](./filters.md#selector-filter) and [bounds](./filters.md#bound-filter). This value must be set to `false` for correct behavior for filtering `ARRAY` typed values. | Defaults to same value as `druid.generic.useDefaultValueForNull`. |

writer-jill · 2023-08-07T13:46:35Z

@@ -490,20 +806,31 @@ should be specified as if the timestamp values were strings.

 If the user wishes to interpret the timestamp with a specific format, timezone, or locale, the [Time Format Extraction Function](./dimensionspecs.md#time-format-extraction-function) is useful.


Suggested change

If the user wishes to interpret the timestamp with a specific format, timezone, or locale, the [Time Format Extraction Function](./dimensionspecs.md#time-format-extraction-function) is useful.

If you want to interpret the timestamp with a specific format, timezone, or locale, use the [Time Format Extraction Function](./dimensionspecs.md#time-format-extraction-function).

writer-jill · 2023-08-07T13:47:44Z

+```
+will successfully match the entire row, but not match a row with value `['a', 'c']`.
+
+To express this filter in SQL, one would need to use [SQL multi-value string functions](./sql-multivalue-string-functions.md) such as `MV_CONTAINS`, which can be optimized by the planner to the same native filters.


Suggested change

To express this filter in SQL, one would need to use [SQL multi-value string functions](./sql-multivalue-string-functions.md) such as `MV_CONTAINS`, which can be optimized by the planner to the same native filters.

To express this filter in SQL, use [SQL multi-value string functions](./sql-multivalue-string-functions.md) such as `MV_CONTAINS`, which can be optimized by the planner to the same native filters.

writer-jill

Made some suggestions!

vtlim

Some nits on style

vtlim · 2023-08-08T20:06:33Z

+| -------- | ----------- | -------- |
+| `type` | Must be "equality".| Yes |
+| `column` | Input column or virtual column name to filter. | Yes |
+| `matchValueType` | String specifying the type of value to match, for example `STRING`, `LONG`, `DOUBLE`, `FLOAT`, `ARRAY<STRING>`, `ARRAY<LONG>`, or any other Druid type. The `matchValueType` determines how Druid interprets the `matchValue` to assist in converting to the type of the matched `column`. | Yes |


Suggested change

| `matchValueType` | String specifying the type of value to match, for example `STRING`, `LONG`, `DOUBLE`, `FLOAT`, `ARRAY<STRING>`, `ARRAY<LONG>`, or any other Druid type. The `matchValueType` determines how Druid interprets the `matchValue` to assist in converting to the type of the matched `column`. | Yes |

| `matchValueType` | String specifying the type of value to match. For example, `STRING`, `LONG`, `DOUBLE`, `FLOAT`, `ARRAY<STRING>`, `ARRAY<LONG>`, or any other Druid type. The `matchValueType` determines how Druid interprets the `matchValue` to assist in converting to the type of the matched `column`. | Yes |

vtlim · 2023-08-08T20:07:06Z

+Note that the column comparison filter converts all values to strings prior to comparison. This allows differently-typed input columns to match without a cast operation.

-Search filters can be used to filter on partial string matches.
+### Example: equivalent of `WHERE someColumn = someLongColumn`.


Suggested change

### Example: equivalent of `WHERE someColumn = someLongColumn`.

### Example: equivalent of `WHERE someColumn = someLongColumn`

vtlim · 2023-08-08T20:07:42Z

+
+Note that the bound filter matches null values if you don't specify a lower bound. Use the range filter if SQL-compatible behavior.
+
+### Example: equivalent to `WHERE 21 <= age <= 31`:


Suggested change

### Example: equivalent to `WHERE 21 <= age <= 31`:

### Example: equivalent to `WHERE 21 <= age <= 31`

vtlim · 2023-08-08T20:07:54Z

 ```

-This filter expresses the condition `foo <= name <= hoo`, using the default lexicographic sorting order.
+### Example: equivalent to `WHERE 'foo' <= name <= 'hoo'`, using the default lexicographic sorting order.


Suggested change

### Example: equivalent to `WHERE 'foo' <= name <= 'hoo'`, using the default lexicographic sorting order.

### Example: equivalent to `WHERE 'foo' <= name <= 'hoo'`, using the default lexicographic sorting order

vtlim · 2023-08-08T20:07:59Z

 ```

-The user can also specify a one-sided bound by omitting "upper" or "lower". This filter expresses `age < 31`.
+### Example: equivalent to `WHERE age < 31`.


Suggested change

### Example: equivalent to `WHERE age < 31`.

### Example: equivalent to `WHERE age < 31`

vtlim · 2023-08-08T20:13:01Z

+
+All filters return true if any one of the dimension values is satisfies the filter.
+
+#### Example: multi-value match behavior.


Suggested change

#### Example: multi-value match behavior.

#### Example: multi-value match behavior

vtlim · 2023-08-08T20:13:33Z

 the "regex" filter) the numeric column values will be converted to strings during the scan.

-For example, filtering on a specific value, `myFloatColumn = 10.1`:
+#### Example: filtering on a specific value, `myFloatColumn = 10.1`:


Suggested change

#### Example: filtering on a specific value, `myFloatColumn = 10.1`:

#### Example: filtering on a specific value, `myFloatColumn = 10.1`

vtlim · 2023-08-08T20:13:48Z

 ```

-Filtering on a range of values, `10 <= myFloatColumn < 20`:
+#### Example: filtering on a range of values, `10 <= myFloatColumn < 20`:


Suggested change

#### Example: filtering on a range of values, `10 <= myFloatColumn < 20`:

#### Example: filtering on a range of values, `10 <= myFloatColumn < 20`

vtlim · 2023-08-08T20:14:17Z

 ```

-Filtering on day of week:
+#### Example: filtering on day of week using an extractionFn


Suggested change

#### Example: filtering on day of week using an extractionFn

#### Example: filtering on day of week using an extraction function

vtlim · 2023-08-08T20:15:04Z

 |`enableTimeBoundaryPlanning`|If true, SQL queries will get converted to TimeBoundary queries wherever possible. TimeBoundary queries are very efficient for min-max calculation on `__time` column in a datasource |`druid.query.default.context.enableTimeBoundaryPlanning` on the Broker (default: false)|
 |`useNativeQueryExplain`|If true, `EXPLAIN PLAN FOR` will return the explain plan as a JSON representation of equivalent native query(s), else it will return the original version of explain plan generated by Calcite.<br /><br />This property is provided for backwards compatibility. It is not recommended to use this parameter unless you were depending on the older behavior.|`druid.sql.planner.useNativeQueryExplain` on the Broker (default: true)|
 |`sqlFinalizeOuterSketches`|If false (default behavior in Druid 25.0.0 and later), `DS_HLL`, `DS_THETA`, and `DS_QUANTILES_SKETCH` return sketches in query results, as documented. If true (default behavior in Druid 24.0.1 and earlier), sketches from these functions are finalized when they appear in query results.<br /><br />This property is provided for backwards compatibility with behavior in Druid 24.0.1 and earlier. It is not recommended to use this parameter unless you were depending on the older behavior. Instead, use a function that does not return a sketch, such as `APPROX_COUNT_DISTINCT_DS_HLL`, `APPROX_COUNT_DISTINCT_DS_THETA`, `APPROX_QUANTILE_DS`, `DS_THETA_ESTIMATE`, or `DS_GET_QUANTILE`.|`druid.query.default.context.sqlFinalizeOuterSketches` on the Broker (default: false)|
+|`sqlUseBoundAndSelectors`|If false (default behavior if `druid.generic.useDefaultValueForNull=false` in Druid 27.0.0 and later), the SQL planner will use [equality](./filters.md#equality-filter), [null](./filters.md#null-filter), and [range](./filters.md#range-filter) filters instead of [selector](./filters.md#selector-filter) and [bounds](./filters.md#bound-filter). This value must be set to `false` for correct behavior for filtering `ARRAY` typed values. | Defaults to same value as `druid.generic.useDefaultValueForNull`. |


Suggested change

|`sqlUseBoundAndSelectors`|If false (default behavior if `druid.generic.useDefaultValueForNull=false` in Druid 27.0.0 and later), the SQL planner will use [equality](./filters.md#equality-filter), [null](./filters.md#null-filter), and [range](./filters.md#range-filter) filters instead of [selector](./filters.md#selector-filter) and [bounds](./filters.md#bound-filter). This value must be set to `false` for correct behavior for filtering `ARRAY` typed values. | Defaults to same value as `druid.generic.useDefaultValueForNull`. |

|`sqlUseBoundAndSelectors`|If false (default behavior if `druid.generic.useDefaultValueForNull=false` in Druid 27.0.0 and later), the SQL planner will use [equality](./filters.md#equality-filter), [null](./filters.md#null-filter), and [range](./filters.md#range-filter) filters instead of [selector](./filters.md#selector-filter) and [bounds](./filters.md#bound-filter). This value must be set to `false` for correct behavior for filtering `ARRAY` typed values. | Defaults to same value as `druid.generic.useDefaultValueForNull` |

document new filters and stuff

0b54fda

clintropolis added Area - Documentation Area - Querying labels Aug 5, 2023

clintropolis added this to the 27.0 milestone Aug 5, 2023

clintropolis added 2 commits August 4, 2023 17:54

oops

bb55f52

double oops

0506b47

clintropolis commented Aug 5, 2023

View reviewed changes

writer-jill suggested changes Aug 7, 2023

View reviewed changes

adjust for review

52a54bf

vtlim reviewed Aug 8, 2023

View reviewed changes

clintropolis added 3 commits August 8, 2023 14:55

Merge remote-tracking branch 'upstream/master' into new-filter-docs

9bc4f4c

review stuff

7c49710

fixes

ce3171e

vtlim approved these changes Aug 8, 2023

View reviewed changes

vtlim merged commit e57f880 into apache:master Aug 8, 2023

clintropolis deleted the new-filter-docs branch August 8, 2023 23:09

clintropolis added a commit to clintropolis/druid that referenced this pull request Aug 8, 2023

document new filters and stuff (apache#14760)

c3c1114

clintropolis mentioned this pull request Aug 8, 2023

[Backport] document new filters and stuff (#14760) #14785

Merged

AmatyaAvadhanula pushed a commit that referenced this pull request Aug 11, 2023

document new filters and stuff (#14760) (#14785)

42dace8

clintropolis mentioned this pull request Aug 18, 2023

enable sql compatible null handling mode by default #14792

Merged

4 tasks

	\| `value` \| String value to match. \| No, if not specified the filter will match NULL values. \|
	\| `value` \| String value to match. \| No, if not specified the filter matches NULL values. \|

	The selector filter is limited to only being able to match against `STRING` (single and multi-valued), `LONG`, `FLOAT`,
	The selector filter can only match against `STRING` (single and multi-valued), `LONG`, `FLOAT`,

	When the selector filter matches against numeric inputs, the string `value` will be best effort coerced into a numeric value.
	When the selector filter matches against numeric inputs, the string `value` will be best-effort coerced into a numeric value.

	The equality filter is a replacement for the selector filter with the ability to match against any type of column. The equality filter intends to have more SQL compatible behavior than the selector filter and so cannot match NULL values, use the null filter instead.
	The equality filter is a replacement for the selector filter with the ability to match against any type of column. The equality filter is designed to include more SQL-compatible behavior than the selector filter and so can't match null values. To match null values, use the null filter.

	Note that only string columns and columns produced with the ['auto' ingestion spec](../ingestion/ingestion-spec.md#dimension-objects) also used by [type aware schema discovery](../ingestion/schema-design.md#type-aware-schema-discovery) have bitmap indexes. Therefore, queries that filter on other column types will need to
	Note that only string columns and columns produced with the ['auto' ingestion spec](../ingestion/ingestion-spec.md#dimension-objects) also used by [type aware schema discovery](../ingestion/schema-design.md#type-aware-schema-discovery) have bitmap indexes. Queries that filter on other column types must


		### Filtering on multi-value string columns

		All filters will return true if any one of the dimension values is satisfies the filter.

	All filters will return true if any one of the dimension values is satisfies the filter.
	All filters return true if any one of the dimension values satisfies the filter.

		@@ -490,20 +806,31 @@ should be specified as if the timestamp values were strings.

		If the user wishes to interpret the timestamp with a specific format, timezone, or locale, the [Time Format Extraction Function](./dimensionspecs.md#time-format-extraction-function) is useful.

	If the user wishes to interpret the timestamp with a specific format, timezone, or locale, the [Time Format Extraction Function](./dimensionspecs.md#time-format-extraction-function) is useful.
	If you want to interpret the timestamp with a specific format, timezone, or locale, use the [Time Format Extraction Function](./dimensionspecs.md#time-format-extraction-function).

	To express this filter in SQL, one would need to use [SQL multi-value string functions](./sql-multivalue-string-functions.md) such as `MV_CONTAINS`, which can be optimized by the planner to the same native filters.
	To express this filter in SQL, use [SQL multi-value string functions](./sql-multivalue-string-functions.md) such as `MV_CONTAINS`, which can be optimized by the planner to the same native filters.

	\| `matchValueType` \| String specifying the type of value to match, for example `STRING`, `LONG`, `DOUBLE`, `FLOAT`, `ARRAY<STRING>`, `ARRAY<LONG>`, or any other Druid type. The `matchValueType` determines how Druid interprets the `matchValue` to assist in converting to the type of the matched `column`. \| Yes \|
	\| `matchValueType` \| String specifying the type of value to match. For example, `STRING`, `LONG`, `DOUBLE`, `FLOAT`, `ARRAY<STRING>`, `ARRAY<LONG>`, or any other Druid type. The `matchValueType` determines how Druid interprets the `matchValue` to assist in converting to the type of the matched `column`. \| Yes \|

	### Example: equivalent of `WHERE someColumn = someLongColumn`.
	### Example: equivalent of `WHERE someColumn = someLongColumn`


		Note that the bound filter matches null values if you don't specify a lower bound. Use the range filter if SQL-compatible behavior.

		### Example: equivalent to `WHERE 21 <= age <= 31`:

	### Example: equivalent to `WHERE 'foo' <= name <= 'hoo'`, using the default lexicographic sorting order.
	### Example: equivalent to `WHERE 'foo' <= name <= 'hoo'`, using the default lexicographic sorting order

	### Example: equivalent to `WHERE age < 31`.
	### Example: equivalent to `WHERE age < 31`


		All filters return true if any one of the dimension values is satisfies the filter.

		#### Example: multi-value match behavior.

	#### Example: filtering on a specific value, `myFloatColumn = 10.1`:
	#### Example: filtering on a specific value, `myFloatColumn = 10.1`

	#### Example: filtering on a range of values, `10 <= myFloatColumn < 20`:
	#### Example: filtering on a range of values, `10 <= myFloatColumn < 20`

	#### Example: filtering on day of week using an extractionFn
	#### Example: filtering on day of week using an extraction function

Conversation

clintropolis commented Aug 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

writer-jill left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

writer-jill left a comment

Choose a reason for hiding this comment

Uh oh!

vtlim left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

clintropolis commented Aug 5, 2023 •

edited

Loading