feat(native-filters): add search all filter options by villebro · Pull Request #14710 · apache/superset

villebro · 2021-05-19T13:00:46Z

SUMMARY

Add support for "Search all filter options" option to native select filter to obtain feature parity with same functionality in Filter Box. Other changes:

Filter Box performed a manual case insensitive comparison for the search all filter functionality by adding the following WHERE clause to the query: lower(${key}) like '%${input}%', which only worked on those backends that support the lower function (a non-ANSI SQL function). This has been replaced by the following logic
- For string columns, we perform a native case insensitive filter by utilizing the SqlAlchemy ILIKE operator ( https://docs.sqlalchemy.org/en/13/core/sqlelement.html#sqlalchemy.sql.expression.ColumnElement.ilike ). This makes it possible to utilize the ILIKE operator where supported, e.g. on Postgres.
- For numeric columns, perform a >= operator. Previously the Filter Box would fail silently for numeric columns, as it tried to call lower() on the numeric column, which usually doesn't work.
Byproduct: Add ILIKE operator option to adhoc filter popover (see screenshot)

SCREENSHOT

See below how searching for entries that don't fit into the first 1000 entries are found when the feature is enabled:
https://user-images.githubusercontent.com/33317356/119319233-d88fb880-bc82-11eb-9a11-e63aca9e243f.mp4

Also add case case insensitive LIKE operator to adhoc filter popover to support ILIKE SqlAlchemy operator:

Also pull in the following PRs from superset-ui:

TEST PLAN

ADDITIONAL INFORMATION

Has associated issue:
Changes UI
Includes DB Migration (follow approval process in SIP-59)
- Migration is atomic, supports rollback & is backwards-compatible
- Confirm DB migration upgrade and downgrade tested
- Runtime estimates and downtime expectations provided
Introduces new feature or API
Removes existing feature or API

codecov · 2021-05-21T13:00:14Z

Codecov Report

Merging #14710 (5996b58) into master (bee6f3b) will decrease coverage by 0.01%.
The diff coverage is 66.27%.

@@            Coverage Diff             @@
##           master   #14710      +/-   ##
==========================================
- Coverage   77.32%   77.30%   -0.02%     
==========================================
  Files         962      962              
  Lines       49093    49147      +54     
  Branches     6165     6182      +17     
==========================================
+ Hits        37959    37994      +35     
- Misses      10930    10949      +19     
  Partials      204      204

Flag	Coverage Δ
javascript	`72.25% <67.46%> (-0.02%)`	⬇️
mysql	`81.63% <33.33%> (-0.01%)`	⬇️
postgres	`81.66% <33.33%> (-0.01%)`	⬇️
python	`81.74% <33.33%> (-0.01%)`	⬇️
sqlite	`81.26% <33.33%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...tersConfigModal/FiltersConfigForm/DefaultValue.tsx	`27.77% <ø> (ø)`
...nd/src/dashboard/components/nativeFilters/utils.ts	`80.48% <ø> (ø)`
...onents/controls/FilterControl/AdhocFilter/index.js	`97.01% <ø> (ø)`
...tend/src/filters/components/Select/controlPanel.ts	`58.33% <0.00%> (-5.31%)`	⬇️
...et-frontend/src/filters/components/Select/index.ts	`100.00% <ø> (ø)`
...nd/src/filters/components/Select/transformProps.ts	`71.42% <ø> (ø)`
...et-frontend/src/filters/components/Select/types.ts	`100.00% <ø> (ø)`
superset/connectors/sqla/models.py	`88.08% <0.00%> (-0.22%)`	⬇️
...veFilters/FilterBar/FilterControls/FilterValue.tsx	`67.44% <52.63%> (-0.13%)`	⬇️
...c/filters/components/Select/SelectFilterPlugin.tsx	`69.23% <60.46%> (-9.35%)`	⬇️
... and 6 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bee6f3b...5996b58. Read the comment docs.

villebro · 2021-05-21T15:58:09Z

+type DataMaskAction =
+  | { type: 'ownState'; ownState: JsonObject }
+  | {
+      type: 'filterState';
+      extraFormData: ExtraFormData;
+      filterState: { value: SelectValue };
+    };
+
+function reducer(state: DataMask, action: DataMaskAction): DataMask {
+  switch (action.type) {
+    case 'ownState':
+      return {
+        ...state,
+        ownState: {
+          ...(state.ownState || {}),
+          ...action.ownState,
+        },
+      };
+    case 'filterState':
+      return {
+        ...state,
+        extraFormData: action.extraFormData,
+        filterState: {
+          ...(state.filterState || {}),
+          ...action.filterState,
+        },
+      };
+    default:
+      return {
+        ...state,
+      };
+  }
+}


I decided to migrate from multiple useStates and useEffects to useReducer, as the full dataMask will be updated by different actions but needs to be submitted as a whole to setDataMask. Therefore, useReducer seemed like a cleaner approach.

villebro · 2021-05-21T15:59:45Z

-  const initSelectValue: SelectValue =
-    // `defaultValue` can be `FIRST_VALUE` if `defaultToFirstItem` is checked, so need convert it to correct value for Select
-    defaultValue === FIRST_VALUE ? [] : defaultValue ?? [];
-
-  const firstItem: SelectValue = data[0]
-    ? (groupby.map(col => data[0][col]) as string[]) ?? initSelectValue
-    : initSelectValue;

  // If we are in config modal we always need show empty select for `defaultToFirstItem`
-  const [values, setValues] = useState<SelectValue>(
-    defaultToFirstItem && appSection !== AppSection.FILTER_CONFIG_MODAL
-      ? firstItem
-      : initSelectValue,
-  );


The FIRST_VALUE placeholder doesn't really work, as we don't know what the first value is for each individual user, which means that it won't be possible to persist extraFormData at filter configuration time.

michael-s-molina · 2021-05-22T13:29:36Z

@villebro I noticed two things:

I added a filter of countries. When I opened the filter you can see 'Australia' as one of the values. When I type 'aus', Australia is not shown.
When I search for a country and there are no results, there's no way to change my search because the search field is replaced by an empty message

20210522102235847.mp4

villebro · 2021-05-24T09:04:56Z

@michael-s-molina this is weird - the "No Results" behavior you describe should not happen after bumping the superset-ui packages. Can you check again that you have the npm bumps applied?

The search problem you faced is due to previously using the LIKE filter type - I have now updated to use ILIKE, which will perform a case insensitive lookup. So now searching for "aus" will find "Australia" on e.g. Postgres.

michael-s-molina · 2021-05-24T10:40:18Z

@michael-s-molina this is weird - the "No Results" behavior you describe should not happen after bumping the superset-ui packages. Can you check again that you have the npm bumps applied?

The search problem you faced is due to previously using the LIKE filter type - I have now updated to use ILIKE, which will perform a case insensitive lookup. So now searching for "aus" will find "Australia" on e.g. Postgres.

@villebro After pulling the changes and bumping the superset-ui packages I can confirm that both issues are fixed 😃

villebro · 2021-05-24T11:52:46Z

+    // handle changes coming from application, e.g. "Clear all" button
+    if (JSON.stringify(values) !== JSON.stringify(filterState.value)) {
+      handleChange(filterState.value);
+    }
+  }, [JSON.stringify(filterState.value)]);


This will be removed in a follow-up PR when all state handling is moved to the application

kgabryje · 2021-05-24T13:04:13Z

+import React, { useCallback, useEffect, useReducer, useState } from 'react';
 import { Select } from 'src/common/components';
-import { FIRST_VALUE, PluginFilterSelectProps, SelectValue } from './types';
+import { debounce } from 'lodash';


Apparently it's better to import debounce from 'lodash/debounce', as it results in smaller bundle size (and we probably should apply that in other places in superset).
References:
https://stackoverflow.com/questions/35250500/correct-way-to-import-lodash
https://www.blazemeter.com/blog/the-correct-way-to-import-lodash-libraries-a-benchmark

Good to know 👍

villebro · 2021-05-24T13:13:01Z

    granularity_sqla: 'ds',
    metrics: ['count'],
-    row_limit: 10000,
+    row_limit: 1000,


The default for Filter Box is 1000 rows.

* master: (163 commits) fix(native-filters): Manage default value of filters by superset (apache#14785) fix: Additional ResultSet tests (apache#14741) chore: added BasicParametersMixin to Redshift (apache#14752) fix: make dataset list sort case insensitive (apache#14528) fix: use encodeURIComponent when getting table metadata (apache#14790) fix: ensure engine is outside parameters (apache#14787) database modal should close on connect with tab layout (apache#14771) feat(native-filters): add search all filter options (apache#14710) fix: extra query in Dashboard when native filter enabled (apache#14770) chore: Improves the native filters UI/UX - iteration 2 (apache#14753) fix(native filters): Fix explore state (apache#14779) fix(explore): DndColumnSelect not handling controls with "multi: false" (apache#14737) feat: Create BigQuery Parameters for DatabaseModal (apache#14721) feat: enable user impersonation in GSheets (apache#14767) fix: add DB should not say it's Postgres (apache#14766) Revert "fix(dashboard): multiple query trigger when native filter enabled (apache#14734)" (apache#14762) feat: save database with new dynamic form (apache#14583) fix: save non-parameter DBs (apache#14759) chore: Removes ColorSchemeControl.less (apache#14199) fix(explore): Icons width (apache#14717) ...

* feat(native-filters): add search all filter options * add tests * fix default value * implement ILIKE operator * rebump packages * fix test * address comments * fix state changes coming from application * fix debouncer

superset-github-bot Bot added the preset-io label May 19, 2021

pull-request-size Bot added the size/L label May 19, 2021

villebro force-pushed the villebro/select-load branch from 6f2b3c3 to 3998f22 Compare May 19, 2021 14:28

villebro added 3 commits May 21, 2021 14:04

feat(native-filters): add search all filter options

9018ddc

add tests

319ad32

fix default value

604100f

villebro force-pushed the villebro/select-load branch from e6727a5 to 604100f Compare May 21, 2021 12:06

remove redundant action

1f6e6c2

villebro changed the title ~~[WIP] feat(native-filters): add search all filter options~~ feat(native-filters): add search all filter options May 21, 2021

villebro commented May 21, 2021

View reviewed changes

villebro added 3 commits May 24, 2021 09:43

Merge branch 'master' into villebro/select-load

112a6e4

implement ILIKE operator

07a623a

rebump packages

a14b1d5

fix test

e800d8e

simcha90 reviewed May 24, 2021

View reviewed changes

Comment thread superset-frontend/src/filters/components/Select/SelectFilterPlugin.tsx

simcha90 reviewed May 24, 2021

View reviewed changes

Comment thread superset-frontend/src/filters/components/Select/SelectFilterPlugin.tsx Outdated

simcha90 reviewed May 24, 2021

View reviewed changes

Comment thread superset-frontend/src/filters/components/Select/SelectFilterPlugin.tsx Outdated

simcha90 reviewed May 24, 2021

View reviewed changes

Comment thread superset-frontend/src/filters/components/Select/SelectFilterPlugin.tsx Outdated

address comments

9770ee0

fix state changes coming from application

1116195

villebro commented May 24, 2021

View reviewed changes

Merge branch 'master' into villebro/select-load

d9dc6b8

simcha90 approved these changes May 24, 2021

View reviewed changes

fix debouncer

5996b58

kgabryje reviewed May 24, 2021

View reviewed changes

villebro commented May 24, 2021

View reviewed changes

kgabryje approved these changes May 24, 2021

View reviewed changes

michael-s-molina approved these changes May 24, 2021

View reviewed changes

villebro merged commit e9657af into apache:master May 24, 2021

villebro deleted the villebro/select-load branch May 24, 2021 13:34

villebro mentioned this pull request May 24, 2021

fix(native-filters): fix loop caused by external state handler #14788

Merged

8 tasks

villebro mentioned this pull request May 28, 2021

fix(filter box): replace freeform where clause with ilike #14900

Merged

8 tasks

junlincc added the dashboard:native-filters Related to the native filters of the Dashboard label May 28, 2021

john-bodley mentioned this pull request Jun 16, 2023

fix: Native filter dynamic numeric search #24418

Merged

9 tasks

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 1.3.0 First shipped in 1.3.0 labels Mar 12, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(native-filters): add search all filter options#14710

feat(native-filters): add search all filter options#14710
villebro merged 12 commits into
apache:masterfrom
preset-io:villebro/select-load

villebro commented May 19, 2021 •

edited

Loading

Uh oh!

codecov Bot commented May 21, 2021 •

edited

Loading

Uh oh!

villebro May 21, 2021 •

edited

Loading

Uh oh!

villebro May 21, 2021

Uh oh!

michael-s-molina commented May 22, 2021

Uh oh!

villebro commented May 24, 2021 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

michael-s-molina commented May 24, 2021

Uh oh!

villebro May 24, 2021

Uh oh!

kgabryje May 24, 2021

Uh oh!

villebro May 24, 2021

Uh oh!

villebro May 24, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

villebro commented May 19, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

SUMMARY

SCREENSHOT

TEST PLAN

ADDITIONAL INFORMATION

Uh oh!

codecov Bot commented May 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

villebro May 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

villebro May 21, 2021

Choose a reason for hiding this comment

Uh oh!

michael-s-molina commented May 22, 2021

Uh oh!

villebro commented May 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

michael-s-molina commented May 24, 2021

Uh oh!

villebro May 24, 2021

Choose a reason for hiding this comment

Uh oh!

kgabryje May 24, 2021

Choose a reason for hiding this comment

Uh oh!

villebro May 24, 2021

Choose a reason for hiding this comment

Uh oh!

villebro May 24, 2021

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

villebro commented May 19, 2021 •

edited

Loading

codecov Bot commented May 21, 2021 •

edited

Loading

villebro May 21, 2021 •

edited

Loading

villebro commented May 24, 2021 •

edited

Loading