feat: Adds a key-value endpoint to store the state of dashboard filters by michael-s-molina · Pull Request #17536 · apache/superset

michael-s-molina · 2021-11-24T13:23:19Z

SUMMARY

Currently, the filters state is being stored in the URL to allow users to preserve their current interactions in case of a refresh and also to share specific states with other users. The most common example is when a user, after exploring a dashboard using filters, wishes to share their findings with other users.

The problem with this approach is that when the size of the state is big, it provokes URL length problems with the major browsers.

The proposed solution in this PR is to store the filters state on the server-side and keep in the URL only a secure key that gives access to the stored content. To do that, we'll extend the current dashboard endpoint with an additional path that supports the storage of information in the form of key-value entries. By making this endpoint relative to the main resource we can reuse the security model and access grants.

The additional endpoint and the supported operations are:

POST api/v1/dashboard/<id>/filter_state

BODY 
{
   // Any type of JSON supported text.
   value: <string>,
}

RESPONSE
{
   // The key generated on the server-side
   key: <string>
}

PUT api/v1/dashboard/<id>/filter_state/<key>

BODY
{
   // Any type of JSON supported text.
   value: <string>,
}

RESPONSE
{
   // The result of the operation
   message: <string>
}

GET api/v1/dashboard/<id>/filter_state/<key>

RESPONSE
{
   // Any type of JSON supported text.
   value: <string>,
}

DELETE api/v1/dashboard/<id>/filter_state/<key>

RESPONSE
{
   // The result of the operation
   message: <string>
}

The keys are generated on the server-side using the https://docs.python.org/3/library/secrets.html#secrets.token_urlsafe function and the value can be any JSON supported text.

To store the key-value entries, this PR uses Flask-Caching, which enables support to different backends like Redis, Memcached, file system, etc. It also allows the configuration of expiration times for the stored entries in order to discard the ones that are not being used anymore and optimize the required storage space.

This PR also allows a configuration to preserve actively accessed entries by renewing the expiration time every time a value is retrieved.

Follow-up of #17337

TESTING INSTRUCTIONS

1 - Execute Python tests
2 - All tests should pass

ADDITIONAL INFORMATION

Has associated issue:
Required feature flags:
Changes UI
Includes DB Migration (follow approval process in SIP-59)
- Migration is atomic, supports rollback & is backwards-compatible
- Confirm DB migration upgrade and downgrade tested
- Runtime estimates and downtime expectations provided
Introduces new feature or API
Removes existing feature or API

codecov · 2021-11-24T16:27:59Z

Codecov Report

Merging #17536 (b3841eb) into master (a959e6d) will decrease coverage by 0.29%.
The diff coverage is n/a.

❗ Current head b3841eb differs from pull request most recent head b0e5a42. Consider uploading reports for the commit b0e5a42 to get more accurate results

@@            Coverage Diff             @@
##           master   #17536      +/-   ##
==========================================
- Coverage   77.01%   76.71%   -0.30%     
==========================================
  Files        1049     1062      +13     
  Lines       56682    56997     +315     
  Branches     7851     7851              
==========================================
+ Hits        43651    43723      +72     
- Misses      12778    13021     +243     
  Partials      253      253

Flag	Coverage Δ
hive	`?`
mysql	`81.80% <ø> (-0.23%)`	⬇️
postgres	`81.81% <ø> (-0.23%)`	⬇️
presto	`?`
python	`81.89% <ø> (-0.64%)`	⬇️
sqlite	`81.49% <ø> (-0.23%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
superset/db_engines/hive.py	`0.00% <0.00%> (-85.19%)`	⬇️
superset/db_engine_specs/hive.py	`70.27% <0.00%> (-16.99%)`	⬇️
superset/db_engine_specs/presto.py	`83.50% <0.00%> (-6.48%)`	⬇️
superset/utils/cache_manager.py	`96.15% <0.00%> (-3.85%)`	⬇️
superset/views/database/mixins.py	`81.03% <0.00%> (-1.73%)`	⬇️
superset/connectors/sqla/models.py	`86.85% <0.00%> (-1.58%)`	⬇️
superset/models/core.py	`89.26% <0.00%> (-0.74%)`	⬇️
superset/connectors/base/models.py	`88.19% <0.00%> (-0.49%)`	⬇️
superset/db_engine_specs/base.py	`88.15% <0.00%> (-0.39%)`	⬇️
superset/charts/commands/export.py	`94.11% <0.00%> (-0.17%)`	⬇️
... and 20 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a959e6d...b0e5a42. Read the comment docs.

betodealmeida

Looks great!

Can you add some tests for PUT and DELETE as well?

villebro

Quick first pass, looking really good! Will review/test more tomorrow!

craig-rueda

Left a few comments around moving logic out of DAO layer and into the CMDs. Would also be great to see a few more tests, especially around the CMDs themselves.

michael-s-molina · 2021-11-29T14:51:47Z

@betodealmeida @villebro @craig-rueda Tackled all your comments. Can you review it again? Thanks

betodealmeida

This looks great! I left a few small comments, but it looks good to go, IMHO.

… endpoint name

villebro

A few comments - tested, and works really nicely, really looking forward to this!

villebro

LGTM

…rd filters (apache#17536)" This reverts commit 2f2e8fe.

mayurnewase · 2021-12-29T07:11:22Z

+from tests.integration_tests.fixtures.world_bank_dashboard import (
+    load_world_bank_dashboard_with_slices,
+)
+from tests.integration_tests.test_app import app


app is reimported here.
mypy hook should have given error in ci too as my local does.
not sure how it was skipped.(different version or ignore config or mypy doesn't run in pipeline ?)

Thanks for pointing that out @mayurnewase. I submitted a fix in #17887. @villebro Do you have any idea how pre-commit and CI missed that?

…rs (apache#17536) * feat: Adds a key-value endpoint to store the state of dashboard filters * Fixes pylint issues * Adds openapi schemas * Adds more tests, move logic to commands and use singular form for the endpoint name * Fixes model description * Removes database model * Adds open api specs * Simplifies the commands * Adds more tests * Validates the value content and submits the correct http status code * Fixes import order * Skips flakky test * Fixes tests * Updates UPDATING.md

michael-s-molina requested review from betodealmeida, craig-rueda, dpgaspar, john-bodley, rusackas and villebro November 24, 2021 13:23

pull-request-size Bot added the size/XL label Nov 24, 2021

betodealmeida reviewed Nov 24, 2021

View reviewed changes

Comment thread superset/key_value/utils.py Outdated

villebro reviewed Nov 24, 2021

View reviewed changes

Comment thread superset/config.py Outdated

Comment thread superset/key_value/api.py Outdated

Comment thread superset/key_value/schemas.py Outdated

Comment thread superset/key_value/utils.py Outdated

betodealmeida added the risk:db-migration PRs that require a DB migration label Nov 24, 2021

craig-rueda requested changes Nov 24, 2021

View reviewed changes

Comment thread superset/dashboards/filters_state/api.py Outdated

Comment thread superset/dashboards/filters_state/dao.py Outdated

Comment thread superset/dashboards/filters_state/dao.py Outdated

michael-s-molina requested review from betodealmeida, craig-rueda and villebro November 29, 2021 14:51

betodealmeida approved these changes Nov 29, 2021

View reviewed changes

Comment thread superset/config.py Outdated

Comment thread superset/dashboards/filter_state/api.py Outdated

Comment thread superset/key_value/commands/create.py Outdated

craig-rueda reviewed Nov 29, 2021

View reviewed changes

Comment thread superset/models/key_value.py Outdated

michael-s-molina added 6 commits November 29, 2021 15:48

feat: Adds a key-value endpoint to store the state of dashboard filters

8074963

Fixes pylint issues

588a016

Adds openapi schemas

2f8f429

Adds more tests, move logic to commands and use singular form for the…

d1f8bcb

… endpoint name

Fixes model description

adc0409

Removes database model

b9004e0

michael-s-molina force-pushed the add-filter-state-endpoint branch from d81faf1 to b9004e0 Compare November 29, 2021 18:49

Adds open api specs

db29e6f

craig-rueda reviewed Nov 29, 2021

View reviewed changes

Comment thread superset/key_value/commands/create.py Outdated

craig-rueda reviewed Nov 29, 2021

View reviewed changes

Comment thread superset/key_value/api.py Outdated

craig-rueda reviewed Nov 29, 2021

View reviewed changes

Comment thread superset/key_value/commands/update.py Outdated

craig-rueda reviewed Nov 29, 2021

View reviewed changes

Comment thread superset/key_value/commands/update.py Outdated

Simplifies the commands

0f44be5

craig-rueda reviewed Nov 29, 2021

View reviewed changes

Comment thread tests/integration_tests/dashboards/filter_state/api_tests.py Outdated

pull-request-size Bot added size/XXL and removed size/XL labels Nov 30, 2021

Adds more tests

7c76172

michael-s-molina force-pushed the add-filter-state-endpoint branch from f2b70e5 to 7c76172 Compare November 30, 2021 16:06

villebro reviewed Nov 30, 2021

View reviewed changes

Comment thread superset/config.py Outdated

Comment thread superset/dashboards/filter_state/api.py Outdated

Comment thread superset/key_value/api.py

craig-rueda approved these changes Nov 30, 2021

View reviewed changes

Validates the value content and submits the correct http status code

4af801a

pull-request-size Bot added size/XL and removed size/XXL labels Nov 30, 2021

Fixes import order

458d468

michael-s-molina force-pushed the add-filter-state-endpoint branch from 06d3836 to 458d468 Compare November 30, 2021 18:07

pull-request-size Bot added size/XXL and removed size/XL labels Nov 30, 2021

michael-s-molina added 2 commits November 30, 2021 15:24

Skips flakky test

01249af

Fixes tests

ff22b33

villebro approved these changes Dec 1, 2021

View reviewed changes

Updates UPDATING.md

b0e5a42

michael-s-molina merged commit 2f2e8fe into apache:master Dec 1, 2021

This was referenced Dec 2, 2021

Lots of filters leads to too large url #17086

Closed

[native filters] Include native filter values in tabbed dashboard link #17209

Closed

Dashboard Share URL excludes Experimental Filters and Cross Filters values #16580

Closed

pkdotson added a commit to preset-io/superset that referenced this pull request Dec 7, 2021

Revert "feat: Adds a key-value endpoint to store the state of dashboa…

16c6d13

…rd filters (apache#17536)" This reverts commit 2f2e8fe.

mayurnewase reviewed Dec 29, 2021

View reviewed changes

michael-s-molina mentioned this pull request Feb 16, 2022

fix: Explore long URL problem #18181

Merged

9 tasks

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 1.5.0 First shipped in 1.5.0 labels Mar 13, 2024

Conversation

michael-s-molina commented Nov 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

SUMMARY

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

Uh oh!

codecov Bot commented Nov 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

betodealmeida left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

villebro left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

craig-rueda left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

michael-s-molina commented Nov 29, 2021

Uh oh!

betodealmeida left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

villebro left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

villebro left a comment

Choose a reason for hiding this comment

Uh oh!

mayurnewase Dec 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

michael-s-molina Dec 29, 2021

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

michael-s-molina commented Nov 24, 2021 •

edited

Loading

codecov Bot commented Nov 24, 2021 •

edited

Loading

mayurnewase Dec 29, 2021 •

edited

Loading