chore(cache): default to SimpleCache in debug mode by villebro · Pull Request #18976 · apache/superset

villebro · 2022-03-01T08:54:30Z

SUMMARY

Changes:

Default FILTER_STATE_CACHE_CONFIG and EXPLORE_FORM_DATA_CACHE_CONFIG to SimpleCache when app is in debug mode.
Replace old CACHE_TYPE references to the deprecated null type with NullCache and others (see: https://flask-caching.readthedocs.io/en/latest/#configuring-flask-caching)
Clean up CacheConfig type that had a remnant from [Caching] Ensure cache is always created #9020 where a callable CacheConfig was supported, but was removed in chore: Using cache factory method #10887. TL;DR: now only Dict[str, Any] types are supported.
Raise an exception if trying to start up the application with a NullCache backend for required caches (Explore or Dashboard)
Update docs

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

Has associated issue:
Required feature flags:
Changes UI
Includes DB Migration (follow approval process in SIP-59)
- Migration is atomic, supports rollback & is backwards-compatible
- Confirm DB migration upgrade and downgrade tested
- Runtime estimates and downtime expectations provided
Introduces new feature or API
Removes existing feature or API

codecov · 2022-03-01T09:16:44Z

Codecov Report

Merging #18976 (4a6a9e0) into master (bd63a1b) will increase coverage by 0.00%.
The diff coverage is 88.88%.

❗ Current head 4a6a9e0 differs from pull request most recent head d5f8d42. Consider uploading reports for the commit d5f8d42 to get more accurate results

@@           Coverage Diff           @@
##           master   #18976   +/-   ##
=======================================
  Coverage   66.58%   66.58%           
=======================================
  Files        1641     1641           
  Lines       63533    63548   +15     
  Branches     6424     6424           
=======================================
+ Hits        42303    42315   +12     
- Misses      19551    19554    +3     
  Partials     1679     1679

Flag	Coverage Δ
hive	`52.61% <88.88%> (+0.01%)`	⬆️
mysql	`81.83% <88.88%> (-0.01%)`	⬇️
postgres	`81.88% <88.88%> (-0.01%)`	⬇️
presto	`52.45% <88.88%> (+0.01%)`	⬆️
python	`82.31% <88.88%> (-0.01%)`	⬇️
sqlite	`81.57% <88.88%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
superset/utils/cache_manager.py	`93.75% <86.36%> (-6.25%)`	⬇️
superset/config.py	`91.92% <100.00%> (ø)`
superset/typing.py	`97.95% <100.00%> (-0.09%)`	⬇️
superset/utils/cache.py	`74.52% <100.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update bd63a1b...d5f8d42. Read the comment docs.

michael-s-molina

This is awesome. Thanks for the improvements. Left a comment.

michael-s-molina · 2022-03-02T16:25:04Z

+        if app.debug and cache_type is None:
+            cache_threshold = cache_config.get("CACHE_THRESHOLD", math.inf)
+            cache_config.update(
+                {"CACHE_TYPE": "SimpleCache", "CACHE_THRESHOLD": cache_threshold,}


Suggested change

{"CACHE_TYPE": "SimpleCache", "CACHE_THRESHOLD": cache_threshold,}

{"CACHE_TYPE": "SimpleCache", "CACHE_THRESHOLD": cache_threshold}

I'm surprised this is not catched by black.

Yeah this has been annoying me for a long time. I'm sure it's been fixed in a new version of black, would be nice to take a stab at updating it.

michael-s-molina · 2022-03-02T16:27:43Z

+        self._init_cache(app, self._cache, "CACHE_CONFIG")
+        self._init_cache(app, self._data_cache, "DATA_CACHE_CONFIG")
+        self._init_cache(app, self._thumbnail_cache, "THUMBNAIL_CACHE_CONFIG")
+        self._init_cache(
+            app, self._filter_state_cache, "FILTER_STATE_CACHE_CONFIG", required=True


Not required but it would be a good idea to extract these strings into constants to avoid typing errors.

Good idea - If I need to push changes to this PR I'll add that

ktmud · 2022-03-02T16:29:08Z


 ### Breaking Changes

+- [18976](https://github.com/apache/superset/pull/18976): When running the app in debug mode, the app will default to use `SimpleCache` for `FILTER_STATE_CACHE_CONFIG` and `EXPLORE_FORM_DATA_CACHE_CONFIG`. When running in non-debug mode, a cache backend will need to be defined, otherwise the application will fail to start. For installations using Redis or other caching backends, it is recommended to use the same backend for both cache configs.


For installations using Redis or other caching backends, it is recommended to use the same backend for both cache configs.

I'm wondering if there is a real need in configuring different cache expiration for both cache storages? If not, maybe we should just merge them---into something like SEMI_PERSISTENT_CACHE_CONFIG (albeit it'd be a bigger breaking change)? I don't think it matters if we save explore state cache for 23 days more.

@ktmud I think the jury is still out on a few aspects of this feature:

will people predominantly use direct URLs or permalinks for sharing

what's the sweet spot timeout for these caches

will we ever need multiple key-value states on a resource, or can we just collapse them into one key-value pair per resource

etc. I'm happy to keep iterating on this, and when we do a 3.0 release, I think it would be a perfect time to clean up some of the cruft we've accumulated here.

michael-s-molina

I really liked how this PR progressed with the suggestions. It's clean and well documented. Thanks for all the hard work @villebro 👏🏼

* chore(cache): default to SimpleCache in debug mode * lint * clean up type * use util * fix integration test cache configs * remove util from cache manager * remove trailing comma * fix more tests * fix truthiness check * fix tests and improve deprecation notice * fix default cache threshold * move debug check to cache_manager * remove separate getter * update docs * remove default cache config (cherry picked from commit a04f1d4)

* chore(cache): default to SimpleCache in debug mode * lint * clean up type * use util * fix integration test cache configs * remove util from cache manager * remove trailing comma * fix more tests * fix truthiness check * fix tests and improve deprecation notice * fix default cache threshold * move debug check to cache_manager * remove separate getter * update docs * remove default cache config

superset-github-bot Bot added the preset-io label Mar 1, 2022

pull-request-size Bot added the size/M label Mar 1, 2022

villebro force-pushed the villebro/default-cache branch from 5186888 to bab168a Compare March 1, 2022 09:07

villebro force-pushed the villebro/default-cache branch from bab168a to 7b1926e Compare March 1, 2022 10:34

pull-request-size Bot added size/L and removed size/M labels Mar 1, 2022

villebro force-pushed the villebro/default-cache branch 2 times, most recently from 62cd84f to 8b71ba1 Compare March 1, 2022 13:21

villebro commented Mar 1, 2022

View reviewed changes

Comment thread superset/typing.py Outdated

villebro changed the title ~~[WIP] chore(cache): default to SimpleCache in debug mode~~ chore(cache): default to SimpleCache in debug mode Mar 1, 2022

villebro requested a review from michael-s-molina March 1, 2022 13:40

villebro mentioned this pull request Mar 1, 2022

fix: Explore long URL problem #18181

Merged

9 tasks

michael-s-molina reviewed Mar 1, 2022

View reviewed changes

Comment thread superset/utils/cache_manager.py Outdated

Comment thread superset/config.py Outdated

john-bodley approved these changes Mar 1, 2022

View reviewed changes

Comment thread superset/config.py Outdated

Comment thread superset/utils/cache.py Outdated

villebro force-pushed the villebro/default-cache branch from a976056 to c7b7c2e Compare March 1, 2022 21:13

villebro commented Mar 1, 2022

View reviewed changes

Comment thread superset/config.py Outdated

ktmud reviewed Mar 1, 2022

View reviewed changes

Comment thread superset/config.py Outdated

villebro added 12 commits March 2, 2022 07:46

chore(cache): default to SimpleCache in debug mode

1214e7c

lint

8044865

clean up type

507d579

use util

f0f9065

fix integration test cache configs

fdacbaf

remove util from cache manager

b953fd4

remove trailing comma

bcf9849

fix more tests

4101407

fix truthiness check

d1a0ad1

fix tests and improve deprecation notice

e5d3a11

fix default cache threshold

9a92f9a

move debug check to cache_manager

d9c8639

remove separate getter

dd77329

villebro force-pushed the villebro/default-cache branch from c7b7c2e to 05f0c80 Compare March 2, 2022 10:55

update docs

9c5f209

villebro force-pushed the villebro/default-cache branch from 05f0c80 to 9c5f209 Compare March 2, 2022 11:01

michael-s-molina reviewed Mar 2, 2022

View reviewed changes

Comment thread docs/docs/installation/cache.mdx Outdated

villebro force-pushed the villebro/default-cache branch from 61e9a11 to 501fb98 Compare March 2, 2022 13:51

villebro commented Mar 2, 2022

View reviewed changes

Comment thread tests/integration_tests/cache_tests.py Outdated

villebro force-pushed the villebro/default-cache branch from 501fb98 to d44a85d Compare March 2, 2022 13:59

remove default cache config

d5f8d42

villebro force-pushed the villebro/default-cache branch from d44a85d to d5f8d42 Compare March 2, 2022 15:38

michael-s-molina reviewed Mar 2, 2022

View reviewed changes

ktmud reviewed Mar 2, 2022

View reviewed changes

michael-s-molina approved these changes Mar 2, 2022

View reviewed changes

ktmud approved these changes Mar 2, 2022

View reviewed changes

villebro merged commit a04f1d4 into apache:master Mar 2, 2022

villebro deleted the villebro/default-cache branch March 2, 2022 19:53

michael-s-molina mentioned this pull request Mar 14, 2022

feat: improve color consistency #19038

Merged

9 tasks

villebro added the lts-v1 label Mar 29, 2022

	{"CACHE_TYPE": "SimpleCache", "CACHE_THRESHOLD": cache_threshold,}
	{"CACHE_TYPE": "SimpleCache", "CACHE_THRESHOLD": cache_threshold}


		### Breaking Changes

		- [18976](https://github.com/apache/superset/pull/18976): When running the app in debug mode, the app will default to use `SimpleCache` for `FILTER_STATE_CACHE_CONFIG` and `EXPLORE_FORM_DATA_CACHE_CONFIG`. When running in non-debug mode, a cache backend will need to be defined, otherwise the application will fail to start. For installations using Redis or other caching backends, it is recommended to use the same backend for both cache configs.

Conversation

villebro commented Mar 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

SUMMARY

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

ADDITIONAL INFORMATION

Uh oh!

codecov Bot commented Mar 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

michael-s-molina left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

michael-s-molina Mar 2, 2022

Choose a reason for hiding this comment

Uh oh!

ktmud Mar 2, 2022

Choose a reason for hiding this comment

Uh oh!

villebro Mar 2, 2022

Choose a reason for hiding this comment

Uh oh!

michael-s-molina Mar 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

villebro Mar 2, 2022

Choose a reason for hiding this comment

Uh oh!

ktmud Mar 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

villebro Mar 2, 2022

Choose a reason for hiding this comment

Uh oh!

michael-s-molina left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

villebro commented Mar 1, 2022 •

edited

Loading

codecov Bot commented Mar 1, 2022 •

edited

Loading

michael-s-molina Mar 2, 2022 •

edited

Loading

ktmud Mar 2, 2022 •

edited

Loading