Skip to content

Celery task for warming up cache#7148

Merged
xtinec merged 50 commits into
apache:lyftgafrom
lyft:VIZ-386
Apr 3, 2019
Merged

Celery task for warming up cache#7148
xtinec merged 50 commits into
apache:lyftgafrom
lyft:VIZ-386

Conversation

@betodealmeida
Copy link
Copy Markdown
Member

We have a REST endpoint for warming up caches — let's use it! 😄

I wrote a celery task that periodically hits up the endpoint, based on different strategies. I implemented:

  1. A "dummy" strategy that warms up all charts.
  2. A "top-n dashboards", that warms up charts in the top-n dashboards.
  3. A strategy that warms up charts with a given tag, or charts in dashboards with a given tag.

More strategies can be easily added.

khtruong and others added 28 commits March 5, 2019 11:11
* Exclude venv for python linter to ignore

* Fix NaN error
This PR sets the background-color css property on `.ace_scroller` instead of `.ace_content` to prevent the white background shown during resizing of the SQL editor before drag ends.
* added more functionalities for query context and object.

* fixed cache logic

* added default value for groupby

* updated comments and removed print

(cherry picked from commit d5b9795)
* Add ability to change font sizes in Big Number

* rename big number to header

* Add comment to clarify font size values
…g. (apache#7037)

* Update datasource.py

* Update datasource.py

(cherry picked from commit c771625)
* Adding extraOverrides to line chart

* Updating extraOverrides to fit with more cases

* Moving extraOverrides to index.js

* Removing webpack-merge in package.json

* Fixing metrics control clearing metric

(cherry picked from commit e619405)
* Fix filter_box migration PR apache#6523

* Fix druid-related bug

(cherry picked from commit b210742)
This PR includes the following layout and css tweaks:
- Using flex to layout the north and south sub panes of query pane so resizing works properly in both Chrome and Firefox
- Removal of necessary wrapper divs and tweaking of css in sql lab so we can scroll to the bottom of both the table list and the results pane
- Make sql lab's content not overflow vertically and layout the query result area to eliminate double scroll bars
- css tweaks on the basic.html page so the loading animation appears in the center of the page across the board

(cherry picked from commit 71f1bbd)
(cherry picked from commit e83a07d)
Package maintainers should really never delete packages, but it appears
this happened with croniter and resulted in breaking our builds.

This PR bumps to a more recent existing version of the library

(cherry picked from commit 215ed39)
@kristw kristw added the enhancement:request Enhancement request submitted by anyone from the community label Mar 28, 2019
@xtinec xtinec added the lyft Related to Lyft label Mar 28, 2019
@xtinec xtinec force-pushed the lyftga branch 3 times, most recently from a15703f to 62cfb2d Compare April 2, 2019 05:04
khtruong and others added 13 commits April 2, 2019 11:02
* Exclude venv for python linter to ignore

* Fix NaN error
This PR includes the following layout and css tweaks:
- Using flex to layout the north and south sub panes of query pane so resizing works properly in both Chrome and Firefox
- Removal of necessary wrapper divs and tweaking of css in sql lab so we can scroll to the bottom of both the table list and the results pane
- Make sql lab's content not overflow vertically and layout the query result area to eliminate double scroll bars
- css tweaks on the basic.html page so the loading animation appears in the center of the page across the board

(cherry picked from commit 71f1bbd)
@codecov-io
Copy link
Copy Markdown

Codecov Report

Merging #7148 into lyftga will increase coverage by 0.06%.
The diff coverage is 76.92%.

Impacted file tree graph

@@            Coverage Diff             @@
##           lyftga    #7148      +/-   ##
==========================================
+ Coverage    64.5%   64.57%   +0.06%     
==========================================
  Files         424      425       +1     
  Lines       20668    20785     +117     
  Branches     2268     2268              
==========================================
+ Hits        13332    13422      +90     
- Misses       7213     7240      +27     
  Partials      123      123
Impacted Files Coverage Δ
superset/tasks/__init__.py 100% <100%> (ø) ⬆️
superset/tasks/cache.py 76.72% <76.72%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 62cfb2d...1688226. Read the comment docs.

Copy link
Copy Markdown

@graceguo-supercat graceguo-supercat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@graceguo-supercat
Copy link
Copy Markdown

thanks for the work! Let's see how it works and adding more improvements.

@xtinec xtinec merged commit 4ab89db into apache:lyftga Apr 3, 2019
@mistercrunch mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 0.34.0 First shipped in 0.34.0 labels Feb 28, 2024
qfcwell pushed a commit to qfcwell/superset that referenced this pull request May 12, 2026
* Sparkline dates aren't formatting in Time Series Table (apache#6976)

* Exclude venv for python linter to ignore

* Fix NaN error

* Fix the white background shown in SQL editor on drag (apache#7021)

This PR sets the background-color css property on `.ace_scroller` instead of `.ace_content` to prevent the white background shown during resizing of the SQL editor before drag ends.

* Show tooltip with time frame (apache#6979)

* Fix time filter control (apache#6978)

* Enhancement of query context and object. (apache#6962)

* added more functionalities for query context and object.

* fixed cache logic

* added default value for groupby

* updated comments and removed print

(cherry picked from commit 326708e)

* [fix] /superset/slice/id url is too long (apache#6989)


(cherry picked from commit e0fea40)

* [WIP] fix user specified JSON metadata not updating dashboard on refresh (apache#7027)


(cherry picked from commit 77d114a)

* feat: add ability to change font size in big number (apache#7003)

* Add ability to change font sizes in Big Number

* rename big number to header

* Add comment to clarify font size values

* Allow LIMIT to be specified in parameters (apache#7052)

* [fix] Cursor jumping when editing chart and dashboard titles (apache#7038)


(cherry picked from commit c697955)

* Changing time table viz to pass formatTime a date (apache#7020)

(cherry picked from commit ada7626)

* [db-engine-spec] Aligning Hive/Presto partition logic (apache#7007)


(cherry picked from commit fafff47)

* [fix] explore chart from dashboard missed slice title (apache#7046)


(cherry picked from commit 908c608)

* fix inaccurate data calculation with adata rolling and contribution (apache#7035)


(cherry picked from commit 7a30ad4)

* Adding warning message for sqllab save query (apache#7028)


(cherry picked from commit ea2ea16)

* [datasource] Ensuring consistent behavior of datasource editing/saving. (apache#7037)

* Update datasource.py

* Update datasource.py

(cherry picked from commit 65a6e40)

* [csv-upload] Fixing message encoding (apache#6971)


(cherry picked from commit 574e213)

* [sql-parse] Fixing LIMIT exceptions (apache#6963)


(cherry picked from commit bf90829)

* Adding custom control overrides (apache#6956)

* Adding extraOverrides to line chart

* Updating extraOverrides to fit with more cases

* Moving extraOverrides to index.js

* Removing webpack-merge in package.json

* Fixing metrics control clearing metric

(cherry picked from commit eb603c7)

* [sqlparse] Fixing table name extraction for ill-defined query (apache#7029)


(cherry picked from commit d0d9cba)

* [missing values] Removing replacing missing values (apache#4905)


(cherry picked from commit ed93b4c)

* [SQL Lab] Improved query and results tabs rendering reliability (apache#7082)

closes apache#7080

(cherry picked from commit 7ee4b18)

* Fix filter_box migration PR apache#6523 (apache#7066)

* Fix filter_box migration PR apache#6523

* Fix druid-related bug

(cherry picked from commit 7063e6c)

* SQL editor layout makeover (apache#7102)

This PR includes the following layout and css tweaks:
- Using flex to layout the north and south sub panes of query pane so resizing works properly in both Chrome and Firefox
- Removal of necessary wrapper divs and tweaking of css in sql lab so we can scroll to the bottom of both the table list and the results pane
- Make sql lab's content not overflow vertically and layout the query result area to eliminate double scroll bars
- css tweaks on the basic.html page so the loading animation appears in the center of the page across the board

(cherry picked from commit 62c1a8d)

* [forms] Fix handling of NULLs

(cherry picked from commit e83a07d)

* handle null column_name in sqla and druid models

(cherry picked from commit 2ff721a)

* Use metric name instead of metric in filter box (apache#7106)


(cherry picked from commit 542125b)

* Bump python lib croniter to an existing version (apache#7132)

Package maintainers should really never delete packages, but it appears
this happened with croniter and resulted in breaking our builds.

This PR bumps to a more recent existing version of the library

(cherry picked from commit d7b90c4)

* Revert PR apache#6933 (apache#7162)

* Celery worker for warming up cache

* Remove testing changes

* Add documentation

* Fix lint

* WIP dashboard filters

* Use new cache so it works with dashboards

* Add more unit tests, fix old ones

* Fix flake8 and docs

* Sparkline dates aren't formatting in Time Series Table (apache#6976)

* Exclude venv for python linter to ignore

* Fix NaN error

* Changing time table viz to pass formatTime a date (apache#7020)

(cherry picked from commit ada7626)

* SQL editor layout makeover (apache#7102)

This PR includes the following layout and css tweaks:
- Using flex to layout the north and south sub panes of query pane so resizing works properly in both Chrome and Firefox
- Removal of necessary wrapper divs and tweaking of css in sql lab so we can scroll to the bottom of both the table list and the results pane
- Make sql lab's content not overflow vertically and layout the query result area to eliminate double scroll bars
- css tweaks on the basic.html page so the loading animation appears in the center of the page across the board

(cherry picked from commit 62c1a8d)

* Celery worker for warming up cache

* Remove testing changes

* Add documentation

* Fix lint

* WIP dashboard filters

* Use new cache so it works with dashboards

* Add more unit tests, fix old ones

* Fix flake8 and docs

* Fix bad merge and pylint
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels enhancement:request Enhancement request submitted by anyone from the community lyft Related to Lyft 🚢 0.34.0 First shipped in 0.34.0

Projects

None yet

Development

Successfully merging this pull request may close these issues.