fix: Retroactively add granularity param to charts

[etr2460]

SUMMARY

We ran into some data accuracy issues in our environment where time range filters wouldn't apply to certain charts (like Big Number) when set on a dashboard. We determined that this was because these charts somehow didn't have a granularity param set on them, so the time range wasn't appropriately set on the chart query. This was especially dangerous, because the chart showed as applying the filter, but it wasn't actually applied to the query.

The fix is this db migration that adds the granularity param onto charts that should have one but don't.

TEST PLAN

Before the migration, see big number charts without the granularity or granularity_sqla param not filter by time range correctly.

Run the migration, and see the dashboard perform the filter appropriately.

ADDITIONAL INFORMATION

• Has associated issue:
• Changes UI
• Requires DB Migration.
• Confirm DB Migration upgrade and downgrade tested.
• Introduces new feature or API
• Removes existing feature or API

to: @john-bodley @ktmud @graceguo-supercat @villebro
cc: @junlincc

[john-bodley]

Should this mention that this mimics the behavior of the frontend?

[etr2460]

good call, done

[john-bodley]

Is this performant? I wonder if the join should be part of the slice query.

[etr2460]

This ended up only altering 150 slices in our DB (and only got to this step for about 4k) so i'm not sure performance matters that much. It's a trade off between doing the join with a much larger number of slices (200k+) vs. waiting until we get to this step. idk

[codecov-io]

Codecov Report

Merging #12960 (30660ce) into master (9982fde) will decrease coverage by 2.43%.
The diff coverage is 0.00%.

@@            Coverage Diff             @@
##           master   #12960      +/-   ##
==========================================
- Coverage   69.14%   66.70%   -2.44%     
==========================================
  Files        1025      491     -534     
  Lines       48767    28888   -19879     
  Branches     5188        0    -5188     
==========================================
- Hits        33718    19269   -14449     
+ Misses      14915     9619    -5296     
+ Partials      134        0     -134     

Flag	Coverage Δ
cypress	?
javascript	?
python	66.70% <0.00%> (-0.92%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...43f2fdb_add_granularity_to_charts_where_missing.py	0.00% <0.00%> (ø)
superset/sql_validators/postgres.py	50.00% <0.00%> (-50.00%)	⬇️
superset/views/database/views.py	62.69% <0.00%> (-24.88%)	⬇️
superset/dataframe.py	91.66% <0.00%> (-8.34%)	⬇️
superset/databases/commands/create.py	83.67% <0.00%> (-8.17%)	⬇️
superset/databases/commands/update.py	85.71% <0.00%> (-8.17%)	⬇️
superset/sql_validators/base.py	93.33% <0.00%> (-6.67%)	⬇️
superset/db_engine_specs/sqlite.py	90.62% <0.00%> (-6.25%)	⬇️
superset/db_engine_specs/base.py	79.85% <0.00%> (-6.15%)	⬇️
superset/db_engine_specs/presto.py	82.25% <0.00%> (-5.63%)	⬇️
... and 577 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9982fde...30660ce. Read the comment docs.

[villebro]

@etr2460 I noticed a similar weirdness when I was fixing a regression in the table chart. For some reason the chart looked different in the dashboard compared to the Explore view. When looking at the metadata I noticed that the chart was missing a control value that had been added to the control panel after the chart had been created. Upon closer inspection it turned out Explore merges the chart metadata on top of the default control values, but dashboard doesn't. I didn't yet have time to look into this more closely, but I believe making sure the metadata flow is the same in Dashboard and Explore view might solve this problem, potentially making the migration unnecessary.

[ktmud]

I think this is a safe migration if the query can be tuned to be more performant. Unifying form_data and control defaults merging logics between Dashboard and Explore might take a lot more time than relatively straightforward migration so I'll vote +1 on moving this forward.

[ktmud]

This query will fetch all chart slices with SQLA datasource and run JSON parse in Python. Could we filter out only those didn't have granularity or granularity_sqla instead?

Suggested change

	for slc in session.query(Slice).filter(Slice.datasource_type == "table").all():
	for slc in (
	session.query(Slice)
	.filter(and_(
	Slice.datasource_type == "table",
	not Slice.params.like('%"granularity%'))
	))
	.yield_per(500)
	):

Not that matters in practicality, but I'd also try to stream things whenever I try to fetch an unknown number of all results (.yield_per(500)).

[etr2460]

so you're saying do a plain text filter first to remove most of the slices that aren't eligible, and only do the json parse on what's remaining? It looks janky, but it should work, will update

[villebro]

I think this makes sense - no point in pulling in slices that aren't applicable

[ktmud]

Yeah, since we don't need complex regexp here so a simple text match should work.

[villebro]

@etr2460 I noticed a similar weirdness when I was fixing a regression in the table chart. For some reason the chart looked different in the dashboard compared to the Explore view. When looking at the metadata I noticed that the chart was missing a control value that had been added to the control panel after the chart had been created. Upon closer inspection it turned out Explore merges the chart metadata on top of the default control values, but dashboard doesn't. I didn't yet have time to look into this more closely, but I believe making sure the metadata flow is the same in Dashboard and Explore view might solve this problem, potentially making the migration unnecessary.

I looked into this, and it turns out default values are in fact applied on the Chart form data on the Dashboard similarly as on the Explore view. The control panel on the Table chart was just setting the value of the queryModel control (which had the default value null) while rendering the control panel based on other form data.

Going forward we should potentially make default values "smarter", by making it possible to introduce hooks that return defaults based on other context. In the case of queryMode in table model, it would be inferred from the other formData, and in the case of granularity, it could default to main_dttm_col in the dataset if missing/unset.

[villebro]

LGTM

[ktmud]

Close and reopen to trigger docker-build and python-lint (3.7):

[villebro]

FYI a migration was merged today, so I believe we need to update the downgrade revision id. #13052

[etr2460]

good call, updating now. And I agree @villebro there's some weirdness going on with passing parameters the same way between dashboard and explore. This might not be a long term fix, but more so fixes weirdness that's been cropping up currently