[druid] Updating refresh logic

[john-bodley]

Though the term "refresh" is somewhat vague from a Druid metadata perspective I sense this translates to create or update. Previously we were creating or updates Druid columns but only creating Druid metrics when the Druid metadata was synced/refreshed.

This PR ensures that refreshing is consistent for both Druid columns and metrics and specifically addresses the following:

1. Removes redundancy by only controlling metric specifications within the DruidMetric class. Previously there was somewhat duplicate logic for both the DruidColumn and DruidMetric class.
2. Renames generate_metrics with refresh_metrics to imply that we're both creating and updating.
3. Updates the SQL filters to use IN rather than a series of ORs.
4. Added metric creation for float types.
5. Adds the missing migration for creating Druid uniqueness constraints to the columns and metrics tables which were added in Adding YAML Import-Export for Datasources to CLI #3978. Note to avoid this MySQL issue the metric_name column was reduced from 512 to 255 characters.
6. Corrects the --merge options for the refresh_druid command, which should be a flag (true/false) rather than an option.
7. Note I didn't want to change the structure of get_metrics in terms of checking whether said metric already exists to ensure consistency with SQLA, hence why the merging logic is handled in refresh_metrics.

@fabianmenges I only added the missing Druid migrations however I believe there are additional migrations from your PR (#3978) which are missing for the following tables:

• table_columns
• tables

to: @mistercrunch @Mogball

[codecov-io]

Codecov Report

Merging #4655 into master will increase coverage by 0.12%.
The diff coverage is 90%.

@@            Coverage Diff             @@
##           master    #4655      +/-   ##
==========================================
+ Coverage   71.37%   71.49%   +0.12%     
==========================================
  Files         190      190              
  Lines       14918    14911       -7     
  Branches     1102     1102              
==========================================
+ Hits        10648    10661      +13     
+ Misses       4267     4247      -20     
  Partials        3        3

Impacted Files	Coverage Δ
superset/cli.py	48.38% <ø> (ø)	⬆️
superset/connectors/druid/views.py	68.02% <0%> (ø)	⬆️
superset/connectors/druid/models.py	78.85% <100%> (+2.5%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ec06967...a334220. Read the comment docs.

[fabianmenges]

Looks good to me. I'll check how the constraints for the table you mentioned are.

[fabianmenges]

Are we not testing aggregators anymore?

[john-bodley]

@fabianmenges I'm not the original author of this code so I don't have complete context, but from what I observed whilst debugging was these aggregators were not being tested. I've re-added this in case I was wrong.

[Mogball]

Running a loop to issue a query for each metric/column means that many queries have to be made, as opposed to just one to grab them all at once. If you've got tons of metrics and such, this adds up and can be reasonably slow

[john-bodley]

@Mogball that was one of my concerns and I agree with your comment. I've refactored the logic to do one query per DruidColumn.

[mistercrunch]

could go col_obj.is_num() that comes from the base column class and includes all these

[Mogball]

I mean the same could apply here as well, where it's possible to combine all of the columns' metrics into one query

[mistercrunch]

Hit an issue on this line while upgrading our staging. I wrapped the statement in a try block locally so that I could move forward

[mistercrunch]

For the record it was something to the effect of the constraint existing already

[john-bodley]

@mistercrunch so you recall if the constraint which existed previously was added manually? As far as I can tell this constraint never existed and thus the upgrade/downgrade logic should be sound.

[mistercrunch]

Can't recall creating it. Could also be that it failed half way through before or timed out and got this error the next time around... Dunno.