perf: memoize db_engine_spec in database

[villebro]

SUMMARY

A recent PR #14547 introduced a performance regression causing dataset metadata fetching to become very slow for datasets with large numbers of columns. I originally thought the type regexes were the problem, but when researching the problem more closely it turns out that just referencing self.table.database.db_engine_spec in a TableColumn instance cost ~6ms on my local machine. Multiply that by 1000 columns ~= 6000 ms. To get around this I added memoization to the semi-expensive regex, but also added memoizing for Database.db_engine_spec. This should also speed up query rendering a bit, as there was similar logic there.

BEFORE #14547 (pre-regression)

For the World Bank dataset (328 cols), fetching the data took slightly less than 180ms before on my local machine (including the unnecessary 20 ms redirect):

CURRENT (master)

For the same dataset, retrieval of data now takes ~10s!

AFTER

Retrieval is now slightly quicker than originally (including no redirect):

TEST PLAN

ADDITIONAL INFORMATION

• Has associated issue:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[codecov]

Codecov Report

Merging #14638 (06f90f6) into master (e4d2424) will decrease coverage by 0.09%.
The diff coverage is 93.10%.

@@            Coverage Diff             @@
##           master   #14638      +/-   ##
==========================================
- Coverage   77.47%   77.38%   -0.10%     
==========================================
  Files         958      958              
  Lines       48486    48480       -6     
  Branches     5679     5683       +4     
==========================================
- Hits        37565    37514      -51     
- Misses      10721    10766      +45     
  Partials      200      200              

Flag	Coverage Δ
hive	80.94% <92.85%> (-0.03%)	⬇️
javascript	72.52% <100.00%> (+<0.01%)	⬆️
mysql	81.21% <92.85%> (-0.03%)	⬇️
postgres	81.23% <92.85%> (-0.03%)	⬇️
presto	?
python	81.60% <92.85%> (-0.19%)	⬇️
sqlite	80.85% <92.85%> (-0.03%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
superset/connectors/sqla/models.py	88.64% <92.00%> (-1.62%)	⬇️
...-frontend/src/datasource/ChangeDatasourceModal.tsx	90.90% <100.00%> (+3.89%)	⬆️
superset/db_engine_specs/base.py	88.45% <100.00%> (+0.02%)	⬆️
superset/models/core.py	89.40% <100.00%> (+0.27%)	⬆️
superset/db_engine_specs/presto.py	84.42% <0.00%> (-5.90%)	⬇️
superset/connectors/base/models.py	88.03% <0.00%> (-2.66%)	⬇️
superset-frontend/src/components/Tabs/Tabs.tsx	96.55% <0.00%> (-0.33%)	⬇️
...components/DashboardBuilder/DashboardContainer.tsx	100.00% <0.00%> (ø)
... and 1 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e4d2424...06f90f6. Read the comment docs.

[kgabryje]

Can we remove the cypress timeout overrides introduced in the other PR?

[villebro]

Can we remove the cypress timeout overrides introduced in the other PR?

Good idea; I'll do that. I've also been looking into introducing perf tests on the backend to identify these easier (will be a follow-up PR).

[dpgaspar]

consider renaming to get_generic_type gives an head warning that it will do some computation around it

[villebro]

This is added to the column payload in the dataset request to complement the existing type field, so I think we need to keep it as a property.