chore(sqllab): Cleanup /tables/... endpoint

[john-bodley]

SUMMARY

The /tables/... endpoint was augmented (#1466) when SQL Lab supported:

1. Fetching tables/views across either a single or all schemas, i.e., the SEE TABLE SCHEMA functioned (if configured) without having a schema specified.
2. Backend filtering.

however the endpoint is now always called with a specified schema and filtering (substring table name matching) is never invoked.

Given these observations this PR refactors the endpoint—by removing unnecessary logic and providing a more performant ORM query given that all table/views are from a specific schema—and reduces the size of the response payload to improve performance (especially for schemas with a large number of tables/views).

Additionally as a byproduct of the refactor this PR removes:

1. The (now unused) get_all_table_names_in_database and get_all_view_names_in_database utility methods as well as the get_all_datasource_names DB engine spec method.
2. The (now unused) dbs.allow_multi_schema_metadata_fetch column
3. The non-functional MAX_TABLE_NAMES config key, i.e., the logic (which potentially was wrong) only capped the number of tables/views if and only if a filter was defined, however this was never invoked.
4. The superset update-datasources-cache CLI command.
5. Unnecessary keys in the response payload (to reduce the payload size).

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

Updated unit tests and manually tested.

ADDITIONAL INFORMATION

• Has associated issue:
• Required feature flags:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

cc: @justinpark

[ktmud]

LGTM with a couple of questions

[ktmud]

Should the default be false? I think enabling multi schema fetch for all data sources is a bigger risk than disabling it for all.

[ktmud]

Since we are here, can we change force_refresh to a search params, too?

[codecov]

Codecov Report

Merging #21284 (f44908c) into master (094400c) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master   #21284   +/-   ##
=======================================
  Coverage   66.55%   66.56%           
=======================================
  Files        1791     1791           
  Lines       68591    68497   -94     
  Branches     7319     7319           
=======================================
- Hits        45651    45595   -56     
+ Misses      21049    21011   -38     
  Partials     1891     1891           

Flag	Coverage Δ
hive	53.03% <35.71%> (+0.09%)	⬆️
javascript	52.73% <100.00%> (ø)
mysql	80.84% <100.00%> (+0.06%)	⬆️
postgres	80.89% <100.00%> (+0.06%)	⬆️
presto	52.93% <35.71%> (+0.09%)	⬆️
python	81.36% <100.00%> (+0.06%)	⬆️
sqlite	79.43% <100.00%> (+0.05%)	⬆️
unit	51.02% <35.71%> (+0.06%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...frontend/src/components/DatabaseSelector/index.tsx	95.23% <ø> (ø)
.../CRUD/data/database/DatabaseModal/ExtraOptions.tsx	83.33% <ø> (ø)
superset/cli/update.py	0.00% <ø> (ø)
superset/config.py	91.58% <ø> (-0.03%)	⬇️
superset/dashboards/schemas.py	99.46% <ø> (-0.01%)	⬇️
superset/databases/api.py	95.49% <ø> (ø)
superset/db_engine_specs/base.py	89.56% <ø> (+1.23%)	⬆️
superset/db_engine_specs/hive.py	87.35% <ø> (+0.24%)	⬆️
superset/db_engine_specs/presto.py	87.78% <ø> (-0.18%)	⬇️
superset/db_engine_specs/sqlite.py	96.42% <ø> (-0.87%)	⬇️
... and 6 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[justinpark]

@john-bodley please mark Removes existing feature or API since this commit removes allow_multi_schema_metadata_fetch (Allow Multi Schema Metadata Fetch) from Database option.

[justinpark]

LGTM

[EugeneTorap]

Hi @john-bodley, the PR reproduces this tables loading issue in master branch. Can you fix it in next PR?
Can we also cover the case with a unit test?

[betodealmeida]

@john-bodley It looks like this broke loading tables in SQL Lab... I'm getting 404 now when it tries to load the list.

Just saw your fix.

[john-bodley]

Apologies this PR did not adhere to SIP-15, i.e., the database migration should have been added in a future PR and included in the subsequent major release.

[rusackas]

@john-bodley someone was just pointing out that this PR broke a sync job. I think technically since this changes the endpoints/URIs of the API, this would be considered a breaking change, namely as things move from

@expose("/tables/<int:db_id>/<schema>/<substr>/")
@expose("/tables/<int:db_id>/<schema>/<substr>/<force_refresh>/")
@expose("/tables/<int:db_id>/<schema>/<substr>/<force_refresh>/<exact_match>")

to

@expose("/tables/<int:db_id>/<schema>/")
@expose("/tables/<int:db_id>/<schema>/<force_refresh>/")
@deprecated(new_target="api/v1/database/<int:pk>/tables/")

I'm not sure if there's a way to patch this to be non-breaking at this point, but maybe adding a breaking change notice to UPDATING.md would at least raise awareness.