chore: split CLI into multiple files

[betodealmeida]

SUMMARY

This PR splits the superset/cli.py file into multiple files.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

Before:

% superset --help
Loaded your LOCAL configuration at [/Users/beto/Projects/incubator-superset/superset_config.py]
--------------------------------------------------------------------------------
                                    WARNING
--------------------------------------------------------------------------------
A Default SECRET_KEY was detected, please use superset_config.py to override it.
Use a strong complex alphanumeric string and use a tool to help you generate
a sufficiently random sequence, ex: openssl rand -base64 42
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
logging was configured successfully
2022-01-18 17:06:25,085:INFO:superset.utils.logging_configurator:logging was configured successfully
2022-01-18 17:06:25,092:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'>
2022-01-18 17:06:26,509:WARNING:root:ENABLE_SCHEDULED_EMAIL_REPORTS is deprecated and will be removed in version 2.0.0
2022-01-18 17:06:26,523:WARNING:root:ENABLE_ALERTS is deprecated and will be removed in version 2.0.0
Usage: superset [OPTIONS] COMMAND [ARGS]...

  This is a management script for the Superset application.

Options:
  --version  Show the flask version
  --help     Show this message and exit.

Commands:
  alert                     Run the alert scheduler loop
  compute-thumbnails        Compute thumbnails
  db                        Perform database migrations.
  export-dashboards         Export dashboards to ZIP file
  export-datasources        Export datasources to ZIP file
  fab                       FAB flask group commands
  flower                    Runs a Celery Flower web server Celery Flower
                            is...

  import-dashboards         Import dashboards from ZIP file
  import-datasources        Import datasources from ZIP file
  import-directory          Imports configs from a given directory
  init                      Inits the Superset application
  load-examples             Loads a set of Slices and Dashboards and a...
  load-test-users           Loads admin, alpha, and gamma user for testing...
  re-encrypt-secrets
  refresh-druid             Refresh druid datasources
  routes                    Show the routes for the app.
  run                       Run a development server.
  set-database-uri          Updates a database connection URI
  shell                     Run a shell in the app context.
  sync-tags                 Rebuilds special tags (owner, type, favorited...
  update-api-docs           Regenerate the openapi.json file in docs
  update-datasources-cache  Refresh sqllab datasources cache
  version                   Prints the current version number
  worker                    Starts a Superset worker for async SQL query...

After:

% superset --help
Loaded your LOCAL configuration at [/Users/beto/Projects/incubator-superset/superset_config.py]
--------------------------------------------------------------------------------
                                    WARNING
--------------------------------------------------------------------------------
A Default SECRET_KEY was detected, please use superset_config.py to override it.
Use a strong complex alphanumeric string and use a tool to help you generate
a sufficiently random sequence, ex: openssl rand -base64 42
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
logging was configured successfully
2022-01-18 17:05:24,845:INFO:superset.utils.logging_configurator:logging was configured successfully
2022-01-18 17:05:24,855:INFO:root:Configured event logger of type <class 'superset.utils.log.DBEventLogger'>
2022-01-18 17:05:26,370:WARNING:root:ENABLE_SCHEDULED_EMAIL_REPORTS is deprecated and will be removed in version 2.0.0
2022-01-18 17:05:26,385:WARNING:root:ENABLE_ALERTS is deprecated and will be removed in version 2.0.0
Usage: superset [OPTIONS] COMMAND [ARGS]...

  This is a management script for the Superset application.

Options:
  --version  Show the flask version
  --help     Show this message and exit.

Commands:
  alert                     Run the alert scheduler loop
  compute-thumbnails        Compute thumbnails
  db                        Perform database migrations.
  export-dashboards         Export dashboards to ZIP file
  export-datasources        Export datasources to ZIP file
  fab                       FAB flask group commands
  flower                    Runs a Celery Flower web server Celery Flower
                            is...

  import-dashboards         Import dashboards from ZIP file
  import-datasources        Import datasources from ZIP file
  import-directory          Imports configs from a given directory
  init                      Inits the Superset application
  load-examples             Loads a set of Slices and Dashboards and a...
  load-test-users           Loads admin, alpha, and gamma user for testing...
  re-encrypt-secrets
  refresh-druid             Refresh druid datasources
  routes                    Show the routes for the app.
  run                       Run a development server.
  set-database-uri          Updates a database connection URI
  shell                     Run a shell in the app context.
  sync-tags                 Rebuilds special tags (owner, type, favorited...
  update-api-docs           Regenerate the openapi.json file in docs
  update-datasources-cache  Refresh sqllab datasources cache
  version                   Prints the current version number
  worker                    Starts a Superset worker for async SQL query...

TESTING INSTRUCTIONS

Nothing should have changed, and all superset commands should run as before.

I tested init, db upgrade, and version.

ADDITIONAL INFORMATION

• Has associated issue:
• Required feature flags:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[dpgaspar]

kind of unfortunate we have to do this, since it's slightly harder to maintain. What's was the problem?

[betodealmeida]

At this point the file is still manageable, but I'm planning to add more commands in the next months (for syncing to/from DBT and other metric stores) and I'm worried it's going to get too big.

I followed the pattern described in the docs (https://click.palletsprojects.com/en/6.x/quickstart/#nesting-commands). Not sure if there's a better way of doing this that doesn't require touching this file when adding a new command. :-/

[dpgaspar]

previously we only needed to add has a decorator superset.command. No need to issue superset.add_command(<Callable>)

[betodealmeida]

Right, but I don't think that would work here.

For example, in cli/celery.py we could try this:

from superset.cli.main import superset

@superset.command()  # instead of @click.command
def some_command():
    ...

But the command wouldn't be registered until we explicitly import the cli.celery module. So we'd have to add this to cli/main.py:

from superset.cli import celery

So we'd have an import that is needed because of an implicit side effect, which IMHO is worse than explicitly having to register everything.

[betodealmeida]

@dpgaspar I was able to load them dynamically.

[codecov]

Codecov Report

Merging #18082 (6fc34e6) into master (810cfc1) will decrease coverage by 0.18%.
The diff coverage is 36.03%.

@@            Coverage Diff             @@
##           master   #18082      +/-   ##
==========================================
- Coverage   66.01%   65.83%   -0.19%     
==========================================
  Files        1570     1577       +7     
  Lines       61775    61828      +53     
  Branches     6244     6244              
==========================================
- Hits        40780    40702      -78     
- Misses      19395    19526     +131     
  Partials     1600     1600              

Flag	Coverage Δ
hive	52.14% <18.62%> (-0.30%)	⬇️
mysql	?
postgres	81.07% <36.03%> (-0.36%)	⬇️
presto	51.99% <18.62%> (-0.30%)	⬇️
python	81.47% <36.03%> (-0.41%)	⬇️
sqlite	80.78% <36.03%> (-0.35%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
superset/cli/celery.py	0.00% <0.00%> (ø)
superset/cli/examples.py	0.00% <0.00%> (ø)
superset/cli/main.py	0.00% <0.00%> (ø)
superset/cli/thumbnails.py	0.00% <0.00%> (ø)
superset/cli/update.py	0.00% <0.00%> (ø)
superset/cli/lib.py	61.53% <61.53%> (ø)
superset/cli/importexport.py	72.43% <72.43%> (ø)
superset/cli/test.py	83.72% <83.72%> (ø)
superset/common/utils/dataframe_utils.py	85.71% <0.00%> (-7.15%)	⬇️
superset/db_engine_specs/mysql.py	93.97% <0.00%> (-3.62%)	⬇️
... and 6 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 810cfc1...6fc34e6. Read the comment docs.

[dpgaspar]

LGTM!

We should have more test coverage on these.