fix: Bump prophet, re-enable tests, and remedy column eligibility logic

[john-bodley]

SUMMARY

The prophet package (formally fbprophet) historically has been "difficult" to work with from an installation perspective—atypical requirement definitions (fbstan needed to be installed first), no pre-built wheels, etc.—though thankfully in v1.1 there are pre-built binaries for MacOS, Windows, and Linux.

This PR bumps the version of the package and defines it as a requirement for the testing environment removing the need for the pytest.importorskip("prophet")—likely present due to problematic build. This ensures that the associated tests will now always run.

The frozen requirements were updated by running:

pip-compile-multi --no-upgrade

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TESTING INSTRUCTIONS

CI.

ADDITIONAL INFORMATION

• Has associated issue:
• [prophet] prophet installation fails with ModuleNotFoundError: No module named 'lunarcalendar' #20901
• Tests fail for optional prophet forecasting functionality  #24406
• Required feature flags:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[john-bodley]

This is likely a legacy dependency resulting from not having run pip-compile-multi. The presto extra requirement is specified in requirements/development.in which is an input to requirements/testing.in and thus shouldn't be re-included here.

[john-bodley]

pystan is now no longer required for prophet v1.1.

[john-bodley]

I'm really not sure how this test worked before it was skipped without the inclusion of the fixture which ensures (hopefully) that the tests are idempotent.

[john-bodley]

@villebro I would love your input on this as I'm somewhat perplexed as to how this was working. After reenabling the tests it seemed to be trying to fit non-numerical columns within the pd.DataFrame. I couldn't find any options where we explicitly define which columns to produce a forecast for.

The original logic I had was for determining numeric columns was,

df.select_dtypes(include=np.number).columns

however the MySQL tests were failing because it seems like some of the numeric columns where of type object which is typically used to encode strings.

[john-bodley]

I'm not sure if 47 or 103 is correct. I assume these tests have been disabled for a long time and thus the underlying data might have changed.

[codecov]

Codecov Report

Merging #24129 (fdf434b) into master (e20b695) will increase coverage by 0.02%.
The diff coverage is 100.00%.

❗ Current head fdf434b differs from pull request most recent head 13597d1. Consider uploading reports for the commit 13597d1 to get more accurate results

@@            Coverage Diff             @@
##           master   #24129      +/-   ##
==========================================
+ Coverage   69.06%   69.08%   +0.02%     
==========================================
  Files        1901     1901              
  Lines       74019    74020       +1     
  Branches     8116     8116              
==========================================
+ Hits        51121    51137      +16     
+ Misses      20787    20772      -15     
  Partials     2111     2111              

Flag	Coverage Δ
hive	54.06% <100.00%> (+0.16%)	⬆️
mysql	79.50% <100.00%> (+0.08%)	⬆️
postgres	79.58% <100.00%> (+0.08%)	⬆️
presto	53.96% <100.00%> (+0.16%)	⬆️
python	83.53% <100.00%> (+0.04%)	⬆️
sqlite	78.09% <100.00%> (+0.08%)	⬆️
unit	54.70% <100.00%> (+0.04%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
superset/utils/pandas_postprocessing/prophet.py	81.03% <100.00%> (+26.64%)	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[john-bodley]

@villebro I might need some additional 👀 on this as I'm somewhat out of my league in terms of expectations and inconsistencies regarding prophet.

[john-bodley]

I'm not sure why (on line #473) result["data"] is [] for only MySQL.

[villebro]

I think we should update the query context so that it's not using the legacy properties (granularity, is_timeseries etc). I wonder if that may be causing issues.

[sebastianliebscher]

This PR will also fix #24406

[john-bodley]

@sebastianliebscher and @villebro I fixed the CI issue—there was an inconsistency with how the Pandas Dataframe columns where being encoded—and thus this should be ready to review.

[sebastianliebscher]

A new version of prophet is already available :)

Suggested change

	prophet==1.1.3
	prophet==1.1.4

[sebastianliebscher]

If I think about it: "prophet": ["prophet>=1.1.0, <2.0.0"], would always install latest prophet 1.x, but CI would only test the pinned release. Maybe we should pin prophet in setup.py?

[john-bodley]

@sebastianliebscher CI uses the pinned version in the frozen requirements/testing.txt file.

[sebastianliebscher]

I mean, with future releases of prophet if a user enables the optional dependency with pip install -e '.[prophet]', they would potentially install a version that is not tested against CI.

[john-bodley]

The issue you describe is true for any package (or sub-package) which is not explicitly pinning.

Note we (Airbnb) typically install Superset as a service by:

1. Defining a requirements/production.in file which includes the lines

-e .[prophet,…]
ddtrace
…

2. Running pip-compile-multi to compile the dependencies—taking into account the prophet, ddtrace, etc. production dependencies.
3. Installing Superset via,

python3 -m pip install -r requirements/production.txt
python3 -m pip install -e .

which ensures we always install Superset in a deterministic manner which is super critical.

Why doesn’t Superset explicitly pin every package (and sub-package) in setup.py? Though packages should provide flexibility, one could argue this is valid for a service—given it typically isn’t used as a dependency—however we run into the issue of creating an infeasible dependency space when installation dependent requirements are added. By relaxing constraints (by way of removing explicit pins) we allow pip-compile-multi to solve this somewhat difficult problem on our behalf.

[sebastianliebscher]

A bit late, but I thank you very much for your detailed explanation! This helped a lot to understand how Superset manages dependencies.

[villebro]

Looks good, thanks for cleaning this up. After this is merged I can follow up by refactoring those tests so that they're using the generic x-axis syntax in the query context. I'll also make sure the rowcount is correct. At any rate, I think things are better after this is merged than before, so LGTM 🚢

[villebro]

I think we should update the query context so that it's not using the legacy properties (granularity, is_timeseries etc). I wonder if that may be causing issues.