feat: add Shillelagh DB engine spec

[betodealmeida]

SUMMARY

Shillelagh is a library for querying APIs via SQL. It's currently used by Superset in order to connect to Google Sheets, but it offers more APIs:

Datasette

Datasette is an open-source tool for exploring and publishing data, developed by one of the creators of Django. "Datasettes" are self-hosted micro web apps that serve a few tables, with a friendly UI. For example, this list of trees in SF.

Shillelagh can query any Datasette table directly:

% shillelagh
sql> SELECT * FROM "https://san-francisco.datasettes.com/sf-trees/Street_Tree_List" LIMIT 5;
  TreeID    qLegalStatus    qSpecies  qAddress             SiteOrder    qSiteInfo    PlantType    qCaretaker  qCareAssistant    PlantDate                 DBH  PlotSize    PermitNotes                XCoord       YCoord    Latitude    Longitude  Location
--------  --------------  ----------  -----------------  -----------  -----------  -----------  ------------  ----------------  ----------------------  -----  ----------  --------------------  -----------  -----------  ----------  -----------  -------------------------------------
  141565               1           1  501X Baker St                1            1            1             1                    07/21/1988 12:00:00 AM     21  Width 0ft   Permit Number 25401   6.00061e+06  2.11083e+06     37.776      -122.441  (37.7759676911831, -122.441396661871)
  232565               2           2  940 Elizabeth St             1            2            1             1                    03/20/2017 12:00:00 AM      3  Width 4ft   Permit Number 779625  6.0004e+06   2.102e+06       37.7517     -122.441  (37.7517102172731, -122.441498017841)
  119263               3           3  495X Lakeshore Dr            1            3            1             2                                                   10x30
  207368               2           4  920 Kirkham St               1            1            1             1                                                6  Width 1ft                         5.99201e+06  2.10527e+06     37.7602     -122.471  (37.760210314285, -122.47073935813)
  188702               3           5  1501 Evans Ave               2            4            1             2                                               17  Width 4ft                         6.01599e+06  2.09822e+06     37.7422     -122.387  (37.7422086702947, -122.387293152263)
sql>

We can create a dataset in Superset and make a map of red maple trees in SF:

Socrata

The Socrata Open Data API is a simple API used by many governments, non-profits, and NGOs around the world, including the CDC. Shillelagh can connect directly to any Socrata dataset by using its API URL. Here's an example showing the percentage of adults partially vaccinated for COVID-19 in the US:

% shillelagh
sql> SELECT date, administered_dose1_recip_4 FROM "https://data.cdc.gov/resource/unsk-b7fc.json" WHERE location='US' LIMIT 5;
date          administered_dose1_recip_4
----------  ----------------------------
2021-08-23                          73.1
2021-08-22                          73
2021-08-21                          72.9
2021-08-20                          72.7
2021-08-19                          72.5
sql>

We can also create a dataset in Superset and make a time series:

WeatherAPI

This adapter offers access to historical weather data from WeatherAPI. It requires signing up and an API key, but there's a free account.

Here's an example chart showing the perceived temperature in Fahrenheit in Bodega Bay:

% shillelagh
sql> SELECT AVG(feelslike_f) AS "AVG(feelslike_f)"
FROM "https://api.weatherapi.com/v1/history.json?q=94923"
LIMIT 10000
OFFSET 0;
  AVG(feelslike_f)
------------------
           57.6054
sql>

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

N/A

TESTING INSTRUCTIONS

Shillelagh supports two dialects, shillelagh:// and shillelagh+safe://. The former is disabled in Superset if PREVENT_UNSAFE_DB_CONNECTIONS is true, since it can read and write to disk. The latter allows only safe adapters to be loaded, and they have to be listed explicitly:

An API key is needed for WeatherAPI:

ADDITIONAL INFORMATION

• Has associated issue:
• Required feature flags:
• Changes UI
• Includes DB Migration (follow approval process in SIP-59)
  • Migration is atomic, supports rollback & is backwards-compatible
  • Confirm DB migration upgrade and downgrade tested
  • Runtime estimates and downtime expectations provided
• Introduces new feature or API
• Removes existing feature or API

[codecov]

Codecov Report

Merging #16416 (f330351) into master (50d896f) will increase coverage by 0.00%.
The diff coverage is 67.77%.

❗ Current head f330351 differs from pull request most recent head 424abf2. Consider uploading reports for the commit 424abf2 to get more accurate results

@@           Coverage Diff           @@
##           master   #16416   +/-   ##
=======================================
  Coverage   76.51%   76.51%           
=======================================
  Files        1000     1001    +1     
  Lines       53479    53506   +27     
  Branches     6815     6814    -1     
=======================================
+ Hits        40917    40938   +21     
- Misses      12326    12330    +4     
- Partials      236      238    +2     

Flag	Coverage Δ
hive	81.26% <80.88%> (+0.03%)	⬆️
mysql	?
postgres	81.52% <86.02%> (-0.03%)	⬇️
presto	81.33% <80.14%> (?)
python	82.02% <86.76%> (+0.11%)	⬆️
sqlite	81.20% <85.92%> (+0.05%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
superset-frontend/src/components/Modal/Modal.tsx	100.00% <ø> (ø)
...-frontend/src/components/OmniContainer/Omnibar.tsx	100.00% <ø> (ø)
...rset-frontend/src/components/ReportModal/index.tsx	93.75% <ø> (+20.83%)	⬆️
...et-frontend/src/dashboard/components/Dashboard.jsx	78.84% <ø> (ø)
...rontend/src/dashboard/containers/DashboardPage.tsx	0.00% <0.00%> (ø)
...c/views/CRUD/data/database/DatabaseModal/index.tsx	44.24% <0.00%> (-0.12%)	⬇️
superset/charts/commands/importers/dispatcher.py	80.64% <ø> (ø)
superset/cli.py	52.85% <0.00%> (ø)
superset/commands/importers/v1/__init__.py	94.73% <ø> (ø)
superset/commands/importers/v1/examples.py	39.53% <ø> (ø)
... and 102 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 50d896f...424abf2. Read the comment docs.

[etr2460]

any breaking changes here?

[betodealmeida]

No breaking changes, just fixes and features.

[etr2460]

any breaking changes here?

[betodealmeida]

No breaking changes, just fixes and features.