feat: Add parquet upload by exemplary-citizen · Pull Request #14449 · apache/superset

exemplary-citizen · 2021-05-03T18:05:05Z

SUMMARY

Allow CSV upload form to accept parquet file. Went in this direction so as not to exacerbate what was brought up in #13834 by adding a new form specifically for parquet files. I believe small modifications can be made to this PR to accommodate feather and orc files.

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TEST PLAN

Added a test similar to the ones already in csv_upload_tests.py

ADDITIONAL INFORMATION

Fixes #14020

Has associated issue: 'Upload parquet' option #14020
Changes UI
Includes DB Migration (follow approval process in SIP-59)
- Migration is atomic, supports rollback & is backwards-compatible
- Confirm DB migration upgrade and downgrade tested
- Runtime estimates and downtime expectations provided
Introduces new feature or API
Removes existing feature or API

exemplary-citizen · 2021-05-19T17:56:04Z

@john-bodley @villebro just wanted to follow up on this in case it got lost in the shuffle. How should we proceed on this PR?

villebro · 2021-05-20T18:11:12Z

@exemplary-citizen sorry for dropping the ball on this - I'll have this reviewed within the next 24 hours

…ad_parquet

robdiciuccio · 2021-05-20T22:03:34Z

/testenv up

robdiciuccio · 2021-05-20T22:10:02Z

+                "iterator": True,
+                "keep_default_na": not form.null_values.data,
+                "mangle_dupe_cols": form.mangle_dupe_cols.data,
+                "usecols": form.usecols.data,


This appears to change the behavior of the existing CSV upload functionality by specifying columns. Can you add some tests around this?

added a scenario to test_import_csv that tests uploading a CSV with specific columns

robdiciuccio · 2021-05-20T22:10:20Z

+            "If not None, only these columns will be read from the file."
+        ),
+        validators=[Optional()],
+    )


Can you provide a screenshot of the updated form UI?

Added a screenshot to the summary above

github-actions · 2021-05-21T01:36:35Z

@robdiciuccio Ephemeral environment spinning up at http://34.214.127.48:8080. Credentials are admin/admin. Please allow several minutes for bootstrapping and startup.

villebro

A few comments:

While it's convenient to add this functionality to the CSV upload form, I feel this should be in a form of its own, as the majority of the current fields are specific to CSV only (IIUC only usecols is needed for Parquet upload)
If the CSV upload form will also handle Parquet, the title needs to be updated to reflect this. However, I'd personally prefer moving this into a form of its own.
it would be nice if the form could handle directories/zip files, as it's fairly common to have partitioned data that is split up into multiple Parquet files. As pandas also supports uploading from a directory path, this would be a great feature to avoid having to manually append upload each file.

villebro · 2021-05-24T06:11:07Z

-                config["ALLOWED_EXTENSIONS"].intersection(config["CSV_EXTENSIONS"]),
+                config["ALLOWED_EXTENSIONS"].intersection(
+                    config["CSV_EXTENSIONS"].union(config["OTHER_EXTENSIONS"])
+                ),


You need to add the union to the error message below.

exemplary-citizen · 2021-05-25T07:31:51Z

Yeah I agree that this probably belongs in a separate form. Just went in this direction because creating a new form would mean that we'd effectively be abandoning #13834. I'll go ahead and get started working on a new form for parquet/directory/zip unless @junlincc @srinify @Steejay have any ideas on how to better consolidate these forms. I believe wtforms has a MultipleFileField class that should allow us to upload entire directories. We can also introduce this change to the CsvToDatabaseForm so users can upload multiple CSVs at a time.

exemplary-citizen · 2021-08-23T23:50:25Z

@villebro can you restart CI?

…ad_parquet

exemplary-citizen · 2021-08-26T07:08:59Z

@villebro should come back green now

villebro · 2021-08-26T12:37:00Z

@exemplary-citizen there's a linting error in one of the files. You can either setup pre-commit hooks or apply the diff below to fix the problem:

diff --git a/superset/views/database/views.py b/superset/views/database/views.py
index 8d0a92f6c..3863b165c 100644
--- a/superset/views/database/views.py
+++ b/superset/views/database/views.py
@@ -406,17 +406,23 @@ class ColumnarToDatabaseView(SimpleFormView):
     def form_get(self, form: ColumnarToDatabaseForm) -> None:
         form.if_exists.data = "fail"

-    def form_post(self, form: ColumnarToDatabaseForm) -> Response:  # pylint: disable=too-many-locals
+    def form_post(
+        self, form: ColumnarToDatabaseForm
+    ) -> Response:  # pylint: disable=too-many-locals
         database = form.con.data
         columnar_table = Table(table=form.name.data, schema=form.schema.data)
         files = form.columnar_file.data
         file_type = {file.filename.split(".")[-1] for file in files}

         if file_type == {"zip"}:
-            zipfile_ob = zipfile.ZipFile(form.columnar_file.data[0])  # pylint: disable=consider-using-with
+            zipfile_ob = zipfile.ZipFile(
+                form.columnar_file.data[0]
+            )  # pylint: disable=consider-using-with
             file_type = {filename.split(".")[-1] for filename in zipfile_ob.namelist()}
             files = [
-                io.BytesIO((zipfile_ob.open(filename).read(), filename)[0])  # pylint: disable=consider-using-with
+                io.BytesIO(
+                    (zipfile_ob.open(filename).read(), filename)[0]
+                )  # pylint: disable=consider-using-with
                 for filename in zipfile_ob.namelist()
             ]

exemplary-citizen · 2021-08-26T16:02:52Z

@villebro took care of the code formatting

villebro · 2021-08-31T04:04:01Z

@exemplary-citizen sorry to bother you again, but we've recently updated the version of pylint which appears to be causing some new linting issues. Do you want to fix this or is it ok if I jump in and resolve the remaining ones?

…ad_parquet

villebro

LGTM! While testing I found some edge cases that caused trouble, but those can be improved upon later (I'll try to open up a PR for some of them; will tag you for a review when I do).

github-actions · 2021-08-31T07:20:46Z

Ephemeral environment shutdown and build artifacts deleted.

exemplary-citizen · 2021-08-31T07:26:17Z

@villebro thx for pushing this through. yeah do tag me, i'll be sure to review. in the meantime i'll get started on #16046

villebro · 2021-08-31T07:28:37Z

@villebro thx for pushing this through. yeah do tag me, i'll be sure to review. in the meantime i'll get started on #16046

That sounds great @exemplary-citizen! Oh, and I forgot; thanks so much for your patience with the review process!

* allow csv upload to accept parquet file * fix mypy * fix if statement * add test for specificying columns in CSV upload * clean up test * change order in test * fix failures * upload parquet to seperate table in test * fix error message * fix mypy again * rename other extensions to columnar * add new form for columnar upload * add support for zip files * undo csv form changes except usecols * add more tests for zip * isort & black * pylint * fix trailing space * address more review comments * pylint * black * resolve remaining issues

LeoDiep · 2023-08-16T06:22:53Z

@villebro thx for pushing this through. yeah do tag me, i'll be sure to review. in the meantime i'll get started on #16046

That sounds great @exemplary-citizen! Oh, and I forgot; thanks so much for your patience with the review process!

Hi @villebro and @exemplary-citizen , I am using Superset v2.1.0 docker compose and I couldn't upload a parquet file to Superset. Is this request get deprecated in new version?

LeoDiep · 2023-08-16T08:11:45Z

@villebro thx for pushing this through. yeah do tag me, i'll be sure to review. in the meantime i'll get started on #16046

That sounds great @exemplary-citizen! Oh, and I forgot; thanks so much for your patience with the review process!

Hi @villebro and @exemplary-citizen , I am using Superset v2.1.0 docker compose and I couldn't upload a parquet file to Superset. Is this request get deprecated in new version?

I am sorry for the question above, I see the Upload columnar file to database now.
I am trying to programatic upload file from S3 to Superset db because the performance of slice query to Athena via pyAthena are so slow compare to postgresql query directly to superset db.
As I can see Athena doesn't receive 25 requests at the same time (my dashboard has 25 charts in 1 page) but doing it one by one and send the result back to Superset. I have tried domain sharding, increase gunicorn workers + threads but my Superset on docker compose didn't improve performance.

Is there a way to programatically import parquet files to superset db? Thanks

* allow csv upload to accept parquet file * fix mypy * fix if statement * add test for specificying columns in CSV upload * clean up test * change order in test * fix failures * upload parquet to seperate table in test * fix error message * fix mypy again * rename other extensions to columnar * add new form for columnar upload * add support for zip files * undo csv form changes except usecols * add more tests for zip * isort & black * pylint * fix trailing space * address more review comments * pylint * black * resolve remaining issues

allow csv upload to accept parquet file

4cc378a

pull-request-size Bot added the size/L label May 3, 2021

junlincc requested review from john-bodley and villebro May 3, 2021 19:43

junlincc added the data:csv Related to import/export of CSVs label May 3, 2021

exemplary-citizen requested a review from a team as a code owner May 20, 2021 21:33

exemplary-citizen force-pushed the upload_parquet branch from 06f9602 to 4cc378a Compare May 20, 2021 21:37

betodealmeida self-requested a review May 20, 2021 21:39

Merge branch 'master' of https://github.com/apache/superset into uplo…

c44fcee

…ad_parquet

robdiciuccio reviewed May 20, 2021

View reviewed changes

fix mypy

968c048

exemplary-citizen added 5 commits May 20, 2021 19:42

fix if statement

c23b1ec

add test for specificying columns in CSV upload

97bea75

clean up test

2f8482b

change order in test

77b381f

fix failures

f8b0dfc

villebro reviewed May 24, 2021

View reviewed changes

exemplary-citizen added 3 commits May 23, 2021 23:28

upload parquet to seperate table in test

b40a3e2

fix error message

f2911b5

fix mypy again

dd889b0

exemplary-citizen added 4 commits June 3, 2021 21:07

rename other extensions to columnar

cfdd3be

add new form for columnar upload

5e063bd

add support for zip files

4c74594

undo csv form changes except usecols

9096faf

exemplary-citizen added 2 commits August 25, 2021 23:14

Merge branch 'master' of https://github.com/apache/superset into uplo…

60ae1fe

…ad_parquet

pylint

5b5eb74

pull-request-size Bot added size/XL and removed size/L labels Aug 26, 2021

exemplary-citizen force-pushed the upload_parquet branch from 360e17e to 5b5eb74 Compare August 26, 2021 07:03

pull-request-size Bot added size/L and removed size/XL labels Aug 26, 2021

black

df2930f

pull-request-size Bot added size/XL and removed size/L labels Aug 26, 2021

exemplary-citizen added 2 commits August 30, 2021 21:49

Merge branch 'master' of https://github.com/apache/superset into uplo…

9b4d35b

…ad_parquet

resolve remaining issues

de3509e

villebro approved these changes Aug 31, 2021

View reviewed changes

villebro merged commit d25b096 into apache:master Aug 31, 2021

siegfriedweber mentioned this pull request Feb 9, 2022

Add Superset 1.4.1 stackabletech/docker-images#47

Closed

mistercrunch added 🏷️ bot A label used by `supersetbot` to keep track of which PR where auto-tagged with release labels 🚢 1.4.0 First shipped in 1.4.0 labels Mar 12, 2024

Conversation

exemplary-citizen commented May 3, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

SUMMARY

BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF

TEST PLAN

ADDITIONAL INFORMATION

Uh oh!

exemplary-citizen commented May 19, 2021

Uh oh!

villebro commented May 20, 2021

Uh oh!

robdiciuccio commented May 20, 2021

Uh oh!

robdiciuccio May 20, 2021

Choose a reason for hiding this comment

Uh oh!

exemplary-citizen May 21, 2021

Choose a reason for hiding this comment

Uh oh!

robdiciuccio May 20, 2021

Choose a reason for hiding this comment

Uh oh!

exemplary-citizen May 21, 2021

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented May 21, 2021

Uh oh!

villebro left a comment

Choose a reason for hiding this comment

Uh oh!

villebro May 24, 2021

Choose a reason for hiding this comment

Uh oh!

exemplary-citizen commented May 25, 2021

Uh oh!

exemplary-citizen commented Aug 23, 2021

Uh oh!

exemplary-citizen commented Aug 26, 2021

Uh oh!

villebro commented Aug 26, 2021

Uh oh!

exemplary-citizen commented Aug 26, 2021

Uh oh!

villebro commented Aug 31, 2021

Uh oh!

villebro left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Aug 31, 2021

Uh oh!

exemplary-citizen commented Aug 31, 2021

Uh oh!

villebro commented Aug 31, 2021

Uh oh!

LeoDiep commented Aug 16, 2023

Uh oh!

LeoDiep commented Aug 16, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

exemplary-citizen commented May 3, 2021 •

edited

Loading