Parquet modular encryption

[asfimport]

A mechanism for modular encryption and decryption of Parquet files. Allows to keep data fully encrypted in the storage - while enabling efficient analytics on the data, via reader-side extraction / authentication / decryption of data subsets required by columnar projection and predicate push-down.

Enables fine-grained access control to column data by encrypting different columns with different keys.

Supports a number of encryption algorithms, to account for different security and performance requirements.

Reporter: Gidon Gershinsky / @ggershinsky
Assignee: Gidon Gershinsky / @ggershinsky

Subtasks:

• Thrift crypto metadata structures
• parquet-format-structures encryption
• parquet-mr code changes for encryption support
• Document the modular encryption in parquet-format
• Crypto package in parquet-mr
• Separate iv_prefix for GCM and CTR modes
• RowGroup offset and total compressed size fields
• Enable old readers to access unencrypted columns in files with plaintext footer
• Detailed crypto specification
• Thrift crypto updates
• Update encryption spec for Bloom filter encryption
• Merge crypto spec and structures to format master
• Encryption: Interop and Function test suite for Java version
• Merge encryption branch into master

Related issues:

• Passing Field Metadata to Parquet (Blocked)
• Upgrade Parquet to 1.12.0 (relates to)
• [C++] Parquet modular encryption (depends upon)
• CLONE - [C++] Parquet modular encryption (depends upon)
• Data obfuscation layer for encryption (is depended upon by)
• [C++] Data set integrity tool (is depended upon by)
• Encryption key management tools (is depended upon by)
• High level interface to Parquet encryption (is depended upon by)
• Sample of usage Parquet-1396 and Parquet-1178 for column level encryption with pluggable key access (is depended upon by)

PRs and other links:

• Parquet modular encryption design doc
• Performance estimation

_{Note: This issue was originally created as PARQUET-1178. Please see the migration documentation for further details.}

[asfimport]

Mohammad Islam:
Very good and relevant Jira @ggershinsky.

We ( at Uber ) are also interested about this feature.  How it is coming along?

 

 

[asfimport]

Gidon Gershinsky / @ggershinsky:
Thank you Mohammad, this is moving along nicely.

The design is converging (thanks to Julien's and Marcel's feedback, and Ryan's spot-on comments), the implementation is half-way there, testing will start soon. A pull request should be ready in a couple of weeks.

[asfimport]

Mohammad Islam:
Thanks @ggershinsky for the updates.

I will also review the doc and add some of our requirements into the doc, if anything missing.

Looking forward for  the patch.

 

[asfimport]

Cesar Delgado / @beettlle:
Saw the modular encryption doc were merged a couple days ago.  Very happy to see this Jira moving along.  Can't wait to have something to try in the near future.

[asfimport]

Gidon Gershinsky / @ggershinsky:
Thanks Cesar, sounds good!

[asfimport]

DB Tsai:
This feature will be very important for both Spark community and our use-case. We're looking forward to it! Thanks.

[asfimport]

Gang Ma:
Notice that the encryption format has been merged to the encryption branch of parquet-format repo, and some implementation pr has been filed to parquet-mr repo, so is there any release plan for this feature? will it be available in parquet 1.12.0 ?

[asfimport]

Gabor Szadovszky / @gszadovszky:
@ggershinsky, this is an umbrella jira. Is it related to the parquet-format release 2.7.0? Please, remove the tag if not.

[asfimport]

Jason Brugger:
What's the best way to get started with this on a Databricks cluster? If I install format-2.7.0 as a new library, how would I reference this data source in lieu of the cluster's default parquet library?

[asfimport]

Gidon Gershinsky / @ggershinsky:
[~jasbru]  Currently, this can't be run on a Databricks cluster - besides the Thrift structures in parquet-format-2.7.0, it  will also require a Java implementation of parquet encryption and key management libraries (not merged yet, but we're working on this).

[asfimport]

Bogdan:
I tried out the encryption branch but the latest commit (#614) is not even compiling.

Would you recommend a commit I could checkout and test the encryption feature?

Thanks in advance!

 

[asfimport]

Gidon Gershinsky / @ggershinsky:
[~Vatkov], a number of pull requests are under construction.

Hopefully they will be sent to the repository by the next week. No guarantees they'll be merged by that time, but still you'll be able to assemble the outstanding pr's, and build/run the encryption code.

[asfimport]

Bogdan:
Thanks @ggershinsky!

[asfimport]

Venkata Satya Pradeep Srikakolapu:
This feature nicely fits in my case. Any timeline on this? I am basically looking for a format which an encrypt PII/PHI columns when storing on Data Lake Store in Azure DataBricks cluster. Do you recommend any other alternative if this feature is not going to be available soon?

[asfimport]

Gidon Gershinsky / @ggershinsky:
[~prdpsvs@gmail.com] this feature is already implemented in parquet-cpp code (check Apache Arrow, from version 0.16).
If you need a Java version, it should be available soon in parquet-mr, check #776 - the last pull request in the basic encryption layer. We're working to make it a part of the next parquet-mr release, 1.12. No specific timelines at this point.
On top of the basic encryption layer, we're building a high level interface that will simplify using the parquet encryption, see PARQUET-1568. Updated details are coming up, we plan to try to make it a part of v1.12 too.