[C++][Parquet] Parquet modular encryption

[asfimport]

CPP version of a mechanism for modular encryption and decryption of Parquet files. Allows to keep the data fully encrypted in the storage, while enabling a client to extract a required subset (footer, column(s), pages) and to authenticate / decrypt the extracted data.

Reporter: Gidon Gershinsky / @ggershinsky
Assignee: Deepak Majeti / @majetideepak

Subtasks:

• [C++] Crypto package in parquet-cpp
• [C++] Update cpp crypto package to match signed-off specification
• [C++] Add encrypted parquet files to apache parquet-testing repository

Related issues:

• CLONE - [C++] Parquet modular encryption (is cloned by)
• Parquet modular encryption (is depended upon by)

Original Issue Attachments:

• column_reader.cc
• column_writer.cc
• file_reader.cc
• file_writer.cc
• thrift.h

PRs and other links:

• GitHub Pull Request #2555
• GitHub Pull Request #4826
• GitHub Pull Request #475
• Design

_{Note: This issue was originally created as PARQUET-1300. Please see the migration documentation for further details.}

[asfimport]

Gidon Gershinsky / @ggershinsky:
@majetideepak, per the recent discussion, publishing the current parquet-cpp code with encryption:

• a new file - crypto.h - with the basics of AES-GCM encryption/decryption, and inline functions (like in thrift.h), is in a reasonable shape (I hope), and published as a pull request to the parquet-cpp repository. along with an updated types.h and parquet.thrift files.

• existing parquet-cpp files (column_writer.cc, column_reader.cc, file_writer.cc, file_reader.cc), based on v1.4.0, are modified to parse the new Thrift crypto structures, and to encrypt/decrypt the footer/headers and data/dictionary pages. This works, and was tested for interoperability with Java Parquet encryption. However, as you'll see, this cpp code is in an early prototype stage, with hardcoded encryption keys, no API for passing them or other parameters, etc. I'm attaching the code to this Jira, feel free to modify it as needed, and to assign the Jira to you. Thanks for your help with this!

• last point, on linkage - the libcrypto has to be added to app makefile.
  (eg in examples/low-level-api/CMakeFiles/reader-writer.dir/link.txt, I'm adding /usr/lib/x86_64-linux-gnu/libcrypto.so)

[asfimport]

Tham / @thamht4190:
@majetideepak @ggershinsky I've just discovered this ticket. I'm also working on it (for my project) and almost done. May I know your progress and is there anyway to co-work then we can have this feature sooner?

[asfimport]

Gidon Gershinsky / @ggershinsky:
@thamht4190 this is excellent news! In parquet-cpp, I'm working on the two bottom layers: the format / thrift (PR#463), and the crypto package (PR#464). The former is up to date with the current design and the Java version, while the latter is not. If your crypto code supports both encryption algorithms, and has been tested for interoperability with the current Java Parquet encryption (PARQUET-1178), it'd be good to have it contributed to the project. If helps, we can work on this stuff (second cipher and interop testing) together.

As for layers above the crypto package, deferring to  @majetideepak.

 

[asfimport]

Tham / @thamht4190:
@ggershinsky My code is based on 2 pull requests you created for  PARQUET-1301 &  PARQUET-1227, and I also has to modify some code after looking at 4 uploaded files in this ticket. However, I didn't do any interop testing with Java version (mainly because I have very little experience with Java code). What should I do now? I will finish my code few days later, may I have your review?

[asfimport]

Gidon Gershinsky / @ggershinsky:
@thamht4190 Sounds good, I'll update then the PR#464 to support the second cipher. Lets sync with Deepak and discuss the next steps, details of interop testing, etc.

[asfimport]

Tham / @thamht4190:
Thanks @ggershinsky. Looking forward to your PR with second cipher.

@majetideepak I'm curious and would like to know how you plan to change in parquet-cpp API to pass encryption key, encryption type, etc. Can you please share?

[asfimport]

Deepak Majeti / @majetideepak:
@thamht4190, I have not started writing any code in either of the layers @ggershinsky mentioned. I will be happy to work with you on the parquet-cpp API as well as the Java interop testing. Can you open a new pull request on the parquet-cpp project with your current code? We can get some early feedback and it will help us better design the API.

[asfimport]

Gidon Gershinsky / @ggershinsky:
@thamht4190, I've added the second cipher and made some other modifications (inc driven by [~James C] PR review). Please update your code to accommodate for changes in PR#464 and PR#463.

Also, I'm attaching an additional file (thrift.h) that demonstrates encryption of Parquet metadata, eg footer and headers. Like the other 4 attached files, it requires substantial modifications.

Once you publish your PR, we'll review it and start working on interop testing.

[asfimport]

Tham / @thamht4190:
@ggershinsky Thanks for your updates. I will modify my code due to your changes and create a PR soon.

[asfimport]

Gidon Gershinsky / @ggershinsky:
resolved by Arrow 0.16.0 release

[asfimport]

Wes McKinney / @wesm:
Anyone interested in looking at packaging issues for encryption? I don't think it's being shipped in Arrow packages yet