PARQUET-1300: [C++] Implement encrypted Parquet read and write support#4826

Closed

thamht4190 wants to merge 207 commits intoapache:masterfrom

thamht4190:master

Commits on Aug 21, 2019

encryption (from apache/parquet-cpp github repo)
thamht4190
committed
update thrift change and update encrypted footer
thamht4190
committed
add encryption source files into CMakeLists.txt
thamht4190
committed
add example from old PR of parquet-cpp
thamht4190
committed
change due to new update in crypto package
thamht4190
committed
pass EncryptionProperties into parquet_encryption::Encrypt()/Decrypt() instead of pass it as many params
thamht4190
committed
fix issue of wrong column name in encryption-example and remove FileEncryptionProperties::uniform_encryption_
thamht4190
committed
get column path from ColumnCryptoMetadata when column is encrypted with its own key
thamht4190
committed
let encryption examples to be able to cover more cases
thamht4190
committed
footer plaintext mode
thamht4190
committed
footer plaintext mode example
thamht4190
committed
fix compiling issue
thamht4190
committed
fix plaintext mode verification
thamht4190
committed
fix memory issue when serializing plaintext mode footer
thamht4190
committed
protection of sensitive metadata
thamht4190
committed
fix duplication of variable i
thamht4190
committed
column metadata encryption: read algorithm, aad from FileCryptoMetadata or FileMetadata
thamht4190
committed
keep redacted metadata version for old readers
thamht4190
committed
hidden column exception
thamht4190
committed
remove log
thamht4190
committed
add example for hidden column
thamht4190
committed
handle row group file_offset and total_compressed_size
thamht4190
committed
Apply API changes

revit13
authored and
thamht4190
committed
Add AAD calculation

revit13
authored and
thamht4190
committed
Fix parquet tests to work with the changes required to support AAD

revit13
authored and
thamht4190
committed
verify plaintext footer depends on config of decryption properties
thamht4190
committed
Fix code style

revit13
authored and
thamht4190
committed
Code style fixes in properties.h

revit13
authored and
thamht4190
committed
revert change in parquet.thrift
thamht4190
committed
Move all encrypted related classes from properties.h to new files: encryption_properties.h and encryption_properties.cc

Revital1 Eres
authored and
thamht4190
committed
update crypto API change
thamht4190
committed
fix issue when column is encrypted in footer plaintext mode
thamht4190
committed
remove EncryptionProperties
thamht4190
committed
Change HiddenColumnExceptio message

Revital1 Eres
authored and
thamht4190
committed
Fix indentation in encryption_properties.cc

Revital1 Eres
authored and
thamht4190
committed
Rename functions in DecryptionKeyRetriever

Revital1 Eres
authored and
thamht4190
committed
Add check for aad_prefix to withoutAADPrefixStorage

Revital1 Eres
authored and
thamht4190
committed
Add exception to FromThrift in thrift.h

Revital1 Eres
authored and
thamht4190
committed
Fix prefix aad calculation

Revital1 Eres
authored and
thamht4190
committed
Remove fileAAD from ReaderProperties

Revital1 Eres
authored and
thamht4190
committed
Remove column_map from ReaderProperties

Revital1 Eres
authored and
thamht4190
committed
Fix check for encryption and the existance of file_decryption in file_reader.cc

Revital1 Eres
authored and
thamht4190
committed
Save footer_key_metadata, algorithm, footer_decryptor and footer_signing_encryptor in InternalFileDecryptor and remove redundant AAD calculation in file_reader.cc

Revital1 Eres
authored and
thamht4190
committed
Rename file_decryption to file_decryption_properties in properties.h

Revital1 Eres
authored and
thamht4190
committed
Do not pass file_decryption as function parameter

Revital1 Eres
authored and
thamht4190
committed
Rename is_plaintext_mode to is_encryption_algorithm_set

Revital Sur
authored and
thamht4190
committed
fix function naming
thamht4190
committed
fix const&
thamht4190
committed
make format
thamht4190
committed
Add plaintext_files_allowed

Revital Sur
authored and
thamht4190
committed
Remove file_crypto_metadata_ field from SerializedRowGroup and SerializedFile classes

Revital Sur
authored and
thamht4190
committed
Pass file_aad, algorithm and key_metadata to InternalFileDecryptor constructor

Revital Sur
authored and
thamht4190
committed
Fixes to previous commits

Revital Sur
authored and
thamht4190
committed
Put encryption_properties.h/cc content in encryption.h/cc

Revital Sur
authored and
thamht4190
committed
Remove encryption_properties.cc from CMakeLists.txt

Revital Sur
authored and
thamht4190
committed
Add column_metadata_map_, column_data_map_, footer_signing_encryptor_ and footer_encryptor_ to InternalFileEncryptor

Revital Sur
authored and
thamht4190
committed
Add column_data_map_, column_metadata_map_, footer_data_decryptor_ and footer_metadata_decryptor_ to InternalFileDecryptor

Revital Sur
authored and
thamht4190
committed
Rename aad to update_aad in Encryptor and Decryptor classes

Revital Sur
authored and
thamht4190
committed
Move PARQUET_EMAGIC and PARQUET_MAGIC to file_writer.h and use it in file_reader.cc

Revital Sur
authored and
thamht4190
committed
Rename file_encryption to file_encryption_properties in file_writer.cc

Revital Sur
authored and
thamht4190
committed
Remove unused footer_decryptor_ from InternalFileDecryptor class and fix indentation in GetFooterSigningEncryptor

Revital Sur
authored and
thamht4190
committed
Fix format

Revital1 Eres
authored and
thamht4190
committed
Change implementation of NULL_STRING

Revital1 Eres
authored and
thamht4190
committed
Change ParquetException message format in file_reader.cc

Revital1 Eres
authored and
thamht4190
committed
Make format

Revital1 Eres
authored and
thamht4190
committed
Add comments to encryption-reader-writer.cc example

Revital1 Eres
authored and
thamht4190
committed
Rename enable_plaintext_footer to set_plaintext_footer

Revital1 Eres
authored and
thamht4190
committed
Rename aad variable in NextPage function

Revital1 Eres
authored and
thamht4190
committed
Change comment in GetColumnPageReader

Revital1 Eres
authored and
thamht4190
committed
Change additional comments in GetColumnPageReader

Revital1 Eres
authored and
thamht4190
committed
Add comments in file_writer.cc

Revital1 Eres
authored and
thamht4190
committed
Create both data and metadata decryptors to avoid redundant retrieval of key

Revital1 Eres
authored and
thamht4190
committed
Fix metadata parameter sent to parquet_encryption::AesDecryptor

Revital1 Eres
authored and
thamht4190
committed
Rename aad in GetFooterEncryptor and GetFooterSigningEncryptor

Revital1 Eres
authored and
thamht4190
committed
Rename verify to verify_signature

Revital1 Eres
authored and
thamht4190
committed
Add comments to void WriteTo

Revital1 Eres
authored and
thamht4190
committed
Add additional comment in void WriteTo

Revital1 Eres
authored and
thamht4190
committed
Rename file_encryption to file_encryption_properties in WriterProperties

Revital1 Eres
authored and
thamht4190
committed
Use encrypted_footer instead of footer_signing_key when checking for encrypted footer

Revital1 Eres
authored and
thamht4190
committed
Rename column_encryption_props to column_encryption_properties

Revital1 Eres
authored and
thamht4190
committed
Add comments in thrift.h

Revital1 Eres
authored and
thamht4190
committed
Change parameters order in ColumnChunkMetaData::Make

Revital1 Eres
authored and
thamht4190
committed
Change parameters order in PageReader::Open

Revital1 Eres
authored and
thamht4190
committed
Remove footer_encryption_key and footer_signing_key

Revital1 Eres
authored and
thamht4190
committed
Remove ParquetException in GetFooterSigningEncryptor and GetFooterEncryptor

Revital1 Eres
authored and
thamht4190
committed
make format

Revital1 Eres
authored and
thamht4190
committed
make format in thrift.h

Revital1 Eres
authored and
thamht4190
committed
fix rebase mistake in parquet.thrift
thamht4190
committed
Fix aad settings in thrift.h

Revital1 Eres
authored and
thamht4190
committed
Port key erasure mechanism

Revital1 Eres
authored and
thamht4190
committed
Fix columnMetaData

Revital1 Eres
authored and
thamht4190
committed
Minor fixes to previous code

Revital1 Eres
authored and
thamht4190
committed
fix build issue on MacOS
thamht4190
committed
apply change from crypto package
thamht4190
committed
format code
thamht4190
committed
post-rebase change
thamht4190
committed
add unit tests for encryption properties
thamht4190
committed
write unit tests for metadata
thamht4190
committed
Add encryption samples

Revital1 Eres
authored and
thamht4190
committed
fix lint and format issue
thamht4190
committed
fix metadata set, statistics set issues
thamht4190
committed
Various changes to encryption-reader-writer-all-crypto-options test and to AAD handling in file_reader.cc

Revital1 Eres
authored and
thamht4190
committed
Fix logging error

Revital1 Eres
authored and
thamht4190
committed
post-rebase change
thamht4190
committed
fix isset of column chunk metadata and statistics
thamht4190
committed
temporarily remove encryption-metadata-test
thamht4190
committed
fix windows compiling issue
thamht4190
committed
fix issue of parquet-encryption-example
thamht4190
committed
rename encryption-test.cc to encryption-properties-test.cc
thamht4190
committed
use isset instead of creating a copy of column chunk metadata
thamht4190
committed
Address review comments

Revital1 Eres
authored and
thamht4190
committed
Fix SerializedPageReader initialization

Revital1 Eres
authored and
thamht4190
committed
Fix Format

Revital1 Eres
authored and
thamht4190
committed
let parquet encryption be able to be off (when openssl is not found)
thamht4190
committed
Fix LogicalType

Revital1 Eres
authored and
thamht4190
committed
keep encryption parameters at method declaration
thamht4190
committed
add PARQUET_EXPORT into Builder class of encryption properties
thamht4190
committed
Change assert to ASSERT_EQ in encryption-configurations-test.cc

Revital1 Eres
authored and
thamht4190
committed
fix cmake format
thamht4190
committed
Add MemoryPool field to Decryptors/Encryptors

Revital1 Eres
authored and
thamht4190
committed
keep encryption parameters at method declaration (column_writer.cc/.h)
thamht4190
committed
Write to parquet stream to file in encryption test

Revital1 Eres
authored and
thamht4190
committed
Add file reader and file writer Close to encryption-configurations-test.cc

Revital1 Eres
authored and
thamht4190
committed
Change encryption-configuration-test

Revital1 Eres
authored and
thamht4190
committed
Delete encryption-configuration-encrypted-columns-plaintext-footer.cc test

Revital1 Eres
authored and
thamht4190
committed
Remove FooterSigningEncryptor class

Revital1 Eres
authored and
thamht4190
committed
remove some PARQUET_ENCRYPTION define check
thamht4190
committed
remove PARQUET_ENCRYPTION ifdefs and add encryption_internal-nossl.cc for compile issue without openssl
thamht4190
committed
remove PARQUET_ENCRYPTION defines from CMakeLists.txt
thamht4190
committed
fix comments for encryption_internal_nossl.cc
thamht4190
committed
Format fixes and check that all columns in columnEncryptionProperties exist in the schema.

Revital1 Eres
authored and
thamht4190
committed
Add encryption tests

Revital1 Eres
authored and
thamht4190
committed
Throw exception when files are missing from parquet-testing repo

Revital1 Eres
authored and
thamht4190
committed
update parquet_testing submodule with new encrypted files
thamht4190
committed
add crypto dependency to R build

Deepak Majeti
authored and
thamht4190
committed
Print location of OpenSSL library

Deepak Majeti
authored and
thamht4190
committed
try adding crypto dependency to R build again

Deepak Majeti
authored and
thamht4190
committed
add missing crypto deps

Deepak Majeti
authored and
thamht4190
committed
fix ci openssl url

Deepak Majeti
authored and
thamht4190
committed
add crypt32 lib

Deepak Majeti
authored and
thamht4190
committed
Applying revital's const-fix patch & Addressing Deepak's review comment for the 4th decryption config

Gal Lushi
authored and
thamht4190
committed
post-rebase change
thamht4190
committed
fix comments
thamht4190
committed
rename test file using underscore
thamht4190
committed
fix make lint
thamht4190
committed

Commits on Aug 24, 2019

fix a bad merge in r/configure.win
thamht4190
committed

Commits on Sep 4, 2019

merge master to parquet encryption
thamht4190
committed

Commits on Sep 10, 2019

merge master to parquet encryption (2)
thamht4190
committed

Commits on Sep 16, 2019

add PARQUET_EXPORT to Encryptor, Decryptor
thamht4190
committed

Commits on Sep 23, 2019

fix errors when encryption is disabled

Deepak Majeti
authored and
Deepak Majeti
committed
do not build encryption support by default when Parquet is built

Deepak Majeti
authored and
Deepak Majeti
committed
specify PARQUET_REQUIRE_ENCRYPTION in doc, build, CI, etc.

Deepak Majeti
authored and
Deepak Majeti
committed
Merge branch 'master' into master
majetideepak
authored

Commits on Sep 25, 2019

fix format

Deepak Majeti
authored and
Deepak Majeti
committed
refactor column reader

Deepak Majeti
authored and
Deepak Majeti
committed

Commits on Sep 30, 2019

ARROW-6610: [C++] Add cmake option to disable filesystem layer
pitrou
committed
ARROW-6564: [Python] Do not require pandas for invoking ChunkedArray.__array__

jorisvandenbossche
authored and
wesm
committed
ARROW-6729: [C++] Prevent data copying in StlStringBuffer

st-pasha
authored and
wesm
committed
ARROW-6646: [Go] Write no IPC buffer metadata for NullType

sbinet
authored and
wesm
committed
ARROW-6685: [C++] Ignore trailing slashes in S3FS

pitrou
authored and
wesm
committed
ARROW-6740: [C++] Unmap MemoryMappedFile as soon as possible

pitrou
authored and
wesm
committed
ARROW-6708: [C++] Fix hardcoded boost library names

pitrou
authored and
wesm
committed

Commits on Oct 1, 2019

ARROW-6722: [Java] Provide a uniform way to get vector name

liyafan82
authored and
wesm
committed
ARROW-6655: [Python] Filesystem bindings for S3

kszucs
authored and
pitrou
committed
ARROW-6648: [Go] Expose the bitutil package

jsternberg
authored and
sbinet
committed
ARROW-5831: [Release] Add Python program to download binary artifacts in parallel, allow abort/resume
wesm
committed
ARROW-6751: [CI] Fix ccache setup on Travis-CI
pitrou
committed
ARROW-6745: [Rust] Fix a variety of minor typos.

waywardmonkeys
authored and
paddyhoran
committed

Commits on Oct 2, 2019

ARROW-6730: [CI] Use GitHub Actions for "C++ with clang 7" docker image

fsaintjacques
and
kou
committed
ARROW-6752: [Go] make Null array implement Stringer, add tests for Nu…
sbinet
committed
ARROW-6750: [Python] Silence S3 error logs by default

pitrou
authored and
kszucs
committed
ARROW-6755: [Release] Improve Windows release verification script

wesm
authored and
pitrou
committed
ARROW-6614: [C++][Dataset] Add DataSourceDiscovery class

fsaintjacques
authored and
bkietz
committed
ARROW-6581: [C++] Fix fuzzit job submission

pitrou
authored and
kszucs
committed
ARROW-6761: [Rust] Travis build now uses the correct Rust toolchain

andygrove
authored and
paddyhoran
committed

Commits on Oct 3, 2019

Commits on Oct 4, 2019

ARROW-6736: [Rust] [DataFusion] Evaluate the input to the aggregate expression just once per batch
andygrove
committed
ARROW-6580: [Java] Support comparison for unsigned integers

liyafan82
authored and
praveenbingo
committed
ARROW-6657: [Rust] [DataFusion] Add Count Aggregate Expression

sinistersnare
and
andygrove
committed
ARROW-6760: [C++] More informative error messages for JSON parsing errors
bkietz
committed
ARROW-6437: [R] Add AWS SDK to Homebrew formulae

nealrichardson
authored and
fsaintjacques
committed
Merge remote-tracking branch 'upstream/master' into PARQUET-1300

Deepak Majeti
authored and
Deepak Majeti
committed

Commits on Oct 5, 2019

refactor ColumnPath

Deepak Majeti
authored and
Deepak Majeti
committed
make format

Deepak Majeti
authored and
Deepak Majeti
committed
fix tests

Deepak Majeti
authored and
Deepak Majeti
committed
merge master

Deepak Majeti
authored and
Deepak Majeti
committed

Commits on Oct 7, 2019

Fix file_reader

Deepak Majeti
authored and
Deepak Majeti
committed
refactor metadata

Deepak Majeti
authored and
Deepak Majeti
committed
make lint

Deepak Majeti
authored and
Deepak Majeti
committed
cpp cli lint

Deepak Majeti
authored and
Deepak Majeti
committed
fix multi-rowgroup aad

Deepak Majeti
authored and
Deepak Majeti
committed

Commits on Oct 15, 2019

Merge branch 'master' into master
majetideepak
authored

Commits on Oct 30, 2019

Stylistic fixes, remove cruft
wesm
committed