Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
125 commits
Select commit Hold shift + click to select a range
79f8eb0
encryption (from apache/parquet-cpp github repo)
thamht4190 Sep 13, 2018
138feeb
update thrift change and update encrypted footer
thamht4190 Jan 24, 2019
65fa020
add encryption source files into CMakeLists.txt
thamht4190 Jan 28, 2019
859fe6b
add example from old PR of parquet-cpp
thamht4190 Jan 29, 2019
60adabb
change due to new update in crypto package
thamht4190 Jan 31, 2019
4015657
pass EncryptionProperties into parquet_encryption::Encrypt()/Decrypt(…
thamht4190 Feb 1, 2019
700e71e
fix issue of wrong column name in encryption-example and remove FileE…
thamht4190 Feb 26, 2019
73b936e
get column path from ColumnCryptoMetadata when column is encrypted wi…
thamht4190 Feb 26, 2019
015fd0a
let encryption examples to be able to cover more cases
thamht4190 Feb 26, 2019
c0cbac9
footer plaintext mode
thamht4190 Feb 26, 2019
2ba89f4
footer plaintext mode example
thamht4190 Mar 4, 2019
331f94b
fix compiling issue
thamht4190 Mar 7, 2019
2f741e7
fix plaintext mode verification
thamht4190 Mar 8, 2019
1fdd1e2
fix memory issue when serializing plaintext mode footer
thamht4190 Mar 12, 2019
0bbe128
protection of sensitive metadata
thamht4190 Mar 14, 2019
23130e7
fix duplication of variable i
thamht4190 Mar 18, 2019
b221b96
column metadata encryption: read algorithm, aad from FileCryptoMetada…
thamht4190 Mar 19, 2019
21b6dda
keep redacted metadata version for old readers
thamht4190 Mar 20, 2019
42e4b48
hidden column exception
thamht4190 Mar 20, 2019
593a454
remove log
thamht4190 Mar 21, 2019
916b362
add example for hidden column
thamht4190 Mar 21, 2019
991cf35
handle row group file_offset and total_compressed_size
thamht4190 Mar 29, 2019
f7b265f
Apply API changes
revit13 Apr 15, 2019
c58af32
Add AAD calculation
revit13 Apr 16, 2019
c8f1a8a
Fix parquet tests to work with the changes required to support AAD
revit13 Apr 16, 2019
681209e
verify plaintext footer depends on config of decryption properties
thamht4190 Apr 19, 2019
0a2351d
Fix code style
revit13 Apr 22, 2019
6fbfc52
Code style fixes in properties.h
revit13 Apr 22, 2019
6187d40
revert change in parquet.thrift
thamht4190 Apr 24, 2019
c14ff0a
Move all encrypted related classes from properties.h to new files: en…
Apr 29, 2019
447a118
update crypto API change
thamht4190 Apr 25, 2019
6d6a685
fix issue when column is encrypted in footer plaintext mode
thamht4190 May 7, 2019
daec637
remove EncryptionProperties
thamht4190 May 7, 2019
1eb25e2
Change HiddenColumnExceptio message
May 7, 2019
3cc99fb
Fix indentation in encryption_properties.cc
May 7, 2019
0e29c54
Rename functions in DecryptionKeyRetriever
May 7, 2019
cc73f2c
Add check for aad_prefix to withoutAADPrefixStorage
May 7, 2019
cf6104f
Add exception to FromThrift in thrift.h
May 7, 2019
24d304f
Fix prefix aad calculation
May 7, 2019
eacef66
Remove fileAAD from ReaderProperties
May 7, 2019
4f3112c
Remove column_map from ReaderProperties
May 7, 2019
986331f
Fix check for encryption and the existance of file_decryption in file…
May 7, 2019
99c95b2
Save footer_key_metadata, algorithm, footer_decryptor and footer_sign…
May 8, 2019
7012745
Rename file_decryption to file_decryption_properties in properties.h
May 8, 2019
bbec428
Do not pass file_decryption as function parameter
May 8, 2019
a73ebf1
Rename is_plaintext_mode to is_encryption_algorithm_set
May 8, 2019
9b0dca7
fix function naming
thamht4190 May 9, 2019
250e1ed
fix const&
thamht4190 May 9, 2019
a0385a5
make format
thamht4190 May 10, 2019
71a279b
Add plaintext_files_allowed
May 12, 2019
a844c10
Remove file_crypto_metadata_ field from SerializedRowGroup and Serial…
May 12, 2019
674866a
Pass file_aad, algorithm and key_metadata to InternalFileDecryptor co…
May 12, 2019
9826a6d
Fixes to previous commits
May 12, 2019
3ddcd8c
Put encryption_properties.h/cc content in encryption.h/cc
May 12, 2019
5998317
Remove encryption_properties.cc from CMakeLists.txt
May 12, 2019
b70b68f
Add column_metadata_map_, column_data_map_, footer_signing_encryptor_…
May 12, 2019
551649f
Add column_data_map_, column_metadata_map_, footer_data_decryptor_ an…
May 12, 2019
9b5e5be
Rename aad to update_aad in Encryptor and Decryptor classes
May 12, 2019
60ef741
Move PARQUET_EMAGIC and PARQUET_MAGIC to file_writer.h and use it in …
May 12, 2019
e5c2b48
Rename file_encryption to file_encryption_properties in file_writer.cc
May 12, 2019
03d2495
Remove unused footer_decryptor_ from InternalFileDecryptor class and …
May 13, 2019
3537789
Fix format
May 13, 2019
9e18a00
Change implementation of NULL_STRING
May 13, 2019
752e33c
Change ParquetException message format in file_reader.cc
May 13, 2019
a24f1d4
Make format
May 13, 2019
1947381
Add comments to encryption-reader-writer.cc example
May 14, 2019
30c886a
Rename enable_plaintext_footer to set_plaintext_footer
May 14, 2019
8f911c1
Rename aad variable in NextPage function
May 14, 2019
18b598f
Change comment in GetColumnPageReader
May 14, 2019
8be2efe
Change additional comments in GetColumnPageReader
May 14, 2019
131cee6
Add comments in file_writer.cc
May 14, 2019
cb552f6
Create both data and metadata decryptors to avoid redundant retrieval…
May 14, 2019
8cc235f
Fix metadata parameter sent to parquet_encryption::AesDecryptor
May 14, 2019
9662384
Rename aad in GetFooterEncryptor and GetFooterSigningEncryptor
May 14, 2019
7bec5b3
Rename verify to verify_signature
May 14, 2019
4c1cfb3
Add comments to void WriteTo
May 14, 2019
1441ca0
Add additional comment in void WriteTo
May 14, 2019
b219b8d
Rename file_encryption to file_encryption_properties in WriterProperties
May 14, 2019
88687ad
Use encrypted_footer instead of footer_signing_key when checking for …
May 14, 2019
65ab864
Rename column_encryption_props to column_encryption_properties
May 14, 2019
49ff22b
Add comments in thrift.h
May 14, 2019
aef0d7d
Change parameters order in ColumnChunkMetaData::Make
May 14, 2019
b4debc4
Change parameters order in PageReader::Open
May 14, 2019
11d6965
Remove footer_encryption_key and footer_signing_key
May 14, 2019
b4efaac
Remove ParquetException in GetFooterSigningEncryptor and GetFooterEnc…
May 14, 2019
9fd10b1
make format
May 14, 2019
068743c
make format in thrift.h
May 14, 2019
cb11040
fix rebase mistake in parquet.thrift
thamht4190 May 15, 2019
8ba072c
Fix aad settings in thrift.h
May 19, 2019
04507d7
Port key erasure mechanism
May 20, 2019
7636dd4
Fix columnMetaData
May 20, 2019
ca2d8a6
Minor fixes to previous code
May 20, 2019
56204ab
fix build issue on MacOS
thamht4190 May 17, 2019
595716f
apply change from crypto package
thamht4190 May 30, 2019
7336685
format code
thamht4190 May 30, 2019
22a5c1e
post-rebase change
thamht4190 Jun 2, 2019
5697255
add unit tests for encryption properties
thamht4190 Jun 2, 2019
b7bdc8a
write unit tests for metadata
thamht4190 Jun 2, 2019
fb826f5
Add encryption samples
Jun 2, 2019
d1aa339
fix lint and format issue
thamht4190 Jun 3, 2019
44f37b0
fix metadata set, statistics set issues
thamht4190 Jun 4, 2019
72ca77f
Various changes to encryption-reader-writer-all-crypto-options test a…
Jun 5, 2019
899a603
Fix logging error
Jun 5, 2019
b59a3aa
post-rebase change
thamht4190 Jun 7, 2019
afbb8bf
fix isset of column chunk metadata and statistics
thamht4190 Jun 15, 2019
1661ebb
temporarily remove encryption-metadata-test
thamht4190 Jun 16, 2019
d604d10
fix windows compiling issue
thamht4190 Jun 17, 2019
00fbcf6
fix issue of parquet-encryption-example
thamht4190 Jun 17, 2019
905067d
rename encryption-test.cc to encryption-properties-test.cc
thamht4190 Jun 18, 2019
479da8f
use isset instead of creating a copy of column chunk metadata
thamht4190 Jun 18, 2019
7052838
Address review comments
Jun 21, 2019
b7f3ea3
Fix SerializedPageReader initialization
Jun 21, 2019
80710e7
Fix Format
Jun 23, 2019
b3e277e
let parquet encryption be able to be off (when openssl is not found)
thamht4190 Jun 23, 2019
977ef11
Fix LogicalType
Jun 24, 2019
fb8ca61
keep encryption parameters at method declaration
thamht4190 Jun 24, 2019
d7e024c
add PARQUET_EXPORT into Builder class of encryption properties
thamht4190 Jun 25, 2019
0c869dd
Change assert to ASSERT_EQ in encryption-configurations-test.cc
Jun 25, 2019
56b0455
fix cmake format
thamht4190 Jun 25, 2019
d09bf9d
Add MemoryPool field to Decryptors/Encryptors
Jun 26, 2019
4afcccb
keep encryption parameters at method declaration (column_writer.cc/.h)
thamht4190 Jun 26, 2019
afc7131
Write to parquet stream to file in encryption test
Jun 27, 2019
c744cd2
Add file reader and file writer Close to encryption-configurations-te…
Jul 1, 2019
76c2b02
Change encryption-configuration-test
Jul 4, 2019
9011411
Delete encryption-configuration-encrypted-columns-plaintext-footer.cc…
Jul 4, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions cpp/examples/parquet/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,16 @@ target_include_directories(parquet-low-level-example2 PRIVATE low-level-api/)
target_link_libraries(parquet-low-level-example parquet_static)
target_link_libraries(parquet-low-level-example2 parquet_static)

if (ARROW_USE_OPENSSL)
add_definitions(-DPARQUET_ENCRYPTION)
add_executable(parquet-encryption-example low-level-api/encryption-reader-writer.cc)
add_executable(parquet-encryption-example-all-crypto-options low-level-api/encryption-reader-writer-all-crypto-options.cc)
target_include_directories(parquet-encryption-example PRIVATE low-level-api/)
target_include_directories(parquet-encryption-example-all-crypto-options PRIVATE low-level-api/)
target_link_libraries(parquet-encryption-example parquet_static)
target_link_libraries(parquet-encryption-example-all-crypto-options parquet_static)
endif()

add_executable(parquet-arrow-example parquet-arrow/reader-writer.cc)
# Prefer shared linkage but use static if shared build is deactivated
if (ARROW_BUILD_SHARED)
Expand All @@ -34,3 +44,9 @@ add_dependencies(parquet
parquet-low-level-example
parquet-low-level-example2
parquet-arrow-example)

if (ARROW_USE_OPENSSL)
add_dependencies(parquet
parquet-encryption-example
parquet-encryption-example-all-crypto-options)
endif()

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
// Licensed to the Apache Software Foundation (ASF) under one
// or more contributor license agreements. See the NOTICE file
// distributed with this work for additional information
// regarding copyright ownership. The ASF licenses this file
// to you under the Apache License, Version 2.0 (the
// "License"); you may not use this file except in compliance
// with the License. You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.

#include <arrow/io/file.h>
#include <arrow/util/logging.h>

#include <parquet/api/reader.h>
#include <parquet/api/writer.h>

using parquet::ConvertedType;
using parquet::Repetition;
using parquet::Type;
using parquet::schema::GroupNode;
using parquet::schema::PrimitiveNode;

constexpr int FIXED_LENGTH = 10;

static std::shared_ptr<GroupNode> SetupSchema() {
parquet::schema::NodeVector fields;
// Create a primitive node named 'boolean_field' with type:BOOLEAN,
// repetition:REQUIRED
fields.push_back(PrimitiveNode::Make("boolean_field", Repetition::REQUIRED,
Type::BOOLEAN, ConvertedType::NONE));

// Create a primitive node named 'int32_field' with type:INT32, repetition:REQUIRED,
// logical type:TIME_MILLIS
fields.push_back(PrimitiveNode::Make("int32_field", Repetition::REQUIRED, Type::INT32,
ConvertedType::TIME_MILLIS));

fields.push_back(PrimitiveNode::Make("float_field", Repetition::REQUIRED, Type::FLOAT,
ConvertedType::NONE));

fields.push_back(PrimitiveNode::Make("double_field", Repetition::REQUIRED, Type::DOUBLE,
ConvertedType::NONE));

// Create a GroupNode named 'schema' using the primitive nodes defined above
// This GroupNode is the root node of the schema tree
return std::static_pointer_cast<GroupNode>(
GroupNode::Make("schema", Repetition::REQUIRED, fields));
}
Loading