Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
44 commits
Select commit Hold shift + click to select a range
bef853c
Add multi-threaded test reading encrypted parquet
EnricoMi Nov 19, 2024
501eef0
Add uniform encryption test
EnricoMi Dec 5, 2024
89be57b
Test multi-threaded read
EnricoMi Dec 5, 2024
5335a8c
Revert "Test multi-threaded read"
EnricoMi Dec 10, 2024
96c8778
Create new cipher context for each decrypt call
EnricoMi Dec 5, 2024
28e20e9
Move common code into AesEncryptionContext, remove wipeout from AesEn…
EnricoMi Dec 6, 2024
d47f5e9
Avoid segfaults daling with FileDecryptionProperties
EnricoMi Dec 10, 2024
18aa801
Remove WipeOut from nossl implementation
EnricoMi Dec 10, 2024
7663463
Remove all_decryptors_ and mutex_
EnricoMi Dec 11, 2024
24b986a
Remove context duplication
EnricoMi Dec 11, 2024
6a86cff
Rename ctxDeleter to match coding convention
EnricoMi Dec 11, 2024
bd24e54
Remove unused 'class AesEncryptionContext' from encryption_internal.h
EnricoMi Dec 11, 2024
0a3ca35
Restore properties_->WipeOut[De|En]cryptionKeys()
EnricoMi Dec 11, 2024
d74a7e7
Remove InitCipherContext, implement NewCipherContext in derived classes
EnricoMi Dec 11, 2024
5d902b5
Fix linting
EnricoMi Dec 11, 2024
6cc7f56
Use parametrized tests
EnricoMi Dec 12, 2024
e323ffc
Rework concurrency in test, use Arrow Future
EnricoMi Dec 12, 2024
99a1239
Fix lint error
EnricoMi Dec 12, 2024
c625efd
Use per PageReader copy of CryptoContext.[da|me]ta_decryptor
EnricoMi Dec 12, 2024
85bc7c5
Fix test parameter name
EnricoMi Dec 13, 2024
e83517e
Remove thread pool assertions
EnricoMi Dec 13, 2024
373ff36
Move threaded bool into test parameters
EnricoMi Dec 13, 2024
7e39752
Test writing multi-threaded
EnricoMi Dec 13, 2024
6a7788e
Fix code style in tests
EnricoMi Jan 21, 2025
b37b7da
Do not cache footer decryptors but footer key
EnricoMi Jan 21, 2025
a25ba1f
Fix NoSSL AesDecryptor::Make
EnricoMi Jan 21, 2025
813f9aa
Rework auto-freeing EVP_CIPHER_CTX
EnricoMi Feb 6, 2025
7c6e97c
Apply new CipherContext, rename AesEncryptionContext to AesCryptoContext
EnricoMi Feb 6, 2025
c8a1225
Merge std::tuple with EncryptionParam
EnricoMi Feb 6, 2025
9e68c92
Make GetColumnMetaDecryptor return a Decryptor factory
EnricoMi Feb 6, 2025
46288fb
Decryptor factory functions should have their encryption parameters c…
pitrou Feb 11, 2025
76b887f
Enable parallel dataset writing + various nits
pitrou Feb 12, 2025
162b605
Remove outdated comment
pitrou Feb 12, 2025
31ab8ba
Reintroduce mutex to protect footer key updates (+ nits)
pitrou Feb 12, 2025
1db3d24
Use unique_ptr for non-shareable objects + address review comments
pitrou Feb 13, 2025
da544f2
Add debug logs, hopefully we can diagnose this a bit more precisely
pitrou Feb 13, 2025
f175c71
Move decryptor reference into lambda
EnricoMi Mar 24, 2025
d89bcf1
Remove wipeout code, use shared_ptr instead of raw pointer
EnricoMi Mar 24, 2025
608899d
Clear keys in destructor
EnricoMi Mar 28, 2025
3193519
Revert "Add debug logs, hopefully we can diagnose this a bit more pre…
EnricoMi Mar 31, 2025
5c8d4e0
Test with static footer keys in file [en|de]cryption properties
EnricoMi Mar 31, 2025
fe98196
Provide label for test params via << operator
EnricoMi Apr 1, 2025
a587553
Apply minor code review comments
EnricoMi Apr 1, 2025
9961069
Hopefully fix compilation error
pitrou Apr 2, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -429,7 +429,7 @@ void InteropTestReadEncryptedParquetFiles(std::string root_path) {

// Add the current decryption configuration to ReaderProperties.
reader_properties.file_decryption_properties(
vector_of_decryption_configurations[example_id]->DeepClone());
vector_of_decryption_configurations[example_id]);

// Create a ParquetReader instance
std::unique_ptr<parquet::ParquetFileReader> parquet_reader =
Expand Down
21 changes: 11 additions & 10 deletions cpp/src/arrow/dataset/file_parquet.cc
Original file line number Diff line number Diff line change
Expand Up @@ -76,26 +76,25 @@ parquet::ReaderProperties MakeReaderProperties(
}
properties.set_buffer_size(parquet_scan_options->reader_properties->buffer_size());

auto file_decryption_prop =
parquet_scan_options->reader_properties->file_decryption_properties();

#ifdef PARQUET_REQUIRE_ENCRYPTION
auto parquet_decrypt_config = parquet_scan_options->parquet_decryption_config;

if (parquet_decrypt_config != nullptr) {
auto file_decryption_prop =
file_decryption_prop =
parquet_decrypt_config->crypto_factory->GetFileDecryptionProperties(
*parquet_decrypt_config->kms_connection_config,
*parquet_decrypt_config->decryption_config, path, filesystem);

parquet_scan_options->reader_properties->file_decryption_properties(
std::move(file_decryption_prop));
}
#else
if (parquet_scan_options->parquet_decryption_config != nullptr) {
parquet::ParquetException::NYI("Encryption is not supported in this build.");
}
#endif

properties.file_decryption_properties(
parquet_scan_options->reader_properties->file_decryption_properties());
properties.file_decryption_properties(file_decryption_prop);

properties.set_thrift_string_size_limit(
parquet_scan_options->reader_properties->thrift_string_size_limit());
Expand Down Expand Up @@ -527,9 +526,11 @@ Future<std::shared_ptr<parquet::arrow::FileReader>> ParquetFileFormat::GetReader
auto self = checked_pointer_cast<const ParquetFileFormat>(shared_from_this());

return source.OpenAsync().Then(
[=](const std::shared_ptr<io::RandomAccessFile>& input) mutable {
return parquet::ParquetFileReader::OpenAsync(input, std::move(properties),
metadata)
[self = self, properties = std::move(properties), source = source,
options = options, metadata = metadata,
parquet_scan_options = parquet_scan_options](
const std::shared_ptr<io::RandomAccessFile>& input) mutable {
return parquet::ParquetFileReader::OpenAsync(input, properties, metadata)
.Then(
[=](const std::unique_ptr<parquet::ParquetFileReader>& reader) mutable
-> Result<std::shared_ptr<parquet::arrow::FileReader>> {
Expand All @@ -544,7 +545,7 @@ Future<std::shared_ptr<parquet::arrow::FileReader>> ParquetFileFormat::GetReader
// here we know there are no other waiters on the reader.
std::move(const_cast<std::unique_ptr<parquet::ParquetFileReader>&>(
reader)),
std::move(arrow_properties), &arrow_reader));
arrow_properties, &arrow_reader));

// R build with openSUSE155 requires an explicit shared_ptr construction
return std::shared_ptr<parquet::arrow::FileReader>(
Expand Down
Loading
Loading