[GLUTEN-8455][VL] Support encrypted parquet fallback for 3.5#8560
[GLUTEN-8455][VL] Support encrypted parquet fallback for 3.5#8560Yohahaha merged 7 commits intoapache:mainfrom
Conversation
|
Run Gluten ClickHouse CI on ARM |
|
Run Gluten ClickHouse CI on ARM |
|
Run Gluten ClickHouse CI on ARM |
|
Run Gluten ClickHouse CI on ARM |
ArnavBalyan
changed the title
| fileMetaData.getEncryptionType match { | ||
| case EncryptionType.UNENCRYPTED => | ||
| false | ||
| case EncryptionType.PLAINTEXT_FOOTER => | ||
| true | ||
| case _ => | ||
| false | ||
| } |
There was a problem hiding this comment.
why return file is encrypted when EncryptionType.PLAINTEXT_FOOTER?
would you post official parquet encrypt type doc?
There was a problem hiding this comment.
Sorry accidentally deleted the comment, here is the code ref - https://github.com/apache/parquet-java/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/format/converter/ParquetMetadataConverter.java#L1766. We still rely on the exception to check the footer + file encryption. The EncryptionType.PLAINTEXT_FOOTER checks for encrypted file but plain footer. It seems there is no doc on this, but hope the code ref helps. thanks!
There was a problem hiding this comment.
Could you add some comments to it? It should have some EncryptionType description and how we deal with it.
|
Run Gluten ClickHouse CI on ARM |
ArnavBalyan
force-pushed
the
arnavb/parquet-3.5
branch
from
5767320 to
9054a70
Compare
|
Run Gluten ClickHouse CI on ARM |
|
Run Gluten ClickHouse CI on ARM |
| testWithSpecifiedSparkVersion( | ||
| "Detect encrypted Parquet with encrypted footer", | ||
| Array("3.2", "3.3", "3.4")) { | ||
| Array("3.2", "3.3", "3.4", "3.5")) { |
There was a problem hiding this comment.
it's time to replace testWithSpecifiedSparkVersion to test, right?
Yohahaha
left a comment
Yohahaha
left a comment
There was a problem hiding this comment.
LGTM! only one comments
|
Run Gluten ClickHouse CI on ARM |
3 participants
EncryptionTypeprovided by Parquet 1.13 for Spark 3.5.