Spark: Set properties for deletewriter #2859
Conversation
| case PARQUET: | ||
| return Parquet.writeDeletes(file.encryptingOutputFile()) | ||
| .createWriterFunc(msgType -> SparkParquetWriters.buildWriter(lazyEqDeleteSparkType(), msgType)) | ||
| .setAll(properties) |
There was a problem hiding this comment.
Thanks for the contribution, @coolderli ! I also think the newPosDeleteWriter need the properties setting ...
There was a problem hiding this comment.
|
@coolderli, if you have a use case where you found this problem, could you add it to the description? In what cases is this needed? |
@rdblue I was trying to use this writer to implement merge-on-read mode on spark for |
|
@coolderli, what went wrong without this? Also, you should definitely coordinate with @aokolnychyi on the Spark implementations. I believe that he has them fully working at this point. |
Yes, I found @aokolnychyi has implemented new writes #2873, we can use the new writer. |
|
Would appreciate your review on the new writers, @coolderli! |
|
I think we got this covered in the new |
Yes, I think the new |
4 participants
DeleteWrite should call
setAll(properties)to set properties for Parquet and Avro.