Support write mode for Parquet and Avro using table property#302
Support write mode for Parquet and Avro using table property#302arina-ielchiieva wants to merge 1 commit intoapache:masterfrom
Conversation
4b136cb to
cd20e1a
Compare
| public static final String PARQUET_COMPRESSION_DEFAULT = "gzip"; | ||
|
|
||
| public static final String PARQUET_WRITE_MODE = "write.parquet.write-mode"; | ||
| public static final String PARQUET_WRITE_MODE_DEFAULT = "overwrite"; |
There was a problem hiding this comment.
👍 nice, having the "overwrite" default makes this change backwards compatible, right?
There was a problem hiding this comment.
For Parquet - yes, initially it was overwrite. For Avro - no, initially it was create. I made both overwrite to be the same but I can default Avro to create for backward compatibility if needed.
|
Can you update the description with what this changes and why it is useful? |
|
@rdblue updated the description. |
|
@arina-ielchiieva, why is this needed? Shouldn't the mode always be overwrite? |
|
@rdblue
|
|
I think we should make a decision and have a single default behavior instead of adding table properties. I don't see much value in being able to configure this differently across tables. |
|
Closed in favor of PR #318. |
Currently in Parquet and Avro writers write mode is hard-coded. For Parquet - overwrite, Avro - create.
This PR allows to set write mode using table properties for each of the above mentioned writers thus write mode can be controlled based on the needs of Iceberg table user.