Docs: Update defaults for distribution mode #10575
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
docs/docs/spark-configuration.md
Outdated
| | compression-codec | Table write.(fileformat).compression-codec | Overrides this table's compression codec for this write | | ||
| | compression-level | Table write.(fileformat).compression-level | Overrides this table's compression level for Parquet and Avro tables for this write | | ||
| | compression-strategy | Table write.orc.compression-strategy | Overrides this table's compression strategy for ORC tables for this write | | ||
| | distribution-mode | `Range` if Sort Order Defined ; `Hash` if Partition Defined; `None` otherwise | Override this table's distribution mode for this write | |
Member
There was a problem hiding this comment.
Suggested change
| | distribution-mode | `Range` if Sort Order Defined ; `Hash` if Partition Defined; `None` otherwise | Override this table's distribution mode for this write | | |
| | distribution-mode | `Range` if sort order is defined ; `Hash` if partition is defined; `None` otherwise | Override this table's distribution mode for this write | |
dramaticlly
reviewed
Jul 3, 2024
docs/docs/configuration.md
Outdated
| | write.target-file-size-bytes | 536870912 (512 MB) | Controls the size of files generated to target about this many bytes | | ||
| | write.delete.target-file-size-bytes | 67108864 (64 MB) | Controls the size of delete files generated to target about this many bytes | | ||
| | write.distribution-mode | none | Defines distribution of write data: __none__: don't shuffle rows; __hash__: hash distribute by partition key ; __range__: range distribute by partition key or sort key if table has an SortOrder | | ||
| | write.distribution-mode | none (see engines for specific defaults) | Defines distribution of write data: __none__: don't shuffle rows; __hash__: hash distribute by partition key ; __range__: range distribute by partition key or sort key if table has an SortOrder | |
Contributor
There was a problem hiding this comment.
I am wondering if it's helpful to hyperlink the engine specific defaults here or in notes , for the convenience of lookup
like
[Spark](spark-configuration.md#write-options)
dramaticlly
approved these changes
Jul 12, 2024
Member
Author
|
@RussellSpitzer do you have time to take a look at this? |
docs/docs/configuration.md
Outdated
| | write.target-file-size-bytes | 536870912 (512 MB) | Controls the size of files generated to target about this many bytes | | ||
| | write.delete.target-file-size-bytes | 67108864 (64 MB) | Controls the size of delete files generated to target about this many bytes | | ||
| | write.distribution-mode | none | Defines distribution of write data: __none__: don't shuffle rows; __hash__: hash distribute by partition key ; __range__: range distribute by partition key or sort key if table has an SortOrder | | ||
| | write.distribution-mode | none. Engines may override this default, for example [Spark](spark-configuration.md#write-options) | Defines distribution of write data: __none__: don't shuffle rows; __hash__: hash distribute by partition key ; __range__: range distribute by partition key or sort key if table has an SortOrder | |
Member
There was a problem hiding this comment.
I didn't really like this description before, but I think the change is good. Might be nice in a follow up to change this since it isn't clear what "distribution of write data" is.
docs/docs/spark-configuration.md
Outdated
| | compression-codec | Table write.(fileformat).compression-codec | Overrides this table's compression codec for this write | | ||
| | compression-level | Table write.(fileformat).compression-level | Overrides this table's compression level for Parquet and Avro tables for this write | | ||
| | compression-strategy | Table write.orc.compression-strategy | Overrides this table's compression strategy for ORC tables for this write | | ||
| | distribution-mode | `Range` if Sort Order Defined ; `Hash` if Partition Defined ; `None` otherwise | Override this table's distribution mode for this write | |
Member
There was a problem hiding this comment.
Can we change this to a link to the spark-writes section? May be more clear
Member
Author
There was a problem hiding this comment.
@RussellSpitzer good point, I just linked from both sections, let me know what you think now?
szehon-ho
force-pushed
the
distribution_mode_doc
branch
from
3d2b21f to
d5aa46a
Compare
szehon-ho
force-pushed
the
distribution_mode_doc
branch
from
d5aa46a to
ab56f03
Compare
RussellSpitzer
approved these changes
Jul 16, 2024
Member
|
Looks good to me now. :) Feel free to merge |
Member
Author
|
Thanks @RussellSpitzer @dramaticlly @ajantha-bhat for reviews! |
jasonf20
pushed a commit
to jasonf20/iceberg
that referenced
this pull request
Aug 4, 2024
zachdisc
pushed a commit
to zachdisc/iceberg
that referenced
this pull request
Dec 23, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In recent releases of Iceberg-Spark, there have been changes to Spark defaults for distribution mode, by changes like: #7637 (and previous ones).
I have seen many questions regarding why an extra shuffle is included in writes, and it is often this change in default. One source of confusion I have seen is the doc mentioning table property write.distribution-mode which says default is NONE, so this PR tries to add more to here to explain this.