feat: Make `FanoutWriter` writer configurable by jonathanc-n · Pull Request #1872 · apache/iceberg-rust

jonathanc-n · 2025-11-19T05:32:29Z

Which issue does this PR close?

Closes Make fanout/clustered mode configurable for datafusion write #1834.

What changes are included in this PR?

Adds PROPERTY_WRITE_FANOUT_ENABLED to TableProperties

I made it write.fanout.enabled instead of write.fanout-enabled in case there is anything we want configurable with FanoutWriter in the future.

Are these changes tested?

Yes by existing tests.

jonathanc-n · 2025-11-19T05:32:39Z

cc @CTTY

CTTY

Thanks for working on this! I've left some minor comments

CTTY · 2025-11-19T15:36:59Z

+
+    /// Whether to use `FanoutWriter` for partitioned tables (handles unsorted data).
+    /// If false, uses `ClusteredWriter` (requires sorted data, more memory efficient).
+    pub const PROPERTY_WRITE_FANOUT_ENABLED: &str = "write.fanout.enabled";


I think write.datafusion.fanout.enabled will be a better property name since Java has

public static final String SPARK_WRITE_PARTITIONED_FANOUT_ENABLED = "write.spark.fanout.enabled";

Suggested change

pub const PROPERTY_WRITE_FANOUT_ENABLED: &str = "write.fanout.enabled";

pub const PROPERTY_DATAFUSION_WRITE_FANOUT_ENABLED: &str = "write.datafusion.fanout.enabled";

CTTY · 2025-11-19T15:40:49Z

    /// The default format for files.
    pub write_format_default: String,
    /// The target file size for files.
    pub write_target_file_size_bytes: usize,


We need to add the property here as well

I was thinking about this, I think whether or not to place it in the TableProperty struct is currently a bit inconsistent. write_format_default and write_target_file_size_bytes don't seem to be called by table_property.<property> and are manually parsed in write.rs. I'm not sure why they are parsed differently, but should we standardize one way to do it?

The different ways are currently calling try_from or parsing the way it is done in this pull request.

write_format_default and write_target_file_size_bytes don't seem to be called by table_property. and are manually parsed in write.rs

Yes, this is because table.metadata().properties() is actually &HashMap<String, String>, not a TableProperties instance. There is an issue tracking this: #1878

Once the issue above is fixed, we should definitely start thinking about standardize the way of fetching property.

All of these can be treated as a separate issue imo, and we should still add the new properties to TableProperties struct

Should be good for another look!

CTTY

Just one minor change, otherwise looks good to me!

CTTY · 2025-11-24T03:20:27Z

+
+    /// Whether to use `FanoutWriter` for partitioned tables (handles unsorted data).
+    /// If false, uses `ClusteredWriter` (requires sorted data, more memory efficient).
+    pub const PROPERTY_WRITE_FANOUT_ENABLED: &str = "write.fanout.enabled";


Suggested change

pub const PROPERTY_WRITE_FANOUT_ENABLED: &str = "write.fanout.enabled";

pub const PROPERTY_DATAFUSION_WRITE_FANOUT_ENABLED: &str = "write.datafusion.fanout.enabled";

jonathanc-n · 2025-11-25T20:23:35Z

@CTTY I added it and to the default const

CTTY

LGTM! Looks like the CI is being flaky tho

jonathanc-n · 2025-11-27T18:36:19Z

Yes tests seem to fail non deterministically. cc @liurenjie1024 good to go

liurenjie1024 · 2025-12-04T08:13:40Z

    /// The target file size for files.
    pub write_target_file_size_bytes: usize,
+    /// Whether to use `FanoutWriter` for partitioned tables.
+    pub write_fanout_enabled: bool,


Suggested change

pub write_fanout_enabled: bool,

pub write_datafusion_fanout_enabled: bool,

liurenjie1024 · 2025-12-04T08:14:59Z

+                    .map_err(|e| {
+                        Error::new(
+                            ErrorKind::DataInvalid,
+                            "Invalid value for write.fanout.enabled, expected 'true' or 'false'",


Suggested change

"Invalid value for write.fanout.enabled, expected 'true' or 'false'",

"Invalid value for {}, expected 'true' or 'false'", WRITE_DATAFUSION_FANOUT_ENABLED

CTTY · 2025-12-24T00:54:23Z

Hi @jonathanc-n , could you address the comments that Renjie left? Otherwise, I'd be happy to drive this PR to finish

## Which issue does this PR close? - Closes #1834. ## What changes are included in this PR? - Fellow on #1872. ## Are these changes tested? --------- Signed-off-by: StandingMan <jmtangcs@gmail.com>

github-actions · 2026-02-27T00:24:36Z

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

github-actions · 2026-03-06T00:27:54Z

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

feat: Make FanoutWriter writer configurable

7f90197

CTTY reviewed Nov 19, 2025

View reviewed changes

jonathanc-n added 2 commits November 20, 2025 15:37

Add to TableProperties struct

4308d09

Merge branch 'main' into add-fanout_enabled-to-table-properties

e8ec13c

jonathanc-n mentioned this pull request Nov 20, 2025

Use TableProperties in TableMetadata #1878

Closed

jonathanc-n requested a review from CTTY November 23, 2025 17:57

CTTY requested changes Nov 24, 2025

View reviewed changes

jonathanc-n added 2 commits November 25, 2025 15:23

add Datafusion

98da346

Merge branch 'main' into add-fanout_enabled-to-table-properties

1558dcd

CTTY approved these changes Nov 27, 2025

View reviewed changes

liurenjie1024 reviewed Dec 4, 2025

View reviewed changes

CTTY mentioned this pull request Dec 8, 2025

Implement insert_into for IcebergTableProvider #1540

Closed

6 tasks

Standing-Man mentioned this pull request Dec 25, 2025

feat: make FanoutWriter writer configurable #1962

Merged

github-actions Bot added the stale label Feb 27, 2026

github-actions Bot closed this Mar 6, 2026

CTTY reopened this Mar 6, 2026

CTTY closed this Mar 6, 2026

	pub const PROPERTY_WRITE_FANOUT_ENABLED: &str = "write.fanout.enabled";
	pub const PROPERTY_DATAFUSION_WRITE_FANOUT_ENABLED: &str = "write.datafusion.fanout.enabled";

	pub write_fanout_enabled: bool,
	pub write_datafusion_fanout_enabled: bool,

	"Invalid value for write.fanout.enabled, expected 'true' or 'false'",
	"Invalid value for {}, expected 'true' or 'false'", WRITE_DATAFUSION_FANOUT_ENABLED

Conversation

jonathanc-n commented Nov 19, 2025

Which issue does this PR close?

What changes are included in this PR?

Are these changes tested?

Uh oh!

jonathanc-n commented Nov 19, 2025

Uh oh!

CTTY left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CTTY left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jonathanc-n commented Nov 25, 2025

Uh oh!

CTTY left a comment

Choose a reason for hiding this comment

Uh oh!

jonathanc-n commented Nov 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

CTTY commented Dec 24, 2025

Uh oh!

github-actions Bot commented Feb 27, 2026

Uh oh!

github-actions Bot commented Mar 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants