[SPARK-53738][SQL] Fix planned write when query output contains foldable orderings #52584

pan3793 · 2025-10-13T05:48:08Z

What changes were proposed in this pull request?

This is the second try of #52474, following the suggestion from cloud-fan

This PR fixes a bug in plannedWrite, where the query has foldable orderings in the partition columns.

CREATE TABLE t (i INT, j INT, k STRING) USING PARQUET PARTITIONED BY (k);

INSERT OVERWRITE t SELECT j AS i, i AS j, '0' as k FROM t0 SORT BY k, i;

The evaluation of FileFormatWriter.orderingMatched fails because SortOrder(Literal) is eliminated by EliminateSorts.

Why are the changes needed?

V1Writes will override the custom sort order when the query output ordering does not satisfy the required ordering. Before SPARK-53707, when the query's output contains literals in partition columns, the judgment produces a false-negative result, thus causing the sort order not to take effect.

SPARK-53707 partially fixes the issue on the logical plan by adding a Project of query in V1Writes.

Before SPARK-53707

Sort [0 ASC NULLS FIRST, i#280 ASC NULLS FIRST], false
+- Project [j#287 AS i#280, i#286 AS j#281, 0 AS k#282]
   +- Relation spark_catalog.default.t0[i#286,j#287,k#288] parquet

After SPARK-53707

Project [i#284, j#285, 0 AS k#290]
+- Sort [0 ASC NULLS FIRST, i#284 ASC NULLS FIRST], false
   +- Project [i#284, j#285]
      +- Relation spark_catalog.default.t0[i#284,j#285,k#286] parquet

Note, note the issue still exists because there is another place to check the ordering match again in FileFormatWriter.

This PR fixes the issue thoroughly, with new UTs added.

Does this PR introduce any user-facing change?

Yes, it's a bug fix.

How was this patch tested?

New UTs are added.

Was this patch authored or co-authored using generative AI tooling?

No.

…n query output contains literal

pan3793 · 2025-10-13T05:50:04Z

sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala

-        WholeStageCodegenExec(insertInputAdapter(plan))(codegenStageCounter.incrementAndGet())
+        val newId = codegenStageCounter.incrementAndGet()
+        val newPlan = WholeStageCodegenExec(insertInputAdapter(plan))(newId)
+        plan.logicalLink.foreach(newPlan.setLogicalLink)


It appears that WholeStageCodegenExec misses setting logicalLink, is it by design?

interesting, and it never caused issue with AQE before?

Haven't seen the real issues in both production and existing UT.

@cloud-fan if I revert changes in FileFormatWriter.scala, this is not required.

Do you want me to keep it or revert it?

let's revert it to be safe, the logical plan is quite sensitive to AQE. And technically, the CollapseCodegenStages is newly generated at planning phase, it does have have a corresponding logical plan.

@cloud-fan reverted, and thanks for the explanation.

pan3793 · 2025-10-13T05:51:26Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala

+      plan.logicalLink match {
+        case Some(WriteFiles(query, _, _, _, _, _)) =>
+          V1WritesUtils.eliminateFoldableOrdering(ordering, query).outputOrdering
+        case Some(query) =>


the query can be WholeStageCodegenExec, that's why I set logicalLink on WholeStageCodegenExec

pan3793 · 2025-10-13T05:52:34Z

sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/V1WriteCommandSuite.scala


    val listener = new QueryExecutionListener {
      override def onSuccess(funcName: String, qe: QueryExecution, durationNs: Long): Unit = {
+        val conf = qe.sparkSession.sessionState.conf


this is a bugfix, the listener runs in another thread, without this change, conf.getConf actually gets conf from the thread local, thus may cause issues on concurrency running tests

pan3793 · 2025-10-13T06:03:15Z

sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryRelation.scala

+  def withOutput(newOutput: Seq[Attribute]): InMemoryRelation = {
+    val map = AttributeMap(output.zip(newOutput))
+    val newOutputOrdering = outputOrdering
+      .map(_.transform { case a: Attribute => map(a) })
+      .asInstanceOf[Seq[SortOrder]]
+    InMemoryRelation(newOutput, cacheBuilder, newOutputOrdering, statsOfPlanToCache)


issue was identified in previous try, see #52474 (comment)

pan3793 · 2025-10-13T06:03:51Z

sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryRelation.scala


+  override def makeCopy(newArgs: Array[AnyRef]): LogicalPlan = {
+    val copied = super.makeCopy(newArgs).asInstanceOf[InMemoryRelation]
+    copied.statsOfPlanToCache = this.statsOfPlanToCache


ditto, issue was identified in previous try, see #52474 (comment)

pan3793 · 2025-10-13T06:52:59Z

cc @cloud-fan @peter-toth @ulysses-you

pan3793 · 2025-10-13T06:59:15Z

@cloud-fan BTW, the "planned write" switch (an internal config) was added since 3.4, do we have a plan to remove it to simplify code, or tend to preserve it forever?

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/V1Writes.scala

cloud-fan · 2025-10-13T12:12:53Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/V1Writes.scala

    expressions.exists(_.exists(_.isInstanceOf[Empty2Null]))
  }

+  def eliminateFoldableOrdering(ordering: Seq[SortOrder], query: LogicalPlan): LogicalPlan =


let's add comments to explain the reason behind it.

cloud-fan · 2025-10-13T12:13:46Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala

      .getOrElse(materializeAdaptiveSparkPlan(plan))
      .outputOrdering
+
+    val requiredOrdering = {


is this the code path when planned write is disabled?

I think we can leave it unfixed, as this code path is rarely reached and this fix is kind of an optimization: it's only about perf.

it's a necessary change for the "planned write" to make UT happy

if (Utils.isTesting) outputOrderingMatched = orderingMatched

OK this is a necessary for the current codebase, but do we really need to do it in theory? The planned write should have added the sort already, ideally we don't need to try to add sort again here.

The planned write should have added the sort already, ideally we don't need to try to add sort again here.

yes, exactly

peter-toth

LGTM, pending CI.

dongjoon-hyun

+1, LGTM.

cloud-fan · 2025-10-14T01:49:51Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala

+      // columns.
+      val ordering = partitionColumns.drop(numStaticPartitionCols) ++
+        writerBucketSpec.map(_.bucketIdExpression) ++ sortColumns
+      plan.logicalLink match {


I'm a bit worried about this. In AQE we have a fallback to find logical link in the children, so that it's more reliable. Now we have the risk of perf regression if the logical link is not present and we add an extra sort.

Shall we remove the adding sort here completly if planned write is enabled (WriteFiles is present)?

I'm a bit worried about this. In AQE we have a fallback to find logical link in the children, so that it's more reliable.

@cloud-fan do you suggest

- plan.logicalLink match { + plan.logicalLink.orElse { + plan.collectFirst { case p if p.logicalLink.isDefined => p.logicalLink.get } + } match {

Shall we remove the adding sort here completly if planned write is enabled (WriteFiles is present)?

I think the current code has already satisfied your expectation, when planned write is enabled:

if concurrent writer is disabled, the calculated required ordering won't be used.

if concurrent writer is enabled, the calculated required ordering is only used in the concurrent writer step 2.

spark/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatDataWriter.scala

Lines 393 to 406 in 29434ea

/**

* Dynamic partition writer with concurrent writers, meaning multiple concurrent writers are opened

* for writing.

*

* The process has the following steps:

* - Step 1: Maintain a map of output writers per each partition and/or bucket columns. Keep all

* writers opened and write rows one by one.

* - Step 2: If number of concurrent writers exceeds limit, sort rest of rows on partition and/or

* bucket column(s). Write rows one by one, and eagerly close the writer when finishing

* each partition and/or bucket.

*

* Caller is expected to call `writeWithIterator()` instead of `write()` to write records.

*/

class DynamicPartitionDataConcurrentWriter(

@cloud-fan I have updated the code to fallback to find logical link in the children, then setLogicalLink for WholeStageCodegenExec is unnecessary for this PR, please let me know if you want me to keep it or restore it.

cloud-fan · 2025-10-15T18:18:22Z

what happens if we don't modify FileFormatWriter.scala at all? I think it only affects non-planned-write and planned-write with concurrent write, and we can improve them later. (ideally the sort should be handled at the logical plan phase)

I think the only issue is for test: if (Utils.isTesting) outputOrderingMatched = orderingMatched. We can fix the affected tests and specify the required ordering explicitly.

pan3793 · 2025-10-15T18:44:34Z

@cloud-fan I agree with your summary. Only the newly added tests are affected if I don't touch FileFormatWriter.scala, so the simplest way is to skip checking orderingMatched temporarily for the new tests.

Have updated the code, please take another look.

pan3793 · 2025-10-17T08:30:31Z

Kindly ping @cloud-fan, do you have further concerns with this PR?

cloud-fan

LGTM, only one comment: https://github.com/apache/spark/pull/52584/files#r2441550259

pan3793 · 2025-10-21T13:19:10Z

@cloud-fan I have addressed your last comment.

@peter-toth @dongjoon-hyun Can anyone help merge this PR?

…ble orderings ### What changes were proposed in this pull request? This is the second try of #52474, following [the suggestion from cloud-fan](#52474 (comment)) This PR fixes a bug in `plannedWrite`, where the `query` has foldable orderings in the partition columns. ``` CREATE TABLE t (i INT, j INT, k STRING) USING PARQUET PARTITIONED BY (k); INSERT OVERWRITE t SELECT j AS i, i AS j, '0' as k FROM t0 SORT BY k, i; ``` The evaluation of `FileFormatWriter.orderingMatched` fails because `SortOrder(Literal)` is eliminated by `EliminateSorts`. ### Why are the changes needed? `V1Writes` will override the custom sort order when the query output ordering does not satisfy the required ordering. Before SPARK-53707, when the query's output contains literals in partition columns, the judgment produces a false-negative result, thus causing the sort order not to take effect. SPARK-53707 partially fixes the issue on the logical plan by adding a `Project` of query in `V1Writes`. Before SPARK-53707 ``` Sort [0 ASC NULLS FIRST, i#280 ASC NULLS FIRST], false +- Project [j#287 AS i#280, i#286 AS j#281, 0 AS k#282] +- Relation spark_catalog.default.t0[i#286,j#287,k#288] parquet ``` After SPARK-53707 ``` Project [i#284, j#285, 0 AS k#290] +- Sort [0 ASC NULLS FIRST, i#284 ASC NULLS FIRST], false +- Project [i#284, j#285] +- Relation spark_catalog.default.t0[i#284,j#285,k#286] parquet ``` Note, note the issue still exists because there is another place to check the ordering match again in `FileFormatWriter`. This PR fixes the issue thoroughly, with new UTs added. ### Does this PR introduce _any_ user-facing change? Yes, it's a bug fix. ### How was this patch tested? New UTs are added. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #52584 from pan3793/SPARK-53738-rework. Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Peter Toth <peter.toth@gmail.com> (cherry picked from commit f33d8aa) Signed-off-by: Peter Toth <peter.toth@gmail.com>

peter-toth · 2025-10-21T13:28:52Z

Thanks @pan3793 for the fix and @cloud-fan and @dongjoon-hyun for the review!

Merged to master (4.1.0) and branch-4.0 (4.0.2).

@pan3793 , there were conflicts with branch-3.5. Can you please open a separete PR for that branch?

pan3793 · 2025-10-21T13:59:18Z

@peter-toth seems SPARK-46485 didn't land on branch-3.5, I think we should backport that first then this one, or neither.

peter-toth · 2025-10-21T14:47:42Z

Seem like SPARK-46485 was not needed for 3.5.x because SPARK-46378 had never landed in it, so probably we don't need this PR either.

But let's recover branch-4.0 compilation with your fix #52683 first.

pan3793 · 2025-10-21T21:17:36Z

I take a closer look at branch-3.5 - I confirm this issue also affects branch-3.5 by only porting the UT (it fails). SPARK-46485 actually fixes a hidden bug (until exposed by SPARK-46378) that has existed since 3.4.

Therefore, I think we should backport SPARK-46485 and this one. @peter-toth WDYT?

dongjoon-hyun · 2025-10-21T21:21:19Z

For SPARK-46485, please ping there (in the following PR) once more again, @pan3793 , because there are more audience there.

[SPARK-46485][SQL] V1Write should not add Sort when not needed #44458

pan3793 · 2025-10-21T21:26:45Z

@dongjoon-hyun thanks for the reminder.

peter-toth · 2025-10-22T08:07:13Z

I take a closer look at branch-3.5 - I confirm this issue also affects branch-3.5 by only porting the UT (it fails). SPARK-46485 actually fixes a hidden bug (until exposed by SPARK-46378) that has existed since 3.4.

Yeah, IMO in that case it makes sense to backport. Especially that this PR fixes 2 other, so far hidden issues (#52584 (comment), #52584 (comment)).

…ble orderings ### What changes were proposed in this pull request? This is the second try of apache#52474, following [the suggestion from cloud-fan](apache#52474 (comment)) This PR fixes a bug in `plannedWrite`, where the `query` has foldable orderings in the partition columns. ``` CREATE TABLE t (i INT, j INT, k STRING) USING PARQUET PARTITIONED BY (k); INSERT OVERWRITE t SELECT j AS i, i AS j, '0' as k FROM t0 SORT BY k, i; ``` The evaluation of `FileFormatWriter.orderingMatched` fails because `SortOrder(Literal)` is eliminated by `EliminateSorts`. ### Why are the changes needed? `V1Writes` will override the custom sort order when the query output ordering does not satisfy the required ordering. Before SPARK-53707, when the query's output contains literals in partition columns, the judgment produces a false-negative result, thus causing the sort order not to take effect. SPARK-53707 partially fixes the issue on the logical plan by adding a `Project` of query in `V1Writes`. Before SPARK-53707 ``` Sort [0 ASC NULLS FIRST, i#280 ASC NULLS FIRST], false +- Project [j#287 AS i#280, i#286 AS j#281, 0 AS k#282] +- Relation spark_catalog.default.t0[i#286,j#287,k#288] parquet ``` After SPARK-53707 ``` Project [i#284, j#285, 0 AS k#290] +- Sort [0 ASC NULLS FIRST, i#284 ASC NULLS FIRST], false +- Project [i#284, j#285] +- Relation spark_catalog.default.t0[i#284,j#285,k#286] parquet ``` Note, note the issue still exists because there is another place to check the ordering match again in `FileFormatWriter`. This PR fixes the issue thoroughly, with new UTs added. ### Does this PR introduce _any_ user-facing change? Yes, it's a bug fix. ### How was this patch tested? New UTs are added. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#52584 from pan3793/SPARK-53738-rework. Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Peter Toth <peter.toth@gmail.com> (cherry picked from commit f33d8aa) Signed-off-by: Peter Toth <peter.toth@gmail.com>

…foldable orderings Backport #52584 to branch-3.5 ### What changes were proposed in this pull request? This is the second try of #52474, following [the suggestion from cloud-fan](#52474 (comment)) This PR fixes a bug in `plannedWrite`, where the `query` has foldable orderings in the partition columns. ``` CREATE TABLE t (i INT, j INT, k STRING) USING PARQUET PARTITIONED BY (k); INSERT OVERWRITE t SELECT j AS i, i AS j, '0' as k FROM t0 SORT BY k, i; ``` The evaluation of `FileFormatWriter.orderingMatched` fails because `SortOrder(Literal)` is eliminated by `EliminateSorts`. ### Why are the changes needed? `V1Writes` will override the custom sort order when the query output ordering does not satisfy the required ordering. Before SPARK-53707, when the query's output contains literals in partition columns, the judgment produces a false-negative result, thus causing the sort order not to take effect. SPARK-53707 partially fixes the issue on the logical plan by adding a `Project` of query in `V1Writes`. Before SPARK-53707 ``` Sort [0 ASC NULLS FIRST, i#280 ASC NULLS FIRST], false +- Project [j#287 AS i#280, i#286 AS j#281, 0 AS k#282] +- Relation spark_catalog.default.t0[i#286,j#287,k#288] parquet ``` After SPARK-53707 ``` Project [i#284, j#285, 0 AS k#290] +- Sort [0 ASC NULLS FIRST, i#284 ASC NULLS FIRST], false +- Project [i#284, j#285] +- Relation spark_catalog.default.t0[i#284,j#285,k#286] parquet ``` Note, note the issue still exists because there is another place to check the ordering match again in `FileFormatWriter`. This PR fixes the issue thoroughly, with new UTs added. ### Does this PR introduce _any_ user-facing change? Yes, it's a bug fix. ### How was this patch tested? New UTs are added. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #52697 from pan3793/SPARK-53694-3.5. Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Peter Toth <peter.toth@gmail.com>

…ble orderings ### What changes were proposed in this pull request? This is the second try of apache#52474, following [the suggestion from cloud-fan](apache#52474 (comment)) This PR fixes a bug in `plannedWrite`, where the `query` has foldable orderings in the partition columns. ``` CREATE TABLE t (i INT, j INT, k STRING) USING PARQUET PARTITIONED BY (k); INSERT OVERWRITE t SELECT j AS i, i AS j, '0' as k FROM t0 SORT BY k, i; ``` The evaluation of `FileFormatWriter.orderingMatched` fails because `SortOrder(Literal)` is eliminated by `EliminateSorts`. ### Why are the changes needed? `V1Writes` will override the custom sort order when the query output ordering does not satisfy the required ordering. Before SPARK-53707, when the query's output contains literals in partition columns, the judgment produces a false-negative result, thus causing the sort order not to take effect. SPARK-53707 partially fixes the issue on the logical plan by adding a `Project` of query in `V1Writes`. Before SPARK-53707 ``` Sort [0 ASC NULLS FIRST, i#280 ASC NULLS FIRST], false +- Project [j#287 AS i#280, i#286 AS j#281, 0 AS k#282] +- Relation spark_catalog.default.t0[i#286,j#287,k#288] parquet ``` After SPARK-53707 ``` Project [i#284, j#285, 0 AS k#290] +- Sort [0 ASC NULLS FIRST, i#284 ASC NULLS FIRST], false +- Project [i#284, j#285] +- Relation spark_catalog.default.t0[i#284,j#285,k#286] parquet ``` Note, note the issue still exists because there is another place to check the ordering match again in `FileFormatWriter`. This PR fixes the issue thoroughly, with new UTs added. ### Does this PR introduce _any_ user-facing change? Yes, it's a bug fix. ### How was this patch tested? New UTs are added. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#52584 from pan3793/SPARK-53738-rework. Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Peter Toth <peter.toth@gmail.com> (cherry picked from commit 6289672) Signed-off-by: Peter Toth <peter.toth@gmail.com>

LuciferYang · 2025-11-18T09:03:56Z

...ive/src/test/scala/org/apache/spark/sql/hive/execution/command/V1WriteHiveCommandSuite.scala

+
+  test("v1 write to hive table with sort by literal column preserve custom order") {
+    withCovnertMetastore { _ =>
+      withPlannedWrite { enabled =>


@pan3793 This enabled is not actually being utilized in the test, so does this test case really need to be wrapped with withPlannedWrite? This will result in the test content being executed twice.

it's correct that the variable enabled is not referenced, but the wrapper withPlannedWrite is indeed required, we should ensure preserving the user-provided sort w/ and w/o enabling planned write.

…ble orderings ### What changes were proposed in this pull request? This is the second try of apache#52474, following [the suggestion from cloud-fan](apache#52474 (comment)) This PR fixes a bug in `plannedWrite`, where the `query` has foldable orderings in the partition columns. ``` CREATE TABLE t (i INT, j INT, k STRING) USING PARQUET PARTITIONED BY (k); INSERT OVERWRITE t SELECT j AS i, i AS j, '0' as k FROM t0 SORT BY k, i; ``` The evaluation of `FileFormatWriter.orderingMatched` fails because `SortOrder(Literal)` is eliminated by `EliminateSorts`. ### Why are the changes needed? `V1Writes` will override the custom sort order when the query output ordering does not satisfy the required ordering. Before SPARK-53707, when the query's output contains literals in partition columns, the judgment produces a false-negative result, thus causing the sort order not to take effect. SPARK-53707 partially fixes the issue on the logical plan by adding a `Project` of query in `V1Writes`. Before SPARK-53707 ``` Sort [0 ASC NULLS FIRST, i#280 ASC NULLS FIRST], false +- Project [j#287 AS i#280, i#286 AS j#281, 0 AS k#282] +- Relation spark_catalog.default.t0[i#286,j#287,k#288] parquet ``` After SPARK-53707 ``` Project [i#284, j#285, 0 AS k#290] +- Sort [0 ASC NULLS FIRST, i#284 ASC NULLS FIRST], false +- Project [i#284, j#285] +- Relation spark_catalog.default.t0[i#284,j#285,k#286] parquet ``` Note, note the issue still exists because there is another place to check the ordering match again in `FileFormatWriter`. This PR fixes the issue thoroughly, with new UTs added. ### Does this PR introduce _any_ user-facing change? Yes, it's a bug fix. ### How was this patch tested? New UTs are added. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#52584 from pan3793/SPARK-53738-rework. Authored-by: Cheng Pan <chengpan@apache.org> Signed-off-by: Peter Toth <peter.toth@gmail.com>

[SPARK-53738][SQL] PlannedWrite should preserve custom sort order whe…

2c66ba9

…n query output contains literal

github-actions bot added the SQL label Oct 13, 2025

pan3793 commented Oct 13, 2025

View reviewed changes

fix codegenStageCounter

a1bf09e

peter-toth reviewed Oct 13, 2025

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala Outdated Show resolved Hide resolved

peter-toth reviewed Oct 13, 2025

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/V1Writes.scala Outdated Show resolved Hide resolved

peter-toth reviewed Oct 13, 2025

View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/V1Writes.scala Outdated Show resolved Hide resolved

cloud-fan reviewed Oct 13, 2025

View reviewed changes

pan3793 added 2 commits October 13, 2025 20:45

address comments

2384ce8

nit

fd401c0

peter-toth approved these changes Oct 13, 2025

View reviewed changes

dongjoon-hyun approved these changes Oct 13, 2025

View reviewed changes

cloud-fan reviewed Oct 14, 2025

View reviewed changes

fallback get logicalLink from children

7129fb2

pan3793 requested a review from cloud-fan October 15, 2025 03:15

revert change of FileFormatWriter

f1df0ec

revert setting logical link on WholeStageCodegenExec

531c6bf

cloud-fan approved these changes Oct 18, 2025

View reviewed changes

pan3793 requested a review from cloud-fan October 20, 2025 12:33

peter-toth closed this in f33d8aa Oct 21, 2025

pan3793 mentioned this pull request Oct 21, 2025

[SPARK-46485][SQL] V1Write should not add Sort when not needed #44458

Closed

pan3793 mentioned this pull request Oct 22, 2025

[SPARK-53738][SQL][3.5] Fix planned write when query output contains foldable orderings #52697

Closed

peter-toth mentioned this pull request Oct 22, 2025

[SPARK-46485][SQL][3.5] V1Write should not add Sort when not needed #52692

Closed

LuciferYang reviewed Nov 18, 2025

View reviewed changes

	/**
	* Dynamic partition writer with concurrent writers, meaning multiple concurrent writers are opened
	* for writing.
	*
	* The process has the following steps:
	* - Step 1: Maintain a map of output writers per each partition and/or bucket columns. Keep all
	* writers opened and write rows one by one.
	* - Step 2: If number of concurrent writers exceeds limit, sort rest of rows on partition and/or
	* bucket column(s). Write rows one by one, and eagerly close the writer when finishing
	* each partition and/or bucket.
	*
	* Caller is expected to call `writeWithIterator()` instead of `write()` to write records.
	*/
	class DynamicPartitionDataConcurrentWriter(

[SPARK-53738][SQL] Fix planned write when query output contains foldable orderings #52584

[SPARK-53738][SQL] Fix planned write when query output contains foldable orderings #52584

Uh oh!

Conversation

pan3793 commented Oct 13, 2025

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pan3793 commented Oct 13, 2025

Uh oh!

pan3793 commented Oct 13, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pan3793 Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

peter-toth left a comment

Choose a reason for hiding this comment

Uh oh!

dongjoon-hyun left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pan3793 Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cloud-fan commented Oct 15, 2025

Uh oh!

pan3793 commented Oct 15, 2025

Uh oh!

pan3793 commented Oct 17, 2025

Uh oh!

cloud-fan left a comment

Choose a reason for hiding this comment

Uh oh!

pan3793 commented Oct 21, 2025

Uh oh!

peter-toth commented Oct 21, 2025

pan3793 Oct 13, 2025 •

edited

Loading

pan3793 Oct 14, 2025 •

edited

Loading

pan3793 commented Oct 21, 2025 •

edited

Loading

peter-toth commented Oct 21, 2025 •

edited

Loading

dongjoon-hyun commented Oct 21, 2025 •

edited

Loading

peter-toth commented Oct 22, 2025 •

edited

Loading

pan3793 Nov 18, 2025 •

edited

Loading