Skip to content

Conversation

@maropu
Copy link
Member

@maropu maropu commented Jun 14, 2017

What changes were proposed in this pull request?

This pr added a qualifier for column names in pivot aggregations. #16565 wrongly droped a qualifier for aggregated column names (then, it changed the existing behaviour).

// Spark-v2.1
scala> Seq((1, 2)).toDF("id", "v1").createOrReplaceTempView("s")
scala> Seq((1, 2)).toDF("id", "v2").createOrReplaceTempView("t")
scala> val df1 = sql("SELECT * FROM s")
scala> val df2 = sql("SELECT * FROM t")
scala> df1.join(df2, "id" :: Nil).groupBy("id").pivot("id").max("v1", "v2").show
+---+-------------+-------------+                                               
| id|1_max(s.`v1`)|1_max(t.`v2`)|
+---+-------------+-------------+
|  1|            2|            2|
+---+-------------+-------------+

// Master
scala> df1.join(df2, "id" :: Nil).groupBy("id").pivot("id").max("v1", "v2").show
+---+---------+---------+                                                       
| id|1_max(v1)|1_max(v2)|
+---+---------+---------+
|  1|        2|        2|
+---+---------+---------+

// Master with this pr
scala> df1.join(df2, "id" :: Nil).groupBy("id").pivot("id").max("v1", "v2").show
+---+-----------+-----------+                                                       
| id|1_max(s.v1)|1_max(t.v2)|
+---+-----------+-----------+
|  1|          2|          2|
+---+-----------+-----------+

How was this patch tested?

Added tests in DataFramePivotSuite.

@SparkQA
Copy link

SparkQA commented Jun 14, 2017

Test build #78051 has finished for PR 18302 at commit dbe793d.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jun 15, 2017

Test build #78074 has finished for PR 18302 at commit a6aacd0.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jun 15, 2017

Test build #78099 has finished for PR 18302 at commit b423e6f.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jun 15, 2017

Test build #78100 has finished for PR 18302 at commit a8728f6.

  • This patch fails Scala style tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@SparkQA
Copy link

SparkQA commented Jun 15, 2017

Test build #78101 has finished for PR 18302 at commit d5e0eab.

  • This patch fails Spark unit tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@maropu maropu changed the title [SPARK-17237][SQL][FOLLOWUP] Add a qualifier in pretty expressions [SPARK-17237][SQL][FOLLOWUP][WIP] Add a qualifier in pretty expressions Jun 16, 2017
@maropu
Copy link
Member Author

maropu commented Jun 16, 2017

Since this pr changed the existing behaviour, I'll close this for now.

@maropu maropu closed this Jun 16, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants