[SPARK-5658][SQL] Finalize DDL and write support APIs #4446

yhuai · 2015-02-06T23:36:58Z

https://issues.apache.org/jira/browse/SPARK-5658

SparkQA · 2015-02-06T23:37:38Z

Test build #26973 has started for PR 4446 at commit b1e9b1b.

This patch merges cleanly.

SparkQA · 2015-02-06T23:37:44Z

Test build #26973 has finished for PR 4446 at commit b1e9b1b.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public class SaveModes
- protected[sql] class CaseInsensitiveMap(map: Map[String, String]) extends Map[String, String]
- abstract class SaveMode

AmplabJenkins · 2015-02-06T23:37:45Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26973/
Test FAILed.

yhuai · 2015-02-07T04:49:05Z

test this please

SparkQA · 2015-02-07T04:52:55Z

Test build #26988 has started for PR 4446 at commit b1e9b1b.

This patch merges cleanly.

SparkQA · 2015-02-07T04:53:00Z

Test build #26988 has finished for PR 4446 at commit b1e9b1b.

This patch fails RAT tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public class SaveModes
- protected[sql] class CaseInsensitiveMap(map: Map[String, String]) extends Map[String, String]
- abstract class SaveMode

AmplabJenkins · 2015-02-07T04:53:01Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26988/
Test FAILed.

SparkQA · 2015-02-07T19:07:52Z

Test build #27009 has started for PR 4446 at commit e702386.

This patch merges cleanly.

SparkQA · 2015-02-07T19:47:48Z

Test build #27009 has finished for PR 4446 at commit e702386.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds the following public classes (experimental):
- public class SaveModes
- protected[sql] class CaseInsensitiveMap(map: Map[String, String]) extends Map[String, String]
- abstract class SaveMode

AmplabJenkins · 2015-02-07T19:47:51Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27009/
Test FAILed.

SparkQA · 2015-02-08T05:52:57Z

Test build #27030 has started for PR 4446 at commit 7db95ff.

This patch merges cleanly.

SparkQA · 2015-02-08T06:04:15Z

Test build #27030 has finished for PR 4446 at commit 7db95ff.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-08T06:04:18Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27030/
Test FAILed.

SparkQA · 2015-02-09T01:58:00Z

Test build #27065 has started for PR 4446 at commit cf5703d.

This patch merges cleanly.

rxin · 2015-02-09T02:08:14Z

sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala

add "(Scala-specific)" to the beginning.

Conflicts: sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala

SparkQA · 2015-02-10T21:27:28Z

Test build #27232 has started for PR 4446 at commit 2306f93.

This patch merges cleanly.

SparkQA · 2015-02-10T21:40:50Z

Test build #27227 has finished for PR 4446 at commit ae4649e.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-10T21:40:54Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27227/
Test FAILed.

…hon tests.

SparkQA · 2015-02-10T22:27:58Z

Test build #27235 has started for PR 4446 at commit 225ff71.

This patch merges cleanly.

SparkQA · 2015-02-10T22:45:17Z

Test build #27232 has finished for PR 4446 at commit 2306f93.

This patch fails PySpark unit tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-10T22:45:20Z

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27232/
Test FAILed.

davies · 2015-02-10T23:28:38Z

python/pyspark/sql/dataframe.py

I think tableName could be just name.

I am not sure if we can do it. tableName has been used at lots of places.

I saw it's only used int insertInto() and saveAsTable(), and registerAsTable() use name.

It's not a keyword argument, so it's safe to change the name.

davies · 2015-02-10T23:34:18Z

@yhuai After fix the failed test, and some code style issues (good to have), the Python parts look good to me.

SparkQA · 2015-02-10T23:52:27Z

Test build #27244 has started for PR 4446 at commit f3a96f7.

This patch merges cleanly.

SparkQA · 2015-02-11T00:14:08Z

Test build #27235 has finished for PR 4446 at commit 225ff71.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-11T00:14:12Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27235/
Test PASSed.

marmbrus · 2015-02-11T01:16:00Z

sql/hive/src/main/scala/org/apache/spark/sql/hive/execution/commands.scala

this error is going to print even if its a SQL CTAS right? that might be confusing.

SparkQA · 2015-02-11T01:29:47Z

Test build #27244 has finished for PR 4446 at commit f3a96f7.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-11T01:29:51Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27244/
Test PASSed.

https://issues.apache.org/jira/browse/SPARK-5658 Author: Yin Huai <yhuai@databricks.com> This patch had conflicts when merged, resolved by Committer: Michael Armbrust <michael@databricks.com> Closes #4446 from yhuai/writeSupportFollowup and squashes the following commits: f3a96f7 [Yin Huai] davies's comments. 225ff71 [Yin Huai] Use Scala TestHiveContext to initialize the Python HiveContext in Python tests. 2306f93 [Yin Huai] Style. 2091fcd [Yin Huai] Merge remote-tracking branch 'upstream/master' into writeSupportFollowup 537e28f [Yin Huai] Correctly clean up temp data. ae4649e [Yin Huai] Fix Python test. 609129c [Yin Huai] Doc format. 92b6659 [Yin Huai] Python doc and other minor updates. cbc717f [Yin Huai] Rename dataSourceName to source. d1c12d3 [Yin Huai] No need to delete the duplicate rule since it has been removed in master. 22cfa70 [Yin Huai] Merge remote-tracking branch 'upstream/master' into writeSupportFollowup d91ecb8 [Yin Huai] Fix test. 4c76d78 [Yin Huai] Simplify APIs. 3abc215 [Yin Huai] Merge remote-tracking branch 'upstream/master' into writeSupportFollowup 0832ce4 [Yin Huai] Fix test. 98e7cdb [Yin Huai] Python style. 2bf44ef [Yin Huai] Python APIs. c204967 [Yin Huai] Format a10223d [Yin Huai] Merge remote-tracking branch 'upstream/master' into writeSupportFollowup 9ff97d8 [Yin Huai] Add SaveMode to saveAsTable. 9b6e570 [Yin Huai] Update doc. c2be775 [Yin Huai] Merge remote-tracking branch 'upstream/master' into writeSupportFollowup 99950a2 [Yin Huai] Use Java enum for SaveMode. 4679665 [Yin Huai] Remove duplicate rule. 77d89dc [Yin Huai] Update doc. e04d908 [Yin Huai] Move import and add (Scala-specific) to scala APIs. cf5703d [Yin Huai] Add checkAnswer to Java tests. 7db95ff [Yin Huai] Merge remote-tracking branch 'upstream/master' into writeSupportFollowup 6dfd386 [Yin Huai] Add java test. f2f33ef [Yin Huai] Fix test. e702386 [Yin Huai] Apache header. b1e9b1b [Yin Huai] Format. ed4e1b4 [Yin Huai] Merge remote-tracking branch 'upstream/master' into writeSupportFollowup af9e9b3 [Yin Huai] DDL and write support API followup. 2a6213a [Yin Huai] Update API names. e6a0b77 [Yin Huai] Update test. 43bae01 [Yin Huai] Remove createTable from HiveContext. 5ffc372 [Yin Huai] Add more load APIs to SQLContext. 5390743 [Yin Huai] Add more save APIs to DataFrame. (cherry picked from commit aaf50d0) Signed-off-by: Michael Armbrust <michael@databricks.com>

watermen · 2015-02-11T03:14:24Z

sql/core/src/test/java/org/apache/spark/sql/sources/JavaSaveLoadSuite.java

Maybe need to use List<StructField> fields = new ArrayList<StructField>(); instead of List<StructField> fields = new ArrayList<>();

We have fixed it. Thank you!

yhuai added 8 commits February 5, 2015 20:24

Add more save APIs to DataFrame.

5390743

Add more load APIs to SQLContext.

5ffc372

Remove createTable from HiveContext.

43bae01

Update test.

e6a0b77

Update API names.

2a6213a

DDL and write support API followup.

af9e9b3

Merge remote-tracking branch 'upstream/master' into writeSupportFollowup

ed4e1b4

Format.

b1e9b1b

Apache header.

e702386

yhuai added 3 commits February 7, 2015 20:36

Fix test.

f2f33ef

Add java test.

6dfd386

Merge remote-tracking branch 'upstream/master' into writeSupportFollowup

7db95ff

Add checkAnswer to Java tests.

cf5703d

rxin reviewed Feb 9, 2015
View reviewed changes

sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala Outdated

Copy link

Contributor

rxin Feb 9, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add "(Scala-specific)" to the beginning.

yhuai added 2 commits February 10, 2015 13:18

Merge remote-tracking branch 'upstream/master' into writeSupportFollowup

2091fcd

Conflicts: sql/core/src/main/scala/org/apache/spark/sql/DataFrame.scala

Style.

2306f93

Use Scala TestHiveContext to initialize the Python HiveContext in Pyt…

225ff71

…hon tests.

davies reviewed Feb 10, 2015
View reviewed changes

davies's comments.

f3a96f7

marmbrus reviewed Feb 11, 2015
View reviewed changes

asfgit closed this in aaf50d0 Feb 11, 2015

watermen reviewed Feb 11, 2015
View reviewed changes

yhuai deleted the writeSupportFollowup branch February 11, 2015 18:20

[SPARK-5658][SQL] Finalize DDL and write support APIs #4446

[SPARK-5658][SQL] Finalize DDL and write support APIs #4446

Uh oh!

Conversation

yhuai commented Feb 6, 2015

Uh oh!

SparkQA commented Feb 6, 2015

Uh oh!

SparkQA commented Feb 6, 2015

Uh oh!

AmplabJenkins commented Feb 6, 2015

Uh oh!

yhuai commented Feb 7, 2015

Uh oh!

SparkQA commented Feb 7, 2015

Uh oh!

SparkQA commented Feb 7, 2015

Uh oh!

AmplabJenkins commented Feb 7, 2015

Uh oh!

SparkQA commented Feb 7, 2015

Uh oh!

SparkQA commented Feb 7, 2015

Uh oh!

AmplabJenkins commented Feb 7, 2015

Uh oh!

SparkQA commented Feb 8, 2015

Uh oh!

SparkQA commented Feb 8, 2015

Uh oh!

AmplabJenkins commented Feb 8, 2015

Uh oh!

SparkQA commented Feb 9, 2015

Uh oh!

rxin Feb 9, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Feb 10, 2015

Uh oh!

SparkQA commented Feb 10, 2015

Uh oh!

AmplabJenkins commented Feb 10, 2015

Uh oh!

SparkQA commented Feb 10, 2015

Uh oh!

SparkQA commented Feb 10, 2015

Uh oh!

AmplabJenkins commented Feb 10, 2015

Uh oh!

davies Feb 10, 2015

Choose a reason for hiding this comment

Uh oh!

yhuai Feb 10, 2015

Choose a reason for hiding this comment

Uh oh!

davies Feb 10, 2015

Choose a reason for hiding this comment

Uh oh!

davies commented Feb 10, 2015

Uh oh!

SparkQA commented Feb 10, 2015

Uh oh!

SparkQA commented Feb 11, 2015

Uh oh!

AmplabJenkins commented Feb 11, 2015

Uh oh!

marmbrus Feb 11, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Feb 11, 2015

Uh oh!

AmplabJenkins commented Feb 11, 2015

Uh oh!

watermen Feb 11, 2015

Choose a reason for hiding this comment

Uh oh!

yhuai Feb 11, 2015

Choose a reason for hiding this comment

Uh oh!

Reviewers