[SPARK-16803] [SQL] SaveAsTable does not work when source DataFrame is built on a Hive Table

[gatorsmile]

What changes were proposed in this pull request?

In Spark 2.0, SaveAsTable does not work when source DataFrame is built on a Hive Table, but Spark 1.6 works.

Spark 1.6

scala> sql("create table sample.sample stored as SEQUENCEFILE as select 1 as key, 'abc' as value")
res2: org.apache.spark.sql.DataFrame = []

scala> val df = sql("select key, value as value from sample.sample")
df: org.apache.spark.sql.DataFrame = [key: int, value: string]

scala> df.write.mode("append").saveAsTable("sample.sample")

scala> sql("select * from sample.sample").show()
+---+-----+
|key|value|
+---+-----+
|  1|  abc|
|  1|  abc|
+---+-----+

Spark 2.0

scala> df.write.mode("append").saveAsTable("sample.sample")
org.apache.spark.sql.AnalysisException: Saving data in MetastoreRelation sample, sample
 is not supported.;

This PR is to provide a support with by-name resolution. In 1.6, it is by-position resolution. The previous behavior is wrong. We need to adjust the order.

How was this patch tested?

Test cases are added

[SparkQA]

Test build #63645 has finished for PR 14612 at commit 71399f1.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #66252 has finished for PR 14612 at commit d2b2d91.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[gatorsmile]

cc @cloud-fan @yhuai

[gatorsmile]

retest this please

[SparkQA]

Test build #68248 has finished for PR 14612 at commit d2b2d91.

• This patch fails to build.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

is it possible that we can extract the similar logic in CreateDataSourceTableAsSelectCommand here?

[cloud-fan]

hi @gatorsmile do you have time to work on it? I'd like to get this into 2.1

[gatorsmile]

Sure, will finish this tomorrow.

[SparkQA]

Test build #68495 has finished for PR 14612 at commit 3035cfe.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

can we consolidate hive and data source table here?

[gatorsmile]

Let me try another way to fix it. Will submit a new PR.

[gatorsmile]

uh... Actually, I found another bug in our write path.