[SPARK-19881][SQL] Support Dynamic Partition Inserts params with SET command

[dongjoon-hyun]

What changes were proposed in this pull request?

Since Spark 2.0.0, SET commands do not pass the values to HiveClient. In most case, Spark handles well. However, for the dynamic partition insert, users meet the following misleading situation.

scala> spark.range(1001).selectExpr("id as key", "id as value").registerTempTable("t1001")

scala> sql("create table p (value int) partitioned by (key int)").show

scala> sql("insert into table p partition(key) select key, value from t1001")
org.apache.spark.SparkException:
Dynamic partition strict mode requires at least one static partition column.
To turn this off set hive.exec.dynamic.partition.mode=nonstrict

scala> sql("set hive.exec.dynamic.partition.mode=nonstrict")

scala> sql("insert into table p partition(key) select key, value from t1001")
org.apache.hadoop.hive.ql.metadata.HiveException:
Number of dynamic partitions created is 1001, which is more than 1000.
To solve this try to set hive.exec.max.dynamic.partitions to at least 1001.

scala> sql("set hive.exec.max.dynamic.partitions=1001")

scala> sql("set hive.exec.max.dynamic.partitions").show(false)
+--------------------------------+-----+
|key                             |value|
+--------------------------------+-----+
|hive.exec.max.dynamic.partitions|1001 |
+--------------------------------+-----+

scala> sql("insert into table p partition(key) select key, value from t1001")
org.apache.hadoop.hive.ql.metadata.HiveException:
Number of dynamic partitions created is 1001, which is more than 1000.
To solve this try to set hive.exec.max.dynamic.partitions to at least 1001.

The last error is the same with the previous one. HiveClient does not know new value 1001. While users control hive.exec.dynamic.partition.mode, there is no way to change the default value of hive.exec.max.dynamic.partitions of HiveCilent with SET command.

The root cause is that hive parameters are passed to HiveClient on creating. So, the workaround is to use --hiveconf when starting spark-shell. However, it is still unchangeable in spark-shell. We had better handle this case without misleading error messages ending infinite loop.

How was this patch tested?

Manual.

[SparkQA]

Test build #74247 has started for PR 17223 at commit a874455.

[dongjoon-hyun]

We need only 4 among 6 parameters in https://cwiki.apache.org/confluence/display/Hive/LanguageManual+DML#LanguageManualDML-DynamicPartitionInserts .

[dongjoon-hyun]

Retest this please

[SparkQA]

Test build #74255 has finished for PR 17223 at commit a874455.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Could you review this when you have sometime, @gatorsmile ?

[dongjoon-hyun]

Hi, @cloud-fan .
Could you review this when you have sometime?

[gatorsmile]

Should we do it when users issuing the SET command? Is it a general issue?

[dongjoon-hyun]

That will be the best approach since this is a general issue for all unhandled hive param options. The reason to do this here is that SetCommand lives in sql/core and does not interact with this. Is there a way to invoke runSqlHive there?

[cloud-fan]

Since hive client is shared among all sessions, we can't set hive conf dynamically, to keep session isolation. I think we should treat hive conf as static sql conf, and throw exception when users try to change them.

[dongjoon-hyun]

I see. That's the reason why not to support that. Thank you, @cloud-fan.

[dongjoon-hyun]

I'll close this PR and JIRA issue.