From 898172b239418213bc0d03b6b31495f251507f80 Mon Sep 17 00:00:00 2001 From: Kent Yao Date: Tue, 8 Dec 2020 02:35:00 +0800 Subject: [PATCH 1/5] [SPARK-33641][SQL][DOC][FOLLOW-UP] Add migration guide for CHAR VARCHAR types --- docs/sql-migration-guide.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md index 2c86e7a932637..6a264ccc70126 100644 --- a/docs/sql-migration-guide.md +++ b/docs/sql-migration-guide.md @@ -54,6 +54,9 @@ license: | - In Spark 3.1, creating or altering a view will capture runtime SQL configs and store them as view properties. These configs will be applied during the parsing and analysis phases of the view resolution. To restore the behavior before Spark 3.1, you can set `spark.sql.legacy.useCurrentConfigsForView` to `true`. + - In Spark 3.1, CHAR/CHARACTER and VARCHAR types become individual types from string. By default, they can only be used in table schema, not functions/operators. To restore the behavior before Spark 3.1, where treats them as string with length parameter simply ignored, you can set `spark.sql.legacy.charVarcharAsString` to `true`. + + ## Upgrading from Spark SQL 3.0 to 3.0.1 - In Spark 3.0, JSON datasource and JSON function `schema_of_json` infer TimestampType from string values if they match to the pattern defined by the JSON option `timestampFormat`. Since version 3.0.1, the timestamp type inference is disabled by default. Set the JSON option `inferTimestamp` to `true` to enable such type inference. From 3cb87247568d1d5ecb6d8732e38cd4219f873ccf Mon Sep 17 00:00:00 2001 From: Kent Yao Date: Tue, 8 Dec 2020 02:37:17 +0800 Subject: [PATCH 2/5] [SPARK-33641][SQL][DOC][FOLLOW-UP] Add migration guide for CHAR VARCHAR types --- docs/sql-migration-guide.md | 1 - 1 file changed, 1 deletion(-) diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md index 6a264ccc70126..fb4c8f3a91fd5 100644 --- a/docs/sql-migration-guide.md +++ b/docs/sql-migration-guide.md @@ -56,7 +56,6 @@ license: | - In Spark 3.1, CHAR/CHARACTER and VARCHAR types become individual types from string. By default, they can only be used in table schema, not functions/operators. To restore the behavior before Spark 3.1, where treats them as string with length parameter simply ignored, you can set `spark.sql.legacy.charVarcharAsString` to `true`. - ## Upgrading from Spark SQL 3.0 to 3.0.1 - In Spark 3.0, JSON datasource and JSON function `schema_of_json` infer TimestampType from string values if they match to the pattern defined by the JSON option `timestampFormat`. Since version 3.0.1, the timestamp type inference is disabled by default. Set the JSON option `inferTimestamp` to `true` to enable such type inference. From a50e5ecea8d4a3e0d3377eeefda6449790982843 Mon Sep 17 00:00:00 2001 From: Kent Yao Date: Tue, 8 Dec 2020 09:59:14 +0800 Subject: [PATCH 3/5] [SPARK-33641][SQL][DOC][FOLLOW-UP] Add migration guide for CHAR VARCHAR types --- docs/sql-migration-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md index fb4c8f3a91fd5..5930d863924e7 100644 --- a/docs/sql-migration-guide.md +++ b/docs/sql-migration-guide.md @@ -54,7 +54,7 @@ license: | - In Spark 3.1, creating or altering a view will capture runtime SQL configs and store them as view properties. These configs will be applied during the parsing and analysis phases of the view resolution. To restore the behavior before Spark 3.1, you can set `spark.sql.legacy.useCurrentConfigsForView` to `true`. - - In Spark 3.1, CHAR/CHARACTER and VARCHAR types become individual types from string. By default, they can only be used in table schema, not functions/operators. To restore the behavior before Spark 3.1, where treats them as string with length parameter simply ignored, you can set `spark.sql.legacy.charVarcharAsString` to `true`. + - In Spark 3.1, we support CHAR/CHARACTER and VARCHAR types in our type system framework instead of replacing them with STRING types. By default, they can only be used in table schema, not functions/operators. To restore the behavior before Spark 3.1, which treats them as STRING types and ignores a length parameter, e.g. `CHAR(4)`, you can set `spark.sql.legacy.charVarcharAsString` to `true`. ## Upgrading from Spark SQL 3.0 to 3.0.1 From 80d44212db4ca5097fc89b1940c4db2a9df12696 Mon Sep 17 00:00:00 2001 From: Kent Yao Date: Tue, 8 Dec 2020 10:06:04 +0800 Subject: [PATCH 4/5] [SPARK-33641][SQL][DOC][FOLLOW-UP] Add migration guide for CHAR VARCHAR types --- docs/sql-migration-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md index 5930d863924e7..0a31b73405233 100644 --- a/docs/sql-migration-guide.md +++ b/docs/sql-migration-guide.md @@ -54,7 +54,7 @@ license: | - In Spark 3.1, creating or altering a view will capture runtime SQL configs and store them as view properties. These configs will be applied during the parsing and analysis phases of the view resolution. To restore the behavior before Spark 3.1, you can set `spark.sql.legacy.useCurrentConfigsForView` to `true`. - - In Spark 3.1, we support CHAR/CHARACTER and VARCHAR types in our type system framework instead of replacing them with STRING types. By default, they can only be used in table schema, not functions/operators. To restore the behavior before Spark 3.1, which treats them as STRING types and ignores a length parameter, e.g. `CHAR(4)`, you can set `spark.sql.legacy.charVarcharAsString` to `true`. + - In Spark 3.1, we support CHAR/CHARACTER and VARCHAR types in our type system framework instead of replacing them with STRING types. Currently, they can only be used in a table schema, not functions/operators. To restore the behavior before Spark 3.1, which treats them as STRING types and ignores a length parameter, e.g. `CHAR(4)`, you can set `spark.sql.legacy.charVarcharAsString` to `true`. ## Upgrading from Spark SQL 3.0 to 3.0.1 From dc88a0bafcf630a14792d6cb736a6d2fb21b2169 Mon Sep 17 00:00:00 2001 From: Kent Yao Date: Wed, 9 Dec 2020 14:38:46 +0800 Subject: [PATCH 5/5] typo --- docs/sql-migration-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md index 0a31b73405233..2bc04a0a79995 100644 --- a/docs/sql-migration-guide.md +++ b/docs/sql-migration-guide.md @@ -54,7 +54,7 @@ license: | - In Spark 3.1, creating or altering a view will capture runtime SQL configs and store them as view properties. These configs will be applied during the parsing and analysis phases of the view resolution. To restore the behavior before Spark 3.1, you can set `spark.sql.legacy.useCurrentConfigsForView` to `true`. - - In Spark 3.1, we support CHAR/CHARACTER and VARCHAR types in our type system framework instead of replacing them with STRING types. Currently, they can only be used in a table schema, not functions/operators. To restore the behavior before Spark 3.1, which treats them as STRING types and ignores a length parameter, e.g. `CHAR(4)`, you can set `spark.sql.legacy.charVarcharAsString` to `true`. + - Since Spark 3.1, CHAR/CHARACTER and VARCHAR types are supported in the table schema. Table scan/insertion will respect the char/varchar semantic. If char/varchar is used in places other than table schema, an exception will be thrown (CAST is an exception that simply treats char/varchar as string like before). To restore the behavior before Spark 3.1, which treats them as STRING types and ignores a length parameter, e.g. `CHAR(4)`, you can set `spark.sql.legacy.charVarcharAsString` to `true`. ## Upgrading from Spark SQL 3.0 to 3.0.1