[SPARK-27256][CORE][SQL]If the configuration is used to set the number of bytes, we'd better use bytesConf'.

[10110346]

What changes were proposed in this pull request?

Currently, if we want to configure spark.sql.files.maxPartitionBytes to 256 megabytes, we must set spark.sql.files.maxPartitionBytes=268435456, which is very unfriendly to users.

And if we set it like this:spark.sql.files.maxPartitionBytes=256M, we will encounter this exception:

Exception in thread "main" java.lang.IllegalArgumentException:
 spark.sql.files.maxPartitionBytes should be long, but was 256M
        at org.apache.spark.internal.config.ConfigHelpers$.toNumber(ConfigBuilder.scala)

This PR use bytesConf to replace longConf or intConf, if the configuration is used to set the number of bytes.
Configuration change list:
spark.files.maxPartitionBytes
spark.files.openCostInBytes
spark.shuffle.sort.initialBufferSize
spark.shuffle.spill.initialMemoryThreshold
spark.sql.autoBroadcastJoinThreshold
spark.sql.files.maxPartitionBytes
spark.sql.files.openCostInBytes
spark.sql.defaultSizeInBytes

How was this patch tested?

1.Existing unit tests
2.Manual testing

[SparkQA]

Test build #103839 has finished for PR 24187 at commit 761e2b7.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[maropu]

We need to check if the input value exists in the integer range?

[10110346]

Yeah,thanks

[srowen]

I'm not sure this is a net helpful change, as the parameter is maxPartitionBytes. I agree it would have been better to call it maxPartitionSize and accept values like "10m". I'm not strongly against it, as existing values would still work.

For other property values without "Bytes", I agree.

[10110346]

Thanks.
Yeah, the parameter name is a bit confusing, but I think it is not very important whether the parameter name contains "Bytes" or not, I prefer to change it.

[kiszk]

I like this change since both styles (i.e. 1024 and 1k) can be accepted.

[srowen]

OK, I'm fine with it.

[SparkQA]

Test build #103881 has finished for PR 24187 at commit ef67451.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[10110346]

retest this please

[maropu]

We could have the same fix below, too?

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala

Line 61 in 8ec6cb6

val FILESOURCE_TABLE_RELATION_CACHE_SIZE =

Anyway, have you checked all the related places for the same fix?

[SparkQA]

Test build #103894 has finished for PR 24187 at commit ef67451.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[10110346]

We could have the same fix below, too?

spark/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/StaticSQLConf.scala

Line 61 in 8ec6cb6
val FILESOURCE_TABLE_RELATION_CACHE_SIZE =

Anyway, have you checked all the related places for the same fix?

Thanks.
FILESOURCE_TABLE_RELATION_CACHE_SIZE is used to configure the number of entries, not the number of bytes.
I've checked where buildConf and ConfigBuilder are called.

[10110346]

retest this please

[maropu]

ok, to make other reviewers understood easily, could you list up all the configs that this pr changed in the PR description?

[10110346]

ok, to make other reviewers understood easily, could you list up all the configs that this pr changed in the PR description?

Ok, thanks.

[maropu]

what if we dont' have this long cast?

[10110346]

If we don't convert to long first , it will encounter exception like this:
Caused by: java.lang.ClassCastException: java.lang.Long cannot be cast to java.lang.Integer

[maropu]

plz update the message, too.

[10110346]

ok, thanks

[maropu]

LGTM and I leave it to other reviwers. cc: @cloud-fan @srowen @dongjoon-hyun

[SparkQA]

Test build #103900 has finished for PR 24187 at commit ef67451.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #103909 has finished for PR 24187 at commit 206857d.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

thanks, merging to master!