[SPARK-34576][SQL] Fix drop/add columns to a dataset of DESCRIBE COLUMN

[AngersZhuuuu]

What changes were proposed in this pull request?

In the PR, I propose to generate "stable" output attributes per the logical node of the DESCRIBE COLUMN command.

Why are the changes needed?

This fixes the issue demonstrated by the example:

val tbl = "testcat.ns1.ns2.tbl"
sql(s"CREATE TABLE $tbl (c0 INT) USING _")
val description = sql(s"DESCRIBE TABLE $tbl c0")
description.drop("info_name")

[info]   org.apache.spark.sql.AnalysisException: Resolved attribute(s) info_name#74 missing from info_name#25,info_value#26 in operator !Project [info_name#74]. Attribute(s) with the same name appear in the operation: info_name. Please check if the right attribute(s) are used.;
[info] !Project [info_name#74]
[info] +- LocalRelation [info_name#25, info_value#26]

Does this PR introduce any user-facing change?

After this change user drop()/add() works well.

How was this patch tested?

Added UT

[SparkQA]

Test build #135622 has started for PR 31696 at commit 1b03c95.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40202/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40202/

[AngersZhuuuu]

FYI @cloud-fan @HyukjinKwon

[cloud-fan]

shall we pass the output to the v1 command as well?

[AngersZhuuuu]

shall we pass the output to the v1 command as well?

You mean unify V2 and v1 DESC command's output? I think v2's DESC TABLE need to do this too.

[cloud-fan]

yea, you can fix it here as well.

[AngersZhuuuu]

yea, you can fix it here as well.

Done

[SparkQA]

Test build #135632 has finished for PR 31696 at commit 2dc21a0.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40212/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40212/

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40226/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/40226/

[AngersZhuuuu]

retest this please

[SparkQA]

Test build #135660 has finished for PR 31696 at commit 50b9e0c.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

thanks, merging to master!

[cloud-fan]

@AngersZhuuuu can you open a backport PR for 3.1?

[AngersZhuuuu]

@AngersZhuuuu can you open a backport PR for 3.1?

sure

[MaxGekk]

Can this PR be ported to 3.0?

[cloud-fan]

#31676 doesn't go to 3.0 either. Shall we backport both?