[SPARK-43974][CONNECT][BUILD] Upgrade buf to v1.23.1

[panbingkun]

What changes were proposed in this pull request?

The pr aims to upgrade buf from 1.20.0 to 1.23.1

Why are the changes needed?

1.Release Notes:

• https://github.com/bufbuild/buf/releases/tag/v1.23.1
• https://github.com/bufbuild/buf/releases/tag/v1.23.0
• https://github.com/bufbuild/buf/releases/tag/v1.22.0
• https://github.com/bufbuild/buf/releases/tag/v1.21.0

2.The new version brings some bug fixed and improvment, as follow:

• Fix issue where buf beta graph would not print modules within a workspace that
  had no dependencies or dependents.
• Fix issue where buf beta graph would print warnings for missing dependencies
  that were actually present.
• Fix issue where locally-produced images did not have module information if the corresponding
  module was stored in the new cache.
• Remove buf beta registry template.
• Remove buf beta registry plugin {create,deprecate,list,undeprecate,version} and replace with
  buf beta registry plugin {push,delete}.
• Update buf beta price with the latest pricing information.

3.Manually test: dev/connect-gen-protos.sh, this upgrade will not change the generated files.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

• Manually test
• Pass GA

[dongjoon-hyun]

Just a head-up. This is 4th commit since 5/1. Shall we hold on this dependency and do accumulated-update monthly. For example, shall we wait until July 1st, @panbingkun and @zhengruifeng ?

$ git log --oneline --since 2023-05-01 | grep 'Upgrade buf'
d61c77cac1 [SPARK-43890][CONNECT][BUILD] Upgrade buf to v1.20.0
e186e9dfbf [SPARK-43599][CONNECT][BUILD] Upgrade buf to v1.19.0
7992455de2 [SPARK-43401][CONNECT][BUILD] Upgrade buf to v1.18.0

[zhengruifeng]

@dongjoon-hyun I am fine with holding on it until July 1st, buf release is a bit frequent.
We may also need to upgrade it to the latest version before 3.5 rc.

[dongjoon-hyun]

Thank you. Yes, I agree with you. Since the feature freeze is July 16th, maybe after July 10th?

• https://spark.apache.org/versioning-policy.html

[panbingkun]

@dongjoon-hyun Ok, Let's holding on it until July 1st.

[dongjoon-hyun]

Thank you, @panbingkun . I'm fine any date after July 1st~ Feel free to proceed after than.

[zhengruifeng]

Thank you. Yes, I agree with you. Since the feature freeze is July 16th, maybe after July 10th?

• https://spark.apache.org/versioning-policy.html

July 10th also works for me

[LuciferYang]

I set this to draft first to avoid unexpected merging

[panbingkun]

@dongjoon-hyun @zhengruifeng @LuciferYang Before we release the new Spark version, please let's review this PR again. 😄

[zhengruifeng]

merged to master

[LuciferYang]

late LGTM

[panbingkun]

There seems to be a problem with this patch, I need a new followup to solve this problem
@zhengruifeng

[panbingkun]

@zhengruifeng Should we migrate or revert it?
FOLLOWUP PR: #41934

[zhengruifeng]

@panbingkun I think this failure started before this PR got merged.

https://github.com/apache/spark/actions/runs/5511781438/jobs/10047815050

does this upgrade cause so many changes in generated codes? #41934

[LuciferYang]

It's strange that the failure started after upgrading to sbt 1.9.2, which doesn't seem to be related ...

https://github.com/apache/spark/actions/runs/5511636288/jobs/10047488637

[panbingkun]

@zhengruifeng
Yes, when I executed sh dev/connect-gen-protos.sh locally, the automatically generated file in Python changed.
My local buf version is:

[zhengruifeng]

@panbingkun it is weird since IIRC the CI in this PR passed.

but, if this upgrade cause such large changes in generated codes, let's revert it for now to avoid big changes before code freeze.

[panbingkun]

In order to address the #41933 issue, the above PR has been reverted and now resubmitted.
#41937