Revert "[SPARK-45502][BUILD] Upgrade Kafka to 3.6.0"

[LuciferYang]

This reverts commit d1bd21a.

What changes were proposed in this pull request?

This pr aims to revert SPARK-45502 to make the test case KafkaSourceStressSuite stable.

Why are the changes needed?

The test case KafkaSourceStressSuite has become very unstable after the merger of SPARK-45502, with 10 out of the recent 22 tests failing because of it. Revert it for now, and we can upgrade Kafka again after resolving the test issues.

• https://github.com/apache/spark/actions/runs/6497999347/job/17648385705
• https://github.com/apache/spark/actions/runs/6502219014/job/17660900989
• https://github.com/apache/spark/actions/runs/6502591917/job/17661861797
• https://github.com/apache/spark/actions/runs/6503144598/job/17663199041
• https://github.com/apache/spark/actions/runs/6503233514/job/17663413817
• https://github.com/apache/spark/actions/runs/6504416528/job/17666334238
• https://github.com/apache/spark/actions/runs/6509796846/job/17682130466
• https://github.com/apache/spark/actions/runs/6510877112/job/17685502094
• https://github.com/apache/spark/actions/runs/6512948316/job/17691625228
• https://github.com/apache/spark/actions/runs/6516366232/job/17699813649

Does this PR introduce any user-facing change?

No

How was this patch tested?

Pass GitHub Actions

Was this patch authored or co-authored using generative AI tooling?

No

[HyukjinKwon]

I am fine with this but cc @HeartSaVioR

[LuciferYang]

#43362

I see that the author is trying to fix it, we can also wait a little longer. But since the sbt test is in a fail-fast mode, once KafkaSourceStressSuite fails, the modules to be tested later will not run, making it hard to confirm whether the subsequent tests are healthy. This is a risk point.

also cc @dongjoon-hyun @dengziming

[HeartSaVioR]

+1 on reverting. Thanks @LuciferYang to figure out the unstability.

In addition, I really want to see the "benefits" being listed in PR description when we bump the version. Have we run any benchmark to verify there is no regression? Have we looked into the release notes to highlight the benefits and critical bugfixes to provide rationale of the version bump? Could we please be conservative and pick bugfix versions unless the minor release unblocks us/users for some specific functionalities?

[dengziming]

LGTM, I already find some clues but it's still not enough, lets revert it firstly.

[LuciferYang]

https://github.com/LuciferYang/spark/runs/17721718456

GA Passed

[LuciferYang]

Merged into master. Thanks @HyukjinKwon @HeartSaVioR @dengziming ~

[dongjoon-hyun]

Thank you, @LuciferYang and all.

[dengziming]

Could we please be conservative and pick bugfix versions unless the minor release unblocks us/users for some specific functionalities?

Thank you @HeartSaVioR for mention this, I agree with you about picking bugfix versions, however, this bug is introduced by me when I haven't tested it carefully and the last build accidentally succeed, so it was merged by mistake. lets wait and upgrade it to 3.6.1 later.

BTW, I have found the root cause is apache/kafka#12590, we will send a fetch request when closing FetchSession which contains a deleted topic, and we are setting "auto.create.topics.enable"=true so deleted topics will come back again.

[dongjoon-hyun]

Thank you for the info, @dengziming .