KAFKA-18941: Do not test 3.3 in upgrade_tests.py

[josefk31]

The upgrade test in question is not supported for AK 3.3.2 due to a known issue. Previous attempt at solving this left the metadata.log.dir empty which leads to the following crash log:

ERROR Exiting Kafka due to fatal exception (kafka.Kafka$)
org.apache.kafka.common.KafkaException: No `meta.properties` found in  (have you run `kafka-storage.sh` to format the directory?)
	at kafka.server.BrokerMetadataCheckpoint$.$anonfun$getBrokerMetadataAndOfflineDirs$2(BrokerMetadataCheckpoint.scala:172)
	at scala.collection.Iterator.foreach(Iterator.scala:943)
	at scala.collection.Iterator.foreach$(Iterator.scala:943)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
	at scala.collection.IterableLike.foreach(IterableLike.scala:74)
	at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
	at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
	at kafka.server.BrokerMetadataCheckpoint$.getBrokerMetadataAndOfflineDirs(BrokerMetadataCheckpoint.scala:161)
	at kafka.server.KafkaRaftServer$.initializeLogDirs(KafkaRaftServer.scala:184)
	at kafka.server.KafkaRaftServer.<init>(KafkaRaftServer.scala:61)
	at kafka.Kafka$.buildServer(Kafka.scala:79)
	at kafka.Kafka$.main(Kafka.scala:87)
	at kafka.Kafka.main(Kafka.scala)

[josefk31]

I have tested this locally with the ducker-ak tool. The system tests work.

[chia7712]

@josefk31 in #18386 we made the e2e work for the "valid" upgrade path - single folder. Hence, could you please share more details of the description "upgrade test in question is not supported for AK 3.3.2 ", thanks!

[josefk31]

@chia7712 My apologies, I think I linked to the wrong issue, here is the correct one:
https://issues.apache.org/jira/browse/KAFKA-18442

Issue is as follows:
3.3.2 expects a metadata.properties file on startup. If it cannot find that file it will crash with fatal error. If metadata.properties was created by a newer version of kafka, then there will also be a fatal exception. Hence I'm not sure if we want to support an upgrade and downgrade with this version.

[cmccabe]

@josefk31 thanks for the explanation. let's disable this for now, given how old 3.3 is. If someone wants to put some time into getting 3.3 working, that is welcome as well... but at least we can fix the brokenness for now.

[cmccabe]

LGTM

[ijuma]

Hmm, our documentation says that upgrades from 3.3 to 4.0 are supported. So, I'm surprised we removed the test for that version. Also, 3.3.1 was released on October 2022 - that's a bit over 2 years old. Not that old from an upgrade source perspective.

[TaiJuWu]

Hi @josefk31 , thanks for this patch, I do two things to check this issue.

I ran following command without this patch, it pass on local and I can't reproduce this issue.
Could you help to check or share your step ?

TC_PATHS="tests/kafkatest/tests/core/upgrade_test.py::TestUpgrade.test_isolated_mode_upgrade_downgrade" _DUCKTAPE_OPTIONS='--parameters '\''{"from_kafka_version": "3.3.2", "metadata_quorum": "ISOLATED_KRAFT"}'\' bash tests/docker/run_tests.sh

 
I also checked 3.3.2 branch.
From my understanding, there is no need to set metadata.log.dir if I am wrong please correct me, thanks!

kafka/core/src/main/scala/kafka/server/KafkaRaftServer.scala

Lines 182 to 184 in 0a7ea3f

	val logDirs = (config.logDirs.toSet + config.metadataLogDir).toSeq
	val (rawMetaProperties, offlineDirs) = BrokerMetadataCheckpoint.
	getBrokerMetadataAndOfflineDirs(logDirs, ignoreMissing = false)

[josefk31]

@TaiJuWu That is very odd; I tested this on confluents infrastructure which is where the failure stems from... Since the same bits of 3.3.2 are being tested, I assumed that the failure was in AK. I will do more testing and then revert this PR once I discover more information.

[chia7712]

Previous attempt at solving this left the metadata.log.dir empty which leads to the following crash log:

yes, using "empty string" can cause the error you attached, but in the e2e, the "empty string" will be filtered, so we actually remove the value of metadata.log.dir in the e2e.

filtered_configs = {k: v for k, v in configs.items() if v not in [None, ""]}

That is very odd; I tested this on confluents infrastructure which is where the failure stems from... Since the same bits of 3.3.2 are being tested, I assumed that the failure was in AK. I will do more testing and then revert this PR once I discover more information.

yes, please share more details to us. If we can't run upgrade/downgrade the server with single folder, we have to update our upgrade docs to include the (new) bugs ...

[TaiJuWu]

Hi @josefk31 , any update for this?