KAFKA-18346: Fix e2e TestKRaftUpgrade from v3.3.2

[TaiJuWu]

jira: https://issues.apache.org/jira/browse/KAFKA-18346

This failure related to #13130

see #18386 (comment)

Committer Checklist (excluded from commit message)

• Verify design and implementation
• Verify test coverage and CI build status
• Verify documentation (including upgrade notes)

[TaiJuWu]

================================================================================
SESSION REPORT (ALL TESTS)
ducktape version: 0.12.0
session_id:       2025-01-04--014
run time:         4 minutes 44.207 seconds
tests run:        1
passed:           1
flaky:            0
failed:           0
ignored:          0
================================================================================
test_id:    kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_isolated_mode_upgrade_downgrade.from_kafka_version=3.3.2.metadata_quorum=ISOLATED_KRAFT
status:     PASS
run time:   4 minutes 44.023 seconds
--------------------------------------------------------------------------------
test_id:    kafkatest.tests.core.kraft_upgrade_test.TestKRaftUpgrade.test_combined_mode_upgrade.from_kafka_version=3.3.2.metadata_quorum=COMBINED_KRAFT
status:     PASS
run time:   3 minutes 28.845 seconds

[TaiJuWu]

Hi @ijuma , I also hope this PR can bring your attention and I have written the root cause.
But I don't think we should apply this fix.

[ijuma]

I had missed this - don't we need it anymore? cc @mumrah @cmccabe

[TaiJuWu]

I had missed this - don't we need it anymore? cc @mumrah @cmccabe

Even if we apply this fix, just hide this issue in system test but it still exists and the only way is back port to 3.3 but it won’t be released.

[ijuma]

It seems like we are testing that we can upgrade from 3.3 with a single log directory - is that right? If so, that makes sense to me.

[TaiJuWu]

We are testing DOWNGRADE from 4.0 to 3.3 and it ONLY work for single folder but in KRAFT mode it should often be three folders.

The error throw

kafka.common.InconsistentBrokerMetadataException: BrokerMetadata is not consistent across log.dirs. This could happen if multiple brokers shared a log directory (log.dirs) or partial data was manually copied from another broker. Found:
- /mnt/kafka/kafka-metadata-logs -> {node.id=3001, directory.id=hqW-A8W0pbE8dJmM2otDPQ, version=1, cluster.id=I2eXt9rvSnyhct8BYmW6-w}
- /mnt/kafka/kafka-data-logs-2 -> {node.id=3001, directory.id=nbiO8yKUacnyIijAkoFcFg, version=1, cluster.id=I2eXt9rvSnyhct8BYmW6-w}
- /mnt/kafka/kafka-data-logs-1 -> {node.id=3001, directory.id=2-xUs6eYlJtkVUcl6lJmtQ, version=1, cluster.id=I2eXt9rvSnyhct8BYmW6-w}

What #13130 do is split ZK mode and KRAFT mode, and if we work on KRAFT mode we only check clusterID and nodeID to avoid compatibility issue.
That why I said the only solution is back port that PR to 3.3, does it make sense?

By the way, ZK mode we check nodeID + clusterID + directoryID so we have such issue.

[chia7712]

@TaiJuWu I feel that approach has two issue.

1. In rolling-upgrade, the version is changed from 3.3 to trunk. Hence, the metadata folder will be inconsistent as we don't change the path when the version is not 3.3. That means all metadata gets lost after upgrading in e2e.
2. the path in this PR is incorrect. config_property.METADATA_LOG_DIR is key rather than a path

[chia7712]

To mitigate the 3.3 issue, we could add a flag to the KafkaService class to indicate whether different log directories have been used. During node restarts, this flag can be checked when generating properties.

@TaiJuWu what are your thoughts? You have identified the root cause, so it would be excellent if you could complete this PR 😃

[TaiJuWu]

To mitigate the 3.3 issue, we could add a flag to the KafkaService class to indicate whether different log directories have been used. During node restarts, this flag can be checked when generating properties.

This is the issue about this fix and is not related to the root cause, I would like to discuss the root cause about this issue but I still very appreciate for your review.

@TaiJuWu what are your thoughts? You have identified the root cause, so it would be excellent if you could complete this PR 😃

After discussion offline with @chia7712 , the main question becomes should we limit the downgrade path to single folder in our system test or we want to remove this version directly?
The issue about multiple folder downgrade is a known issue (#13130) and we won't release v3.3 again.

The former means we still support limited downgrade and the latter means we don't give downgrade promission about v3.3.
No matter what we should document this.

cc @ijuma @mumrah @cmccabe
ping @dajac since you are RM for 4.0

[TaiJuWu]

reopen for discussion.

[chia7712]

@TaiJuWu you're correct. 3.3 had some issues, and we didn't backport the fix (#13130). However, 3.3 is an outdated release, and we don't need to prioritize "how to downgrade multi-folders Kraft to 3.3." I believe this is a rare use case with limited benefits. In this PR, we can revise the test case to ensure a path exists for downgrading to 3.3. I think this is a sufficient approach."

[chia7712]

we want to remove this version directly?

I prefer to include 3.3 in the upgrading/downgrading tests, as we already know how to successfully execute these scenarios, right?

[chia7712]

@TaiJuWu thanks for this fix.

[chia7712]

Please add this behavior to the document (line#258)

[chia7712]

Could you please revise the comment with more details?

[TaiJuWu]

All comments are address, please take a look.

[chia7712]

LGTM

run the e2e tests/kafkatest/tests/core/kraft_upgrade_test.py on my local with this patch. all pass