KAFKA-12995 [WIP] Allow old Broker compatibility for Metadata calls

[aartigao]

Old Brokers prior to Metadata V4 don't support the allowAutoTopicCreation field.

The implementation of #3098 introduced a check for old Brokers compatibility falling back to true in MetadataRequest's allowAutoTopicCreation field for some operations like describeTopics.

Then, after that backward-compatibility changes, more methods were added to AdminClient like listOffsets in #7296 which unfortunately don't have compatibility with old Brokers into account. One might argue that, given that new methods were not present previously, it's not a compatibility break. While this is strictly true, I believe that one of Kafka Clients' most appreciated feature is the ability to use old Brokers with new Clients.

I've prepared this small PR to fix for this, following the same approach used in #3098.

Committer Checklist (excluded from commit message)

• Verify design and implementation
• Verify test coverage and CI build status
• Verify documentation (including upgrade notes)

[ijuma]

Thanks for the PR. I am tempted to specify list offsets methods to support 0.11.0 brokers and newer. 0.10.x is extremely old now. It was different when we implemented #3098 4+ years ago.

[aartigao]

Thanks for the PR. I am tempted to specify list offsets methods to support 0.11.0 brokers and newer. 0.10.x is extremely old now. It was different when we implemented #3098 4+ years ago.

Well, from my POV I'd say that listOffsets looks like a feature supported in 0.10.x, it's just the metadata headers sent from AdminClient that prevent it from working (if I'm not wrong).

I understand that 0.10.x is an old version, but it's the one that our company uses in Production 🤷🏽 I don't know how many companies are out there like mine, but I'd bet they will be happy to know that if they want to use listOffsets for whatever new thing they develop, they are not forced to upgrade the Kafka Cluster version. We all know that nowadays Kafka is a key component in most companies architecture, and performing a major upgrade is a big thing that needs a huge amount of work and synchronization between teams (at least compared with a client update).

That being said, whatever decision you make, will be fine (at least I've tried 🙂 ).

[ijuma]

From what I can gather very few companies still use 0.10.x. It has serious bugs that can result in data loss. KIP-101 (introduced in 0.11.0) is a must have if you care about your data. Subsequent versions include other very important fixes.

Kafka cluster upgrades are generally easier than exhaustive client upgrades as clusters are typically managed by fewer teams.