Add more config for auto-topic-creation

[fxbing]

Master Issue: #4926

Motivation

Curently the partitioned-topic and non-partitioned topic is a little confuse for users. in PR #3450 we add config for auto-topic-creation.
We could leverage this config to provide some more config for auto-topic-creation.

Modifications

• Add allowAutoTopicCreationType and allowAutoTopicCreationNumPartitions to configuration.
• Users can use both configurations when they decide to create a topic automatically.
• Add test.
• Update doc.

[sijie]

@fxbing it is a nice motivation. but I am feeling that you didn't change the code in the right place. the change in fetchPartitionedTopicMetadataAsync will introduce unnecessary creation of partitioned topic. The auto-creation happens when a producer or a consumer connects to broker. so I think the change should be done in BrokerService. You can check the methods getOrCreateTopic.

Also change "partition" and "non-partition" to "partitioned" and "non-partitioned". It might be also worth considering adding an enum like TopicType?

[fxbing]

@fxbing it is a nice motivation. but I am feeling that you didn't change the code in the right place. the change in fetchPartitionedTopicMetadataAsync will introduce unnecessary creation of partitioned topic. The auto-creation happens when a producer or a consumer connects to broker. so I think the change should be done in BrokerService. You can check the methods getOrCreateTopic.

I first thought about making changes in the BrokerService, but when the client requests to create a Producer or Consumer, it will first get partitions info from zk, decide whether to create partitioned or non-partitioned. When the request arrives at the BrokerService, the client has decided to create a nonpartitioned Producer or Consumer. If the Broker's configuration requires automatic creation of a partitioned topic, Broker needs to send the configuration information to client, client re-establishes the partitioned Producer or Consumer, and the BrokerService updates the partitions in the zk.

I think it's easier to make changes when the client creates the type that determines the creation, and need fewer communications.

Do I need to modify it as I said above?

[fxbing]

Also change "partition" and "non-partition" to "partitioned" and "non-partitioned". It might be also worth considering adding an enum like TopicType?

OK， I will make a change.

[fxbing]

run java8 tests

[fxbing]

run Integration Tests

[sijie]

@fxbing well done. I have two questions to clarify with you and @jiazhai

the first one is about the setting name about the number of partitions.
the second one is whether we need to check if a non-partitioned topic exists or not when the partitioned-topic doesn't exist.

PTAL

[fxbing]

run C++ / Python Tests
run Integration Tests

[fxbing]

run cpp tests

[fxbing]

run cpp tests

[fxbing]

run cpp tests
run integration tests
run java8 tests

[fxbing]

run cpp tests
run integration tests
run java8 tests

[fxbing]

run cpp tests
run integration tests
run java8 tests

[fxbing]

run integration tests

[fxbing]

run java8 tests

[fxbing]

PTAL @sijie @jiazhai @codelipenghui

[codelipenghui]

@fxbing Nice work!
Overall looks good to me, just left a suggestion

[codelipenghui]

Since this method is no longer an asynchronous method, it's better to rename it to createDefaultPartitionedTopic()

[sijie]

@jiazhai @jennifer88huang can you please make sure this feature is well documented?

[Jennifer88huang-zz]

Got it.