[Bug]: Cannot read from Kafka due to short poll timeout of consumer in KafkaIO

[xianhualiu]

What happened?

The default Kafka consumer poll timeout is set to 1 second. It works fine when the the response can get messages from the kafka broker server within this 1 second, such as when client accesses broker within the same region. But if the responding time is more than 1 second, the consumer will not retrieve any messages. One customer reported that throughput of processed messages was extremely low in cross-region read since most of the time the responding time takes more than 1 second.

As a solution, the Kafka consumer polling timeout needs to be configurable, so customer can adjust it according to their needs.

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam YAML
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[xianhualiu]

.take-issue

[jbsabbagh]

This also affects the Python SDK.