Add Apache Kafka integration

[kazanzhy]

There are a few high-level questions for this integration.

1. How to implement this hook?
The first way is similar to PubSubHook when all functionality is in the hook. Like PubSubHook.publish().
The second way is similar to FirehoseHook when the hook is just a wrapper of the boto client which is used to interact with Kinesis.
Talking about the Apache Pulsar (#21618), it has a good client that manages and reuses sessions for producers and consumers. That's why for Pulsar second way is better.
But for Kafka, there are a few separate classes for producer and consumer.

2. What python package to use?
The kafka-python is more popular, but there are no new commits last 2 years.
On the other hand, confluent-kafka-python is actively developing but
less popular and developing by Confluent company.

[potiuk]

@kazanzhy - I think I managed to workaround the pip resolver issue with #21824 - please rebase to latest main.

[eladkal]

Can you add simple example dag?

[eladkal]

Since this is not an Airflow hook.. I think it would be best to use another name to avoid confusion?
Also this class is not covered with unit tests

[eladkal]

can we test this function?

[eladkal]

I don't remember we ever edited a UI file when adding a provider?
cc @bbovenzi

[bbovenzi]

Yeah I would ignore what's in /ui for now. It needs a refresh post 2.3.

[kazanzhy]

Hi @eladkal. Thank you for the review
This PR was created mostly for discussion and I really need suggestions to answer the questions in the description.

There are implemented Hooks for PubSub and Kinesis, so I decided to create integrations for Kafka and Pulsar. It will be very convenient if they will be unified.
Therefore I'm not sure if this implementation is right, because of the creation of one more Class to merge Kafka library classes into one "client".

[github-actions]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.