feat: add kafka broadcast support

[woile]

Hey Tom I've been playing around with kafka and I made broadcaster work with it :)
There are some things to polish, but maybe you want to play with it so I'm sharing the code.
For some reason the kafka test gets stuck, I'm not really proficient in async code so I find it hard to debug it, I need some help with it.

I'm providing some extra files that can be removed, but they are there to guide, let me know if you want me to remove them.

There's a kafka with zookeeper implementation in the docker-compose file (made by @marcosschroh), by just typing docker-compose up, you get a full kafka (no ssl) in localhost:9092, this is useful to play with the kafka_example.py

Maybe the test suit could be improved using different docker-compose files, I don't know how do you envision them.

Cheers!

[lovelydinosaur]

🔥 😎 Yas!

Will take a look over this next week and discuss, but just want to give my 👍👍👍 until then.

[lovelydinosaur]

Maybe the test suit could be improved using different docker-compose files, I don't know how do you envision them.

I've no idea. We don't even have CI up and running on this repo yet, and open to suggestions there.

[lovelydinosaur]

I reckon let's not have seperate examples for different backends, and instead just document a few different valid backend URLs in the single example.py case?

[woile]

Maybe the test suit could be improved using different docker-compose files, I don't know how do you envision them.

I've no idea. We don't even have CI up and running on this repo yet, and open to suggestions there.

Maybe we can use some of this packages:
https://github.com/avast/pytest-docker
https://github.com/pytest-docker-compose/pytest-docker-compose

And docker-compose would be part of the test-requirements.txt

[woile]

I reckon let's not have seperate examples for different backends, and instead just document a few different valid backend URLs in the single example.py case?

Agreed, but not sure how.

Should we keep examples/ folder and rename kafka_example.py to just example.py? maybe using an env we could choose the backend BACKEND=redis examples/example.py ?

The current ./example.py is harder to get it up and running because it has no doc about how to use it.

[lovelydinosaur]

The current ./example.py is harder to get it up and running because it has no doc about how to use it.

It might make more sense to move it into a /example directory?

Then we can have...

• /example/templates/...
• /example/app.py
• /example/requirements.txt
• /example/README.md

[lovelydinosaur]

And populate the URL from something like os.environ.get("BROADCAST_URL", "memory://") so that usage is like so...

BROADCAST_URL=...; uvicorn app:app

[lovelydinosaur]

I wouldn't bother with any scripts stuff in the example directory, just a plain README walkthrough.

[woile]

Perfect, I'll give it a go tomorrow, and then we see how it evolves

[woile]

Updated!

[lovelydinosaur]

Okay, this is looking pretty fantastic actually.

A few very minor suggestions, but probably also happy to merge if any of those are contentious.

[woile]

Nothing serious, I've updated the PR with the changes

[lovelydinosaur]

When testing redis (commenting out the other postgres/kafka cases, and using scripts/start redis), the connection establishes to the docker redis-server on the first run, but hangs if I re-run the tests. Are you able to replicate too, and any idea how to resolve?

[woile]

I'm running pytest -k redis tests/ with no problem several times.

Try removing the volumes from redis when stopping it: docker-compose down -v

[lovelydinosaur]

No dice

[hramezani]

When testing redis (commenting out the other postgres/kafka cases, and using scripts/start redis), the connection establishes to the docker redis-server on the first run, but hangs if I re-run the tests. Are you able to replicate too, and any idea how to resolve?

same for me.