RFC: Message Queue Implementation

[ignatz]

Hi folks,

I was starting to get more serious about adding support for message queues. I've been looking at apalis, which looks like a great starting point. Anyway, there's like 50 things "support" may mean in this context and I don't feel like I have a good grasp on the use-cases y'all have. So here we are...

The way I see it right now, there's a few questions I'd like to answer based on your use-cases:

1. Where does the persistent queue live? The queue could be part of TrailBase or defer to an external solution like RabbitMQ, Redis, celery, Amazon SQS, ... .
2. Where do the message producers live? They could either be external using some API and/or be internal e.g. an API available to the JS runtime (and native Rust code). The latter would allow you to build your own APIs.
3. Where do the message consumers live? They could be external worker tasks and/or be internal e.g. a registered JS function. The latter would also allow you to call out to external workers.
4. Is there a desire for a horizontal back-channel from message consumers (worker) to producer? You can always build this yourself, e.g. a results table keyed by task/job-id, but depending on (3) (internal vs external workers) it may be more or less cumbersome.

I'll start by sharing some inclinations:

Initially I thought the answer to (1) should certainly be internal clearly in line with the single-executable mantra but depending on peoples needs, coupling and pre-existing setups it may not be as clear cut as I initially thought.

For (2) and (3) I'm leaning internal only, at least as a starting point also letting folks implement whatever they need w/o having to provide full AMQP (or similar) support.

For (4) I'm leaning no and more towards providing abstractions that makes it easier to do it yourself with internal workers. I also don't see to much precedent for this with other queues. Thinking about examples it's also not clear to me how useful it would be. For example, sending an email probably wouldn't need a back-channel it's only success/fail. Alternatively, running post-processing or image recognition over an uploaded image would probably write the results back to blob storage or primary table respectively.

I can be haggled with and thanks already for any input 🙏

[ignatz]

Having let this simmer for the last 9 days and received one vote for "Internal queue with internal message producers and external consumers", I'd love to hear a bit more feedback on how this would look like.

Concretely, would this be a AMQP-compatible setup or a custom protocol to invoke the remote cosumers/workers? AFAICT, for Rust there's https://github.com/bytebeamio/rumqtt, which sadly looks a bit stale. W/o stronger signal it feels a bit risky of maintaining my own AMQP broker. A custom protocol either through rust or via the JS runtime would be pretty straight forward but may not be what the voter had in mind. At that point it might be best to rely on a separate RabbitMQ instance to publish to from TB? 🤷‍♀️

This is mostly meant as an icebreaker, would love to get more input.

[vootooguru]

I'm sorry for the mistake.I hope it can support external producers and consumers.Consider this use case:using Caddy or other front-end reverse proxies to support multiple TB instances in the backend.These instances share data storage and message queues to achieve non-stop updates for core business operations.

TB should be a lightweight solution.A solution based on Redis would be a good match—lightweight,simple,and easy to get started with.Both TB and Redis are known for their incredibly fast performance(I love that super-fast feeling).However,Redis lacks robust message queue management,which is where TB could play a significant role and potentially be a deciding factor for users to choose it.

Additionally,as an easy-to-use and lightweight product,TB should not integrate overly complex third-party tools.Using Redis not only enables message queueing but also allows for shared caching.It's simple to install and use,making it a two-in-one solution.

[ignatz]

Thanks for sharing the use-case, this really helps to ground the conversation 👍

I very much agree with your sentiment that there are already good, fast and battle-tested queue solutions out there. Trying to achieve parity would be a journey. Especially with the vote falling on external consumers (and also external producers) building the queue into TB, doesn't feel significant in driving down complexity of the overall stack.

However,Redis lacks robust message queue management,which is where TB could play a significant role and potentially be a deciding factor for users to choose it.

With the queue, consumers and producers externalized, would this mostly be a control plane? Have you looked at something like https://docs.celeryq.dev/en/stable/getting-started/backends-and-brokers/redis.html ?

Lastly and at the risk of going a bit on a tangent, I'd love to understand your proposed use-case better:

I hope it can support external producers and consumers.Consider this use case:using Caddy or other front-end reverse proxies to support multiple TB instances in the backend.These instances share data storage and message queues to achieve non-stop updates for core business operations.

That does sound like Redis, RabbitMq, ... could be a good fit. I know you're building your own TB binaries :), so I'm wondering if you couldn't use any of the Redis or AMQP clients both for Rust as well as JS/TS? Then maybe have something like celery running on the side?

(If you ever feel up for chatting about your plans more concretely, I'd love to hear but I don't wanna intrude. Definitely sounds like a lot of fun what you're going for 🥳 )

[ignatz]

Thanks again for sharing your thoughts. After mediating a while longer on this, I wonder if the best we could do at this point, is to provide an example on how to submit and handle tasks to and from a redis queue in JS. Based on our other conversations, I'd guess you're more interested in the Rust side-of-things, but does that make sense?

At least we can see how things line up and maybe more ways to improve impedance will show up in the process (or in the future).

[ignatz]

Quick update (and memo to myself): I've been trying to set up a redis example with the JS runtime. I've run into issues with the rustyscript/deno JS runtime, where the redis package relies on node's DNS resolution, which the current setup seemingly doesn't satisfy :/. Updating rustyscript is also tricky since, deno started to depend on a stale rusqlite in many places (at least the stale version rustyscript pulls in)... :/ (besides it's unclear if the never version would solve the issue despite its broader experimental node support).

Arguably, none of this is directly related to "message queue integration", however we're practically limited in what we can do about it. Possible routes might be:

• Pick a different integration option
• Wrap a Rust redis client (though that would only brush over the JS runtime limitations).
• Address the JS runtime limitations (the future of rustyscript is a bit unclear, which makes it sort of tempting to either take on more in this space or switch to an alternative. The only real alternative at this point would be WASM. While that would open the door to many integrations and lead to a smaller binary, it would make using JS more annoying... unclear what would be the better trade-off).

Anyway, just wanted to keep readers informed and organize some of my thoughts.