Skip to content

Benchmark and prototype compressing message bus payloads #5239

@Kami

Description

@Kami

Right now we send raw pickled object byte strings over message bus (yeah, pickling is not great and IIRC, there is like a 3-4 year old ticket to move from pickle to something else, but sadly that never materialized and it's not an easy change).

In some scenarios such as when dispatching whole ExecutionDB and TriggerInstanceDB objects, those payloads can be quite large since they contain the whole result / payload values.

We should prototype and benchmark compressing the payload using something like zstandard. Micro benchmarks should be a good starting point.

Since we already send byte strings over wire, I think adding compression should be pretty straight forward and we could also make it backward compatible relatively easily.

That functionality should also be behind a feature flag and only enabled by default if it shows speed improvements for all the common scenarios (small, mid and large sized payloads).

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions