-
-
Notifications
You must be signed in to change notification settings - Fork 782
Description
Right now we send raw pickled object byte strings over message bus (yeah, pickling is not great and IIRC, there is like a 3-4 year old ticket to move from pickle to something else, but sadly that never materialized and it's not an easy change).
In some scenarios such as when dispatching whole ExecutionDB and TriggerInstanceDB objects, those payloads can be quite large since they contain the whole result / payload values.
We should prototype and benchmark compressing the payload using something like zstandard. Micro benchmarks should be a good starting point.
Since we already send byte strings over wire, I think adding compression should be pretty straight forward and we could also make it backward compatible relatively easily.
That functionality should also be behind a feature flag and only enabled by default if it shows speed improvements for all the common scenarios (small, mid and large sized payloads).