-
Notifications
You must be signed in to change notification settings - Fork 2.4k
Description
On low-end (but multicore) systems, having ZSTDMT_JOBSIZE_MIN set to 1MB might be too large. Even for smaller buffers than 1MB, having multithreading enabled would be beneficial. Of course, I could compile with a lower value for ZSTDMT_JOBSIZE_MIN, but that is not very accessible for every user.
So, I thought it might be a good idea to have ZSTDMT_JOBSIZE_MIN configurable as a runtime parameter (like number of threads already is configurable), so that I can enable it already at smaller sizes.
As an alternative, it might be good to at least mention in the API documentation for the ZstdCompressor (I'm using the python API via import zstandard) that there is the limit of 1MB for parallel compression.
Here is an image illustrating the performance jump on an Intel Atom E3940 @ 1.6GHz (measurement is done via python in a jupyter notebook, compressor is initialized like that: ZstdCompressor(zstd_level, threads=-1)).
