Skip to content

Make ZSTDMT_JOBSIZE_MIN runtime configurable #2043

@romankarlstetter

Description

@romankarlstetter

On low-end (but multicore) systems, having ZSTDMT_JOBSIZE_MIN set to 1MB might be too large. Even for smaller buffers than 1MB, having multithreading enabled would be beneficial. Of course, I could compile with a lower value for ZSTDMT_JOBSIZE_MIN, but that is not very accessible for every user.

So, I thought it might be a good idea to have ZSTDMT_JOBSIZE_MIN configurable as a runtime parameter (like number of threads already is configurable), so that I can enable it already at smaller sizes.

As an alternative, it might be good to at least mention in the API documentation for the ZstdCompressor (I'm using the python API via import zstandard) that there is the limit of 1MB for parallel compression.

Here is an image illustrating the performance jump on an Intel Atom E3940 @ 1.6GHz (measurement is done via python in a jupyter notebook, compressor is initialized like that: ZstdCompressor(zstd_level, threads=-1)).
2020-03-19 10_50_35-ZSTD compression throughput test

Metadata

Metadata

Assignees

Labels

long-termvalid topics that are expected take a long time to make progressoptimization

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions