zstd: support c based implementation

[tonistiigi]

follow up #2344

Cgo based implementations should be faster and also use standard compression levels. As dockerfile supports c cross-compiling, supporting cgo is not a problem.

There seems to be two libraries:

• https://github.com/DataDog/zstd
  Tested this internally and the master version seemed to work fine. I don't like that it is making a copy of upstream source code though.

• https://github.com/valyala/gozstd
  Seems like a cleaner implementation but for some bizarre reason that repo vendors prebuilt static libs inside the source code for all arch 🤦 . So that definitely would not work for our vendoring and our platform support is bigger anyway. Looks like if we were to use it we would need to create a mirror without the binary artifacts.

The differ side can be implemented only on buildkit side. The apply side is hardcoded in contained and needs a Compressor opt in ApplyOpt same way as was added on the differ side to support stargz.

Another option would be to shell out to zstd binary like we do for pigz. I have not benchmarked this and likely something like zstd:chunked should still be using golang implementation then.

@ktock

[sanmai-NL]

@tonistiigi From a software security perspective using a separate zstd executable is preferred.

[froody]

The performance difference vs zstd-1.5.7 is significant:

• the C version is >3x faster at any compression level using a single thread
• the C version gets a ~linear speedup when increasing the number of threads, whereas the go version doesn't support parallel compression

So on multicore systems, compression can be easily mean >10x slowdown compared to what's possible.

[polarathene]

The SOCI containerd snapshotter has a config for external decompressor commands (with their parallel mode), with the config for separate commands for gzip or zstd layers I think. That might be a good way to approach it?

---

Performance of klauspost/compress/zstd (Go)

@froody when comparing to existing zstd compression support in BuildKit, be mindful that the compression-level value is mapped to only 4 levels supported by klauspost/compress/zstd.

• That library's documented equivalent zstd compression levels was last updated in 2020, and are not aligned with what BuildKit chose (which seems to be constrained/mapped to the gzip level range?).
• As the issue description notes, actually using standard compression levels of libzstd would be nice to have as an option (presently neither BuildKit nor Docker docs reveal that the Go library used only understands 4 compression levels).

Additionally, your statement about compressing not leveraging multi-threading for concurrency seems inaccurate? (see the library's docs which cover concurrency)

• Since BuildKit calls the zstd compressor without setting the concurrency the default concurrency is instead determined by the GOMAXPROCS ENV.
• If GOMAXPROCS is not set, it defaults to detected logical CPU cores (and from software compiled with Go 1.25 onwards if cgroups are applying CPU constraints to the environment that'll affect the default GOMAXPROCS fallback which may be lower). That linked article on the new fallback also notes how previously environments with those CPU constraints active could regress performance more than expected.

Chances are your test environment may be limiting the concurrency somewhere. Ensure that your comparison is fair against both libraries, otherwise if only one is docker build ... there can be a bunch of other activity actively going on that competes for resources impacting the performance. Better to clarify how you observed this difference?

---

zstd:chunked support

I have not benchmarked this and likely something like zstd:chunked should still be using golang implementation then.

zstd:chunked would be a big win from what I've read. However I don't see much discussion/interest about it on moby, buildkit, buildx, containerd? Has this been delegated to a containerd snapshotter like eStargz?

Podman has build support for this compression format, in addition to the optimized pull support. If a library is required for this feature to be adopted in BuildKit, but an external command is the default for anything else is that acceptable? It would still be faster than pulling and decompressing full layers.

gozstd repo

• Seems like a cleaner implementation but for some bizarre reason that repo vendors prebuilt static libs inside the source code for all arch 🤦

Still the case 4 years later. They do mention why they're doing it in their FAQ (not that I agree with the practice):

This simplifies package installation with go get without the need to perform additional steps for building the libzstd*.a.

You could technically perform a sparse git clone without that dir checked out, then build or supply your own static lib separately (or dynamic link) and have Go build from the local clone?

---

gozstd fork

There's an author of a fork who recently reached out to the eStargz project. That developer is using LLMs to develop and demonstrated a lack of care with their own project commits. They don't appear malicious however, with PR contributions to other projects merged where LLM usage was evident again but didn't seem noticed by the reviewer (or not a concern given feedback was addressed), but I'd be skeptical of adopting their fork, you don't want another serde_yml 😬