Skip to content

Add a crate for wasm zstd implementation#52

Merged
szegedi merged 4 commits intomainfrom
szegedi/zstd
May 28, 2025
Merged

Add a crate for wasm zstd implementation#52
szegedi merged 4 commits intomainfrom
szegedi/zstd

Conversation

@szegedi
Copy link
Copy Markdown
Contributor

@szegedi szegedi commented May 27, 2025

Exposes the Rust zstd compression implementation, the same one used by libdatadog profiler, compiled to WASM. This is primarily meant to be used by Node.js profiler to compress profiles with zstd on Node.js versions earlier than 23.8.0, when zstd support was introduced. This implementation is somewhat inferior to the built-in one in the sense that it is synchronous, but if that proves to be a problem we can spin off a worker thread for it.

libdatadog is soon shipping a version that can use zstd as a compression algorithm. We could've relied on its profiling crate and I could've asked for an API endpoint, basically "here, compress this byte array". But then we'd have to ship the libdatadog profiler crate as a dependency. It seemed much simpler to bind directly to zstd_rs even though it's not a libdatadog binding. It does serve us to get at feature parity with libdatadog though. The generated WASM component clocks at 310kB.

The PR also fixes minor issues that cropped up with there now being more than one WASM component in the repo. Big thanks to @BaptisteFoy for blazing the trail of adding support for WASM crates in the first place!

One known problem is that clang on macOS seems to have trouble compiling zstd to wasm unless brew install llvm is used. However, when Homebrew llvm is used then a different issue crops up. I eventually managed to work around all of that in a build script so local development work isn't blocked on macOS. (The project builds correctly without any modification on Linux, where we release.)

Jira: PROF-11878

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 27, 2025

Overall package size

Self size: 30.12 MB
Deduped: 30.12 MB
No deduping: 30.12 MB

Dependency sizes | name | version | self size | total size | |------|---------|-----------|------------|

🤖 This report was automatically generated by heaviest-objects-in-the-universe

@szegedi szegedi requested review from BaptisteFoy and rochdev May 27, 2025 12:46
Copy link
Copy Markdown
Member

@BridgeAR BridgeAR left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I am not super fond of introducing 300kb for it but I guess memory is still less important in most cases.

@szegedi szegedi merged commit 7b2798c into main May 28, 2025
42 checks passed
Copy link
Copy Markdown
Contributor

@BaptisteFoy BaptisteFoy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🚀

szegedi added a commit that referenced this pull request Jun 3, 2025
* Handle wasm-pack's dash-to-underscore filename transformation

* Add a crate for wasm zstd implementation

* Only run tests for a single crate

* Fix WASM builds on macOS
@szegedi szegedi mentioned this pull request Jun 3, 2025
szegedi added a commit that referenced this pull request Jun 3, 2025
* Handle wasm-pack's dash-to-underscore filename transformation

* Add a crate for wasm zstd implementation

* Only run tests for a single crate

* Fix WASM builds on macOS
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants