symbol lookup error after upgrade to 1.5.1

[jarthod]

Hello 👋

I just tried upgrading from 1.5.0.0 to 1.5.1.0 but I'm getting this exception: symbol lookup error: /home/adrien/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/zstd-ruby-1.5.1.0/lib/zstd-ruby/zstdruby.so: undefined symbol: HUF_decompress4X1_usingDTable_internal_bmi2_asm_loop which is causing the ruby program to exit with status 127. (the gem installation and native extension compilation goes without problem).

I have the same problem on two different machines running Ubuntu 21.10 and 21.04 (if that's of any relevance). I'm using ruby 2.6.4 and I'm using zstd-ruby only for the mongo driver (version 2.17.0). This is the commit showing how the mongo gem is using zstd-ruby: mongodb/mongo-ruby-driver@c07aae3, nothing very special here IMO, looks like it's just basic Zstd.decompress and Zstd.compress.

I can't reproduce with a simple console test:

> Zstd.compress('test')
=> "(\xB5/\xFD \x04!\x00\x00test"
> Zstd.decompress(Zstd.compress('test'))
=> "test"

But I can reproduce reliably with any Mongo query:

> User.first
rails_console: symbol lookup error: /home/adrien/.rbenv/versions/2.6.4/lib/ruby/gems/2.6.0/gems/zstd-ruby-1.5.1.0/lib/zstd-ruby/zstdruby.so: undefined symbol: HUF_decompress4X1_usingDTable_internal_bmi2_asm_loop

(IRB exits with status 127)

Any idea?
Thanks!

[SpringMT]

I try to fix this problem in #38.
Can you test using this branch fix-asm-support?
Ex(in Gemfile)

gem "zstd-ruby", branch: "fix-asm-support"

[SpringMT]

I released v1.5.1.1 fixed the error.
https://rubygems.org/gems/zstd-ruby/versions/1.5.1.1.
Please test it!

[jarthod]

Thanks for the quick fix ! 1.5.1.1 seems to be working fine :)
For my understanding what is "ASM support"? why wasn't my simple test failing?

[SpringMT]

Zstd Compression are 4 block types.
https://github.com/facebook/zstd/blob/dev/doc/zstd_compression_format.md#blocks

I think your simple text compression is Raw_Block.
00 that is 1 byte before test in the result string "(\xB5/\xFD \x04!\x00\x00test" shows Raw_Block.

Raw_Block doesn't use ZSTD_decompressBlock_internal that uses Compressed_Block.
https://github.com/facebook/zstd/blob/e34e58b458d6460b7eabdf4bde9d6b7da90a842d/lib/decompress/zstd_decompress.c#L917-L928

ZSTD_decompressBlock_internal -> ZSTD_decodeLiteralsBlock -> HUF_decompress4X_usingDTable_bmi2
https://github.com/facebook/zstd/blob/d5933a40096605d13b9a21269c6c34d06d66013b/lib/decompress/zstd_decompress_block.c
https://github.com/facebook/zstd/blob/d5933a40096605d13b9a21269c6c34d06d66013b/lib/decompress/zstd_decompress_block.c#L177-L204
https://github.com/facebook/zstd/blob/252ef866fb0f018ea0ff596c49027201442d7b00/lib/decompress/huf_decompress.c

HUF_decompress4X_usingDTable_bmi2 -> HUF_decompress4X1_usingDTable_internal

In HUF_decompress4X1_usingDTable_internal, ZSTD_ENABLE_ASM_X86_64_BMI2 macro is used.

https://github.com/facebook/zstd/blob/859e0500ab98e07c680b270efa0f2dcdf11391cf/lib/common/portability_macros.h#L122-L129

This macro is for ASM macros .
facebook/zstd#2789
facebook/zstd#2722
facebook/zstd#2893

The x86-64 Huffman assembly implementation is .S files.

When 1.5.1.0 release, .S files didn't include src, and maybe your environment matches ZSTD_ENABLE_ASM_X86_64_BMI2, so HUF_decompress4X1_usingDTable_internal -> HUF_decompress4X1_usingDTable_internal_bmi2_asm -> HUF_decompress4X1_usingDTable_internal_bmi2_asm_loop then the error happened.

[jarthod]

I see, thanks for the explanation!