Caused by: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=zOS and os.arch=s390x

[sunz21sunz]

Hi Team,
I am using version 1.1.7.3 of 'snappy-java' and getting below error:

Caused by: org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] no native library is found for os.name=zOS and os.arch=s390x

JVM version::
IBM J9 VM (build 2.9, JRE 1.8.0 z/OS s390x-64-Bit Compressed References 20190124_408237 (JIT enabled, AOT enabled)

[odaira]

snappy-java currently does not have a native binary for zOS/s390x. I contributed the binary for Linux/s390x, but I do not have access to zOS now. Let me poke around....

[sunz21sunz]

@odaira Did you find anything or any ETA, we have install around next month that requires this change

[odaira]

@sunz21sunz Would you have any contact in IBM (or other company supporting zOS) on your deal? If the deadline is critical, I would reach out to the contact if any. I'm working for snappy-java on a voluntary basis, so I'll try, but I cannot commit myself to any schedule.

By the way, I've got access to a zOS instance. It has nothing installed, so I'll start with installing git.

[xerial]

Another idea is adding a fallback to pure-java Snappy (like aircompressor) if no native library is found.

[jamesbroadhead]

@xerial do you have any benchmarks for a pure-java implementation vs this one?

[fosslou]

@odaira you can get access to a free LinuxOne (Linux on Z) environment in two ways.

1. The Marist College Linux One Community CLoud https://linuxone.cloud.marist.edu/cloud/#/login
2. The HyperProtect Virtual Server (Free tier) on the IBM public Cloud https://cloud.ibm.com

Good luck

[xerial]

Added a fallback to pure-java snappy implementation in #244 and this is available since snappy-java 1.1.76

[n-marion]

@xerial , the fallback that you implemented in #244 that is based on aircompressor doesn't support BIGENDIAN platforms

static {
        ByteOrder order = ByteOrder.nativeOrder();
        if (!order.equals(ByteOrder.LITTLE_ENDIAN)) {
            throw new SnappyError(SnappyErrorCode.UNSUPPORTED_PLATFORM, format("pure-java snappy requires a little endian platform (found %s)", order));
        }

[kiwi1969]

I came here to report the bug, but @n-marion beat me :)
If run on Z/OS Unix shell with snappy-java 1.1.7.6, and use pure-java option on command line :

Exception in thread "main" org.xerial.snappy.SnappyError: [UNSUPPORTED_PLATFORM] pure-java snappy requires a little endian platform (found BIG_ENDIAN)
at org.xerial.snappy.pure.UnsafeUtil.(UnsafeUtil.java:38)
at org.xerial.snappy.pure.SnappyRawCompressor.writeUncompressedLength(SnappyRawCompressor.java:383)
at org.xerial.snappy.pure.SnappyRawCompressor.compress(SnappyRawCompressor.java:89)
at org.xerial.snappy.pure.PureJavaSnappy.rawCompress(PureJavaSnappy.java:102)
at org.xerial.snappy.Snappy.rawCompress(Snappy.java:424)
at org.xerial.snappy.Snappy.compress(Snappy.java:105)
at com.equifax.usis.acro.Main.main(Main.java:12)

I really want to use snappy compression within Avro, but this LITTLE_ENDIAN bug in aircompressor is preventing us.
[(https://github.com/airlift/aircompressor/blob/master/src/main/java/io/airlift/compress/snappy/UnsafeUtil.java)]

Specifically, it seems that the lines identified by @xerial for LITTLE_ENDIAN should be deleted.

[xerial]

It's not a bug, rather big-endian architecture is not supported in the pure-java fallback version. To use zOS, s390x we need to add a native library for this.

If https://github.com/dockcross/dockcross supports cross-compilation for zOS/s390s, it's the ideal way to support zOS in snappy java because the build will be reproducible, and we don't need to ask IBM team members like @odaira to rebuild native libraries.

[kiwi1969]

I only referred to it as a bug in pure-java, as it seems the only reason why big endian isn't supported is because there was no effort to reverse the byte order to allow it to work. ie It doesn't seem to be something that is completely unresolvable

This link has examples of methods that could be leveraged to provide the big endian functionality
[]http://cr.openjdk.java.net/~jmelvin/7130404/jdk/webrev.04/raw_files/new/src/share/classes/sun/security/provider/ByteArrayAccess.java

[xerial]

Personally I don't have time for working on big-endian support by checking every detail of the compressor/decompressor implementation (e.g., where byte order matters).

If there is a docker image of g++ cross compiler for zOS, I'd be happy to included it as a part of the build process of snappy-java. Otherwise, I need to wait for contributions from open source community.

[kiwi1969]

This link seems to mention a docker image for cross compile of gcc to Z/OS platform (at bottom of blog). [https://blog.the-leviathan.ch/?p=1344]
My company blocks the link, so I am unable to validate.

In the meantime, I might play around with Pure Java and try to get a local version that for Big Endian, does the reversebytes alternative that seems to be needed around every UNSAFE.getXXX() or UNSAFE.putxxx invocation

[fosslou]

You should be able to test this on Linux on Z... which you can get a VM on the Marist College LInuxOne Community Cloud here: https://developer.ibm.com/components/ibm-linuxone/gettingstarted/

…

On Tue, Aug 25, 2020 at 12:26 PM kiwi1969 ***@***.***> wrote: This link seems to mention a docker image for cross compile of gcc to Z/OS platform (at bottom of blog). [https://blog.the-leviathan.ch/?p=1344] My company blocks the link, so I am unable to validate. In the meantime, I might play around with Pure Java and try to get a local version that for Big Endian, does the reversebytes alternative that seems to be needed around every UNSAFE.getXXX() or UNSAFE.putxxx invocation — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#232 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABHJIVXR2AVESFQLKY55OEDSCPQZTANCNFSM4I7CYR3A> .

-- I/T Architect

[kiwi1969]

@fosslou - I am using Z/OS native, not ZLinux.
So far I have a version of libsnappy.so natively compiled on z/os using c89, without any GNU tools, based upon snappy.c implementation here: [https://github.com/yitzikc/snappy-c]. I am yet to add the JNI stubs to allow JNI to work.
I am not sure if this actually works properly yet, or if it is 100% compatible with dll that snappy-java uses, so I am thinking I may be better off (and perhaps easier) to attempting an update to pure-java for big endian.