Add ZSTD compression

[dbeinder]

These patches integrate zstd compression into NSIS. Current state:

• Tested on MSVC 2019 and Debian buster
• In my limited testing, zstd at level 9 beats zlib & bzip2 slightly in compression ratio and strongly in speed
• For the absolute best compression, lzma is still the best option, but everywhere else on the speed/size spectrum zstd seems to offer the best tradeoff with extremely wide tunability: https://facebook.github.io/zstd/
• Zstd stubs are 75kiB 60kiB
  • Decompression speed is 170MB/s on my machine, but there should still be room for improvement
  • Zstd decompression is always single-threaded, unless exehead is extended to decompressing multiple files in parallel
  • Zstd requires between 2-10MB memory depending on zstd level (1-19)
• Compression in makensis is multithreaded
  • Without further changes this only helps for larger files or when using solid mode
  • Zstd compression uses 20MB per thread at level 9, and about 150MB per thread at level 19
  • Currently it just uses all cores, should we add a script setting to limit this?
• I have made some minimal changes to zstd and added a diff similar to 7zip
  • Once change could be avoided if there is a way to force the compiler to emit code for byteswap intrinsics instead of calling into the CRT
  • All other changes are #ifndef EXEHEAD to strip unused features and reduce code size of the stubs
• The remaining uglyness is necessary to link zstd without a CRT into the stub
  • Zstd requires memcpy, memset, memmove, malloc, calloc, free, and _allmul (int64 multiplication on x86)
  • For now, I have put simple implementations into exehead/crt_replacement.c, but I see there is at least another memcpy implementation in NSIS, those should probably be centralized. Anyway it would be nice to have an optimized version of this code if it is actually used beyond array initialization.
• Documentation updated, zstd License (BSD) added
  • Documented SetCompressionLevel script command to control compression level for zstd (and zlib)
• MakeNsisW & Zip2Exe updated

[dbeinder]

Rebased and fixed the last warnings on OSX/Linux.
Default compression level: I left it at 9, because ZSTD beats BZIP2 for anything I tested and still compresses twice as fast.
I documented SetCompressionLevel because I feel the major advantage of ZSTD is the flexibility and the wide range.

Outstanding:

• Adding ZSTD to the GUI in MakeNsisW and Zip2Exe. What editor is used for the resource script?

[sredna]

Outstanding:

* Adding ZSTD to the GUI in MakeNsisW and Zip2Exe. What editor is used for the resource script?

You can probably use Visual Studio although I personally just use Resource Hacker and/or hand tweaking in Notepad.

There are a few remaining issues for older toolchains and they are rather annoying.

[dbeinder]

Updated MakeNsisW, zip2exe and added your changes to the second commit

[dbeinder]

Last push added a warning when trying to set a compression level for bzip2.
I think it's ready for review

[dbeinder]

Are there any changes you would like me to make? Should I submit the commits via svn on sourceforge? I'm really not familiar with the development workflow of NSIS or svn in general ;)

[sredna]

Your allmul function is just a copy from the MS CRT source? We can't really accept that.

Unless you are willing to install Visual Studio 6 there is probably more work to be done on our end before this can be merged. The multithreaded compressor (if enabled) uses some Vista+ functions and the stubs are missing some other 64-bit math stuff.

[dbeinder]

Re: allmul, I believe this is CRT source, I'll have retrace where I found it, but I don't think it is a copyright issue though:

• API/Interfaces are generally not copyrightable (here: the placement of the parameters in the stack, the registers used for return)
• Once you ignore all lines of code required to satisfy the CRT-API/interface you are left with something so trivial, that I don't see it crossing the threshold of originality for copyright to apply

I have installed VS6 in a Win XP VM, and ugh...
(btw, VS6 compiled the current master only with this change: dbeinder@a99568e)

Making zstd compile under C90 without native support for 64bit types and literals is going to be messy. I hoped zstd could be used with minimal changes, so the lib can be updated in the future without much effort. What is the rationale for sticking with C90 and the VS6 toolchain? Are newer compilers not able to generate executables that run on old versions of Windows? Is the compatibility target of Windows 95 still accurate, both for stubs and makensis?

I can definitely look into the imports for multi threading in makensis.

[sredna]

VS6 creates the smallest code and lets us target Win95. Makensis and friends currently work on 2000 and later but we can probably bump that to XP. Or just disable multithreading when using older compilers.

You can use the INT64 type in our code and whatever else VS6 allows, we don't care about MinGW that old.

[dbeinder]

Apparently the only thing that prevents the stubs compiled by MSVC 2019 from running on W95 is the use of SSE (can be disabled in in modern MSVC) and the PE Header OS & subsystem compatibility fields.
There already is some PE hackery in NSIS, so I extended that, and it works. I tested zstd and the Dialogs plugin on a W95 VM.

Since stubs produced by MSVC 2019 are only 1kb larger than the official v3.06.1, and there is still link time optimization that could be used, I think dropping VS6 should be considered. Especially for future developments on 64bit, losing the dependency on a C90 compiler would make many things easier.
The zstd stubs produced by GCC on Debian were already compatible with W95 but they are all 2x - 3x larger than MSVC.

The multithreading imports are because of ConditionVariables API. I have gone with the minimal effort solution for now: Pull them in using GetProcAddress and disable multithreading if they are missing. Tested on WinXP.
Alternatively makensis could be linked with (https://sourceware.org/pthreads-win32/) which simulates those APIs, pthread is directly supported by zstd.

I put those changes on a separate branch: https://github.com/dbeinder/nsis/commits/zstd_xp for now

[sredna]

Can you test the Math plug-in compiled by 2019 on W95? It is allowed to use the static CRT library.

[dbeinder]

The Math plugin was a problem, the MSVC2019 CRT is >100kB and uses imports that require WinXP.
I have now added a standalone sprintf implementation and math functions (from FreeBSD libm) to the plugin source.
It works on W95 and the Math plugin is now 31kB, half the size of Math.dll in the 3.06.1 build.
The floating point math should select the correct endianness / memory layout when targeting ARM, but I don't have such hardware to test.

Currently in this branch: https://github.com/dbeinder/nsis/commits/zstd_xp
I can move it into this PR if you are okay with the approach of dropping VS6 support.

[kichik]

Please open s new PR or reopen. We had to force push since my local git copy got corrupted.

[justanotheranonymoususer]

I'd like to get and to speed up things in my flow. @dbeinder any update? Thanks for working on this.

[sredna]

The floating point math should select the correct endianness / memory layout when targeting ARM

Windows is always little-endian so for Math.dll that is not an issue.