2x faster compression

[M-Gonzalo]

I've done some tests to determine whether Conor's fastlzma2 library could actually benefit precomp. The answer is yes and by a long shot. The tests have proven that the novel algorithm provides about 2x faster compression, all while compressing up to 10% more and using less memory than multi-threaded lzma. So it's basically double speed for free.

Because of this...

The library can compress using many threads without dividing the input into large chunks which require the duplication of the match-finder tables and chains. Extra memory used per thread is typically no more than a few megabytes.

... the difference on ratio is more visible when working on a high number of threads, which on mt-lzma usually means chunking the input and losing long range compression, but with flzma2 doesn't.

[M-Gonzalo]

Precomp mt-lzma compression VS xz-fastlzma2 comparison and report:

Sumary:

Weighted average speedup for fxz: 195%
Weighted average speedup for fxz -e: 168%
Weighted average ratio gain for fxz: 2.03%
Weighted average ratio gain for fxz -e: 2.20%

Details:

Speed = (size of file.pcf_cn) / (time of [ precomp || fxz ])
Ratio = (size of file) / (size of file.pcf_cn.[ precomp || fxz ])

Sets:
* zpaqgui-win32.win32.x86_64.zip
* APKs.zip, which is just a few Android apps in a zip
    -both after "precomp -cn -intense -brute -t-3 -packjpg-" (because of ARM packjpg and packmp3 bugs)

Machines:
         Android linux, Termux environment, 8-core ARM Cortex A53, 4gb RAM
         Manjaro linux, 4 logical threads Intel Atom, 2gb RAM
         Ubuntu linux, 2-core Intel Xeon, 16gb RAM

Methods:
        precomp = precomp -t+ -e file.in
        fxz= fxz -z -9 -T0 -c file.in >file.in.fxz
        fxz 'extreme'= fxz -z -9 -T0 -c -e file.in >file.in.fxze

===========================================================
zpaqgui-win32.win32.x86_64.zip (pre-compresed):

On ARM Cortex A53, 8 threads:

55.83% - 2906 kbps => fxz 'extreme' [185% faster, 1.4% stronger]
56.00% - 3148 kbps => fxz           [200% faster, 1.2% stronger]
57.22% - 1574 kbps => precomp

On x86-64 Intel Atom, 4 threads:

55.89% - 1095 kbps => fxz 'extreme' [184% faster, 0.65% stronger]
56.03% - 1281 kbps => fxz           [215% faster, 0.51% stronger]
56.54% - 595 kbps  => precomp

On x86-64 Intel Xeon, 2 threads:

55.87% - 3675 kbps => fxz 'extreme' [223% faster, 0.01% stronger]
55.88% - 1650 kbps => precomp
56.03% - 4264 kbps => fxz           [258% faster, 0.45% weaker]

===========================================================
APKs.zip (pre-compresed)

On ARM Cortex A53, 8 threads:

63.09% - 3193 kbps => fxz 'extreme'   [119% faster, 9.8% stronger]
64.06% - 3721 kbps => fxz             [139% faster, 8.8% stronger]
72.89% - 2686 kbps => precomp

On x86-64 Intel Atom, 4 threads:

63.89% - 1244 kbps => fxz 'extreme' [161% faster, 7.32% stronger]
64.05% - 1502 kbps => fxz           [195% faster, 7.16% stronger]
71.21% - 772 kbps  => precomp

On x86-64 Intel Xeon, 2 threads:

63.88% - 4181 kbps => fxz 'extreme'  [173% faster, 2.85% stronger]
64.03% - 4946 kbps => fxz            [205% faster, 2.70% stronger]
66.73% - 2413 kbps => precomp

===========================================================

These are only two datasets that are representative of several other more informal tests during the past months.
BTW: fxz uses ~900-1000m RAM on the default configuration, instead of the ~1300 that precomp does, although it can be configured to use a bigger dictionary and, in doing so, provide even better compression.