Incompatibility between --long=31 and -22 --ultra?

[danluu]

I tried to compress a file with zstd -22 --ultra --long=31 and then decompress it with zstd -d --long=31 and got the following error:

tmp.tar.zst          : 6031 MB...     tmp.tar.zst : Decoding error (36) : Corrupted block detected

Is this a known issue? Just checking before I try to create a repro and/or debug this since it takes about 2 hours to compress the file (it's ~7GB), so it may take a while to reproduce the issue. Unfortunately, the file I'm compressing isn't something I can pass around, so I can't just send it to you.

I've tried this with the same file on two different platforms (different hardware and different Linux kernel versions) and have run into the same things both times. In both cases, the code I had was built from c0eb6c9, by running with make with no extra options specified.

[Cyan4973]

Hi @danluu ,
this seems related to #1653 , which was recently fixed.
Try commit 9038579 (or latest dev) if possible.
Indeed, testing high compression levels on very large data set is quite lengthy ...

[danluu]

Thanks for the quick response! I still can get an error with 096714d, although it happens at a slightly different place when decompressing:

$ zstd -d --long=31 tmp.tar.zst 
tmp.tar.zst          : 6115 MB...     tmp.tar.zst : Decoding error (36) : Corrupted block detected 

From the linked issue, it sounds like it's possible that the it doesn't matter what data is used and only the size & settings matter? Let me see if I can get a repro with an all 0s file of the same size or a /dev/urandom file of the same size. If that doesn't work, I can try generating some synthetic data that's less unnatural.

[danluu]

I wasn't able to reproduce with a toy file from /dev/*, but the file from #1653 (http://kirill.med.u-tokai.ac.jp/data/temp/Picea%20abies%20[genbank%20GCA_5F900067695.1%202016-11-09].fna.gz) gives me a similar looking error.

Because the file is huge, I compressed with 32 threads, zstd -22 --ultra --long=31 -T32. On decompression (zstd -d --long=31), I get 6178 MB... tmp.fna.zst : Decoding error (36) : Corrupted block detected when using 096714d.

[Cyan4973]

Thanks for the feedback @danluu ,
and sorry for the long time it takes testing.

This issue was supposed to be fixed by #1659 .
It was tested with the same fna file, so it's surprising that it's still there.

Maybe the fix doesn't work well enough.
There is also a small remaining possibility that the issue was actually correctly fixed in #1659, corresponding to commit 9038579, but sometimes later another commit generated a problem featuring same symptom (I'm suspecting 857e608, but that's just a wild guess).

To be investigated ...

[danluu]

No problem! Another curious thing is that I also tried single-threaded compression for a few "large" (> 10GB) files and they all hung at 6144MB compressed. One of them was the filed linked above and two of them were files pull of prime numbers that compressed and decompressed correctly when compressed with -T32.

For all of the above, I built the update with make and didn't do make clean, I did a make clean and started all of these again on the off chance it's due to an issue with the Makefile.

[Cyan4973]

That's a very interesting data point, although there is a potential of confusion when labelling "single-threaded". It could be either :

• pass the --single-thread command
• use -T1 (which is the same as default)

--single-thread is the "real" single-threaded version. It's equivalent to ZSTD_compressStream(). This version is supposed to be immune to bug in #1653, and if you nonetheless find that it still fails, then there is something else going on.

-T1 is actually "multi-threaded" by design, and just happens to use only one worker thread. The rest of the machinery works exactly the same as if it was N >= 2 threads, meaning there is a separate I/O thread, and work is split into pre-defined job sizes which are then passed onto workers, which preload data, etc. As a side-note that's why zstd generates exactly the same compressed content whatever the nb of threads (for a given a version and compression level).

So -T1 will suffer exactly the same issue as any -T#, while --single-thread should not.

[danluu]

Thanks for the clarification! My previous comment refers to default threading, no T specified.

[Cyan4973]

Yes, so default is equivalent to -T1, which means, it uses the multi-thread engine, with only 1 active worker.
That's the setup where we identified the issue with --long=31.

[Cyan4973]

Also, the nature of the bug (presuming it's the same one) is that it overflows internal indexes.
And in order to skip all defenses related to index overflow, there are multiple conditions :

• Use a window size of 2 GB (--long=31)
• Use Multi-threading mode with overlapLog==9 (automatically the case with highest compression levels), thus triggering a history preload of 2 GB
• Re-use an existing compression context (CCtx), where the starting index is already high enough to trigger the overflow

Re-using an existing context is something which is done automatically internally. It will necessarily happen if nb_jobs > nb_workers.

But as a consequence, when specifying a very large number of workers, it can reach a point where nb_workers >= nb_jobs, hence each CCtx is only used once. And therefore, the bug should no longer be triggered.

This refers to your test using -T32 "that compressed and decompressed correctly". I don't remember what's the size of a job with --long=31 setting, but I would hazard that it's probably 2 GB. Thus if the file is < 64 GB, -T32 will create more workers than necessary, and each worker will use a brand new CCtx, thus circumventing the problematic re-usage pattern.

[Cyan4973]

Finally, I should have mentioned it earlier :

zstd can also be compiled with all its assert() turned on, so that it would ideally abort during compression, hence much closer to the effective fault point.

This is done by using build macro DEBUGLEVEL

make V=1 clean zstd MOREFLAGS=-DDEBUGLEVEL=1

is one way to produce such a binary.

[danluu]

Thanks, I'll try that in the future!

I was able to reproduce both the failure on decompression and the hang on compression after 6144MB with default T / T=1 with a clean build.

On the hangs, gdb shows them hanging at

(gdb) backtrace
#0  0x00007f756d4e8965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000000046b5a5 in ZSTDMT_waitForLdmComplete ()
#2  0x000000000046b73a in ZSTDMT_tryGetInputRange ()
#3  0x00000000004711cc in ZSTDMT_compressStream_generic ()
#4  0x000000000041ee4b in ZSTD_compressStream2.part.29 ()
#5  0x00000000004b9ec5 in FIO_compressFilename_srcFile ()
#6  0x00000000004bc494 in FIO_compressMultipleFilenames ()
#7  0x00000000004067a1 in main ()

The other spawned process looks less interesting?

#0  0x00007f756d4e8965 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x000000000040a04b in POOL_thread ()
#2  0x00007f756d4e4dd5 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f756d20dead in clone () from /lib64/libc.so.6

[Cyan4973]

Thanks @danluu !
That's very helpful, it gives a clear hint of where to look for !

Indeed, #1659 cleared the issue for the main compression path, but ldm has a separate index data structure, and may well be prone to the same problem.

[phiresky]

I've had this happen for multiple different files when using -T0 --long=31 --ultra -22 and I'd say it's a very severe error because it silently corrupts your data making it impossible to recover - thankfully I still didn't trust zstd enough yet and tried decompressing before deleting the original files first...

Also, it's really not a niche use case because it happened to me for at least three different files and the above flag combination is simply the strongest compression level you can easily find from help. (As in, in my opinion you should release this as a hotfix as soon as possible and mark it as the severe problem it is so people can verify if they unknowingly corrupted their data)

[terrelln]

This is an important bug to fix, and I'll look into it this week. If you replace --long=31 with --zstd=wlog=31 you will probably work around the bug.

I'd argue it is a niche use case since it only occurs with large data on the highest settings, and most users won't spend hours compressing their data. However, just because it is a niche use case doesn't mean it isn't worthy of a quick fix! We take any data corruption bugs extremely seriously.