fix sequence compression API in Explicit Delimiter mode#3023
Conversation
Cyan4973
force-pushed
the
fix_seqCompress_withDelimiter
branch
2 times, most recently
from
0f2816a to
55f5946
Compare
|
Looks like theres an assert in the |
terrelln
left a comment
terrelln
left a comment
There was a problem hiding this comment.
LGTM once the tests pass
lib/common/error_private.c
Outdated
| case PREFIX(frameParameter_unsupported): return "Unsupported frame parameter"; | ||
| case PREFIX(frameParameter_windowTooLarge): return "Frame requires too much memory for decoding"; | ||
| case PREFIX(corruption_detected): return "Corrupted block detected"; | ||
| case PREFIX(corruption_detected): return "Input corruption detected"; |
There was a problem hiding this comment.
nit: Maybe "Data corruption detected"? It feels more natural than "Input corruption"
Cyan4973
force-pushed
the
fix_seqCompress_withDelimiter
branch
from
55f5946 to
87fb8a5
Compare
lib/compress/zstd_compress.c
Outdated
| cctx->blockSize, remaining, | ||
| inSeqs, inSeqsSize, seqPos); | ||
| U32 const lastBlock = (blockSize == remaining); | ||
| assert(blockSize <= remaining); |
There was a problem hiding this comment.
Looks like this assert is failing.
Maybe it just needs to be moved below the FORWARD_IF_ERROR() check?
There was a problem hiding this comment.
Yes, that's exactly the issue,
blockSize can be an error code, so this assert() is only valid after the FORWARD_IF_ERROR().
Cyan4973
force-pushed
the
fix_seqCompress_withDelimiter
branch
2 times, most recently
from
c89e7c8 to
6096d33
Compare
add explicit delimiter mode to libfuzzer test
Cyan4973
force-pushed
the
fix_seqCompress_withDelimiter
branch
from
6096d33 to
fc2ea97
Compare
|
The misplaced |
3 participants
In Explicit Delimiter mode, the Sequence compression API was only working when
the frame has only a single block
or when all blocks are full (except the last one).
This fix makes is possible to select block boundaries wherever the caller wants them.
It still requires that data conforms to
zstdspecification, aka block sizes still have a maximum.The fuzzer capability has been extended to test the explicit delimiter mode too.