Fix dedicated dict search isSupported() requirements.#2540
Merged
senhuang42 merged 1 commit intofacebook:devfrom Mar 18, 2021
Merged
Fix dedicated dict search isSupported() requirements.#2540senhuang42 merged 1 commit intofacebook:devfrom
senhuang42 merged 1 commit intofacebook:devfrom
Conversation
felixhandte
approved these changes
Mar 17, 2021
Contributor
There was a problem hiding this comment.
Cool. Looks good.
As I recall, these asserts are present for the following reasons:
assert(chainLog <= 24);
In the DDSS, the hashtable layout for each entry is:
- 4 bytes: first match index
- 4 bytes: second match index
- 4 bytes: third match index
- 1 byte: number of additional matches in the chain table
- 3 bytes: index of head of chain in chain table
So we only have 24 bits to index into the chain table.
assert(hashLog >= chainLog)
We need this for DDSS table construction, because we first build the chain table in the hash table, and then sort the chains into the chain table proper.
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Currently, asserts in
zstd_lazy.cwill fail if you try to compress withchainLog > 24orhLog < cLogwith dedicated dict search.The following command fails on
dev:make -j zstd DEBUGLEVEL=1 && ./zstd [file] -D [dictFile] --zstd=clog=25,hlog=23 -7fThis PR adds the same checks that are
assert()ed to be true inzstd_lazy.cwhen using dedicated dict search, and includes a unit test that fails without this fix.Though, I will note that dedicated dict search at least seems to work fine when
clog > 24orhLog < cLog, so an alternative could just be to remove the asserts.