-
Notifications
You must be signed in to change notification settings - Fork 3.7k
Optimise coding bit operation in BE #4366
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Optimise coding bit operation in BE #4366
Conversation
|
Not intended to interfere, just curious about how many improvements can be achieved from this PR, are there any benchmarks? |
In my develop computure, when encode_varint64 execute 1 billion times. |
It seems too slow to execute 5 bilion enocde operation in about 95s |
5 billion costs 95s * 5. Compression is time consuming. This encode_varint64 is mainly used to compress low-bit int to variable length instead of int64_t type. It is trade-off between time and space. |
|
@imay From the point of view of implementation, our implementation should be copied from leveldb, but leveldb has removed some unnecessary bit operation in this pr google/leveldb@cf1b5f4#diff-0f30ecb01c7888631c52ec595204f7f0 , so I think we can accept this pr and change to the same implementation as level db, although this change improves performance not particularly big |
Optimize bit operation in variable length coding. Remove unnecessary bit operation.
Proposed changes
Optimise bit operation in variable length coding. Remove unnecessary bit operation.
Detail description: #4367
Types of changes
What types of changes does your code introduce to Doris?
Put an
xin the boxes that applyChecklist
Put an
xin the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.