perf: implement bitpack encoding for LID and MID blocks by cheb0 · Pull Request #328 · ozontech/seq-db

cheb0 · 2026-01-26T06:17:58Z

Description

Replaces varint encoding with faster delta bitpacking. Both LID and MID blocks now use bitpack. Currently, intcomp library is used. The lib doesn't utilize SIMD, so we might update to something else in future.

Measurements

compression
bitpack compresses a lot better: for varints zstd compresses with ratio ~1.7-2.0 while it only compresses delta bitpacked data with ratio ~1.3. Therefore we potentially can disable zstd on benchmarks with a slight dataset size overhead.

dataset size
Overall, we reach approximately same dataset size. For some envs there is a small benefit of around -3% of total dataset.

Search latency (stg fractions repacked with bitpack)

Query	Type	Ids	Total	cold, ms	hot, ms	cold (branch), ms	hot (branch), ms	cold diff	hot diff
service:W AND message:"A B C D" (found)	reg	100	0	5.68	0.13	5.29	0.08	-6.9%	-38.5%
service:W AND message:"A B C D" (not found)	reg	0	0	37.27	0.53	16.19	0.55	-56.6%	3.8%
service:X AND level:3 AND NOT (x6)	reg	0	0	66.21	2.13	32.39	2.02	-51.1%	-5.2%
service:X AND level:[0 to 4]	hist	100	1617439	185.71	50.86	99.57	42.61	-46.4%	-16.2%
zone:Z AND service:X AND level:4 AND job_id:J AND message:"X Y Z"	hist	100	1791	76.59	5.13	32.4	4.81	-57.7%	-6.2%
k8s_container:C AND k8s_namespace:N AND service:X AND level:4 AND source:S	hist	100	11798	93.15	18.97	48.08	18	-48.4%	-5.1%
service:W AND message:"X Y Z A" AND logger:L AND source:S	hist	100	260	68.24	2.52	34.45	1.51	-49.5%	-40.1%
service:W \| group by k8s_pod	agg	100	170764	259.34	188.19	231.81	190	-10.6%	1%
service:O \| group by k8s_pod	agg	100	1619624	273.8	199.66	235.28	201.4	-14.1%	0.9%
service:W \| group by k8s_pod (timeseries)	agg	100	170764	467.04	274.9	405.11	268.39	-13.3%	-2.4%

Fixes #312

I have read and followed all requirements in CONTRIBUTING.md;

cheb0 · 2026-03-12T10:02:52Z

@seqbenchbot up main bulk

seqbenchbot · 2026-03-12T10:02:55Z

Nice, @cheb0 <(-^,^-)=b!

Your request was successfully served.
Identificator for your ongoing benchmark - dc2d4d40.

Here is a list of helpful links:

Take a look at Grafana dashboard;
Live-tailing logs are also available;

Have a great time!

github-actions · 2026-03-12T10:10:32Z

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`AggDeep/size=10000-4`	`03387c`	`b4fae3`
	`45145.00 ns/op`	`52104.00 ns/op`	`1.15`	`🔴`
`AndTree/size=1000-4`	`03387c`	`b4fae3`
	`4.26 ns/op`	`4.85 ns/op`	`1.14`	`🔴`
`Block_Pack-4`	`------`	`b4fae3`
	`NaN B/op`	`27344.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`4.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`82987.00 ns/op`	`NaN`	`🔴`
`Block_Unpack-4`	`------`	`b4fae3`
	`NaN B/op`	`262186.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`2.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`80490.00 ns/op`	`NaN`	`🔴`
`FindSequence_Random/small-4`	`03387c`	`b4fae3`
	`6427.67 MB/s`	`5452.96 MB/s`	`0.85`	`🔴`
	`40.85 ns/op`	`46.95 ns/op`	`1.15`	`🔴`

cheb0 · 2026-03-12T10:12:28Z

@seqbenchbot down dc2d4d40

seqbenchbot · 2026-03-12T10:12:31Z

Nice, @cheb0 <(-^,^-)=b!

The benchmark with identificator dc2d4d40 was finished.
I've prepared a summary for you. Click on Show summary button to see it:

Show summary

Query	Type	`mean (ms)`			`stddev (ms)`			`p(50) (ms)`			`p(95) (ms)`			`p(99) (ms)`			`iterations`
Query	Type	base	comp	diff	base	comp	diff	base	comp	diff	base	comp	diff	base	comp	diff	base	comp	diff
`bulk`	warm	`60.99`	`60.79`	`-0.33%`	`23.17`	`22.77`	`-1.74%`	`54.00`	`54.00`	`0.00%`	`107.00`	`107.00`	`0.00%`	`146.00`	`145.00`	`-0.68%`	`9694.00`	`9652.00`	`-0.43%`

Have a great time!

github-actions · 2026-03-12T10:53:28Z

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`Block_Pack-4`	`------`	`035be1`
	`NaN B/op`	`27344.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`4.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`72861.00 ns/op`	`NaN`	`🔴`
`Block_Unpack-4`	`------`	`035be1`
	`NaN B/op`	`262183.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`2.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`73565.00 ns/op`	`NaN`	`🔴`
`Indexer-4`	`03387c`	`035be1`
	`721669786.00 B/op`	`805572232.00 B/op`	`1.12`	`🔴`

github-actions · 2026-04-10T09:10:26Z

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`Block_Pack-4`	`------`	`4f96b8`
	`NaN B/op`	`113400.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`19.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`109098.00 ns/op`	`NaN`	`🔴`
`Block_Unpack-4`	`------`	`4f96b8`
	`NaN B/op`	`262193.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`2.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`89462.00 ns/op`	`NaN`	`🔴`

github-actions · 2026-04-15T12:49:18Z

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`Block_Pack-4`	`------`	`292ee9`
	`NaN B/op`	`113400.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`19.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`108668.00 ns/op`	`NaN`	`🔴`
`Block_Unpack-4`	`------`	`292ee9`
	`NaN B/op`	`262194.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`2.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`90967.00 ns/op`	`NaN`	`🔴`

github-actions · 2026-04-21T10:21:38Z

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`Block_Pack-4`	`------`	`65fb06`
	`NaN B/op`	`113400.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`19.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`106561.00 ns/op`	`NaN`	`🔴`
`Block_Unpack-4`	`------`	`65fb06`
	`NaN B/op`	`262192.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`2.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`87751.00 ns/op`	`NaN`	`🔴`

github-actions · 2026-04-21T10:37:02Z

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`Block_Pack-4`	`------`	`b8671d`
	`NaN B/op`	`113401.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`19.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`107155.00 ns/op`	`NaN`	`🔴`
`Block_Unpack-4`	`------`	`b8671d`
	`NaN B/op`	`262192.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`2.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`87578.00 ns/op`	`NaN`	`🔴`

# Conflicts: # cmd/index_analyzer/main.go # config/frac_version.go # frac/fraction_test.go # frac/remote.go # frac/sealed.go # frac/sealed/sealing/index.go

github-actions · 2026-05-12T07:25:02Z

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`Block_Pack-4`	`------`	`30869f`
	`NaN B/op`	`113400.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`19.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`91546.00 ns/op`	`NaN`	`🔴`
`Block_Unpack-4`	`------`	`30869f`
	`NaN B/op`	`262189.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`2.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`80694.00 ns/op`	`NaN`	`🔴`

github-actions · 2026-05-12T07:33:33Z

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`Block_Pack-4`	`------`	`60c1be`
	`NaN B/op`	`113400.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`19.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`92272.00 ns/op`	`NaN`	`🔴`
`Block_Unpack-4`	`------`	`60c1be`
	`NaN B/op`	`262187.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`2.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`78120.00 ns/op`	`NaN`	`🔴`

cheb0 · 2026-05-12T07:35:30Z

@seqbenchbot up main bulk

seqbenchbot · 2026-05-12T07:35:36Z

Nice, @cheb0 <(-^,^-)=b!

Your request was successfully served.
Identificator for your ongoing benchmark - d9e8e68e.

Here is a list of helpful links:

Take a look at Grafana dashboard;
Live-tailing logs are also available;

Have a great time!

cheb0 · 2026-05-12T07:45:44Z

@seqbenchbot down d9e8e68e

seqbenchbot · 2026-05-12T07:45:47Z

Nice, @cheb0 <(-^,^-)=b!

The benchmark with identificator d9e8e68e was finished.
I've prepared a summary for you. Click on Show summary button to see it:

Show summary

Query	Type	`mean (ms)`			`stddev (ms)`			`p(50) (ms)`			`p(95) (ms)`			`p(99) (ms)`			`iterations`
Query	Type	base	comp	diff	base	comp	diff	base	comp	diff	base	comp	diff	base	comp	diff	base	comp	diff
`bulk`	warm	`59.79`	`60.23`	`+0.74%`	`18.52`	`18.77`	`+1.32%`	`55.00`	`56.00`	`+1.82%`	`96.00`	`97.00`	`+1.04%`	`129.00`	`129.00`	`0.00%`	`10366.00`	`10382.00`	`+0.15%`

Have a great time!

github-actions · 2026-05-12T12:06:56Z

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table

Name	Previous	Current	Ratio	Verdict
`Block_Pack-4`	`------`	`ddab56`
	`NaN B/op`	`113400.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`19.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`92337.00 ns/op`	`NaN`	`🔴`
`Block_Unpack-4`	`------`	`ddab56`
	`NaN B/op`	`262197.00 B/op`	`NaN`	`🔴`
	`NaN allocs/op`	`2.00 allocs/op`	`NaN`	`🔴`
	`NaN ns/op`	`96681.00 ns/op`	`NaN`	`🔴`

bitpack encoding for LID and MID blocks

d73e08a

ozontech deleted a comment from seqbenchbot Feb 6, 2026

ozontech deleted a comment from github-actions Bot Feb 6, 2026

ozontech deleted a comment from seqbenchbot Feb 6, 2026

dkharms added the performance Features or improvements that positively affect seq-db performance label Feb 10, 2026

ozontech deleted a comment from seqbenchbot Feb 13, 2026

cheb0 changed the title ~~perf: bitpack encoding for LID and MID blocks (draft, work in progress)~~ perf: implement bitpack encoding for LID and MID blocks (draft, work in progress) Mar 10, 2026

cheb0 added 3 commits March 10, 2026 14:57

unify bitpack files

fcc9d26

Merge branch 'main' into 312-bitpack-encoding

5e22f88

simplify intcomp usage, refine tests

53ef5eb

cheb0 force-pushed the 312-bitpack-encoding branch from 3accfd6 to 53ef5eb Compare March 11, 2026 10:27

cheb0 changed the title ~~perf: implement bitpack encoding for LID and MID blocks (draft, work in progress)~~ perf: implement bitpack encoding for LID and MID blocks Mar 11, 2026

ozontech deleted a comment from github-actions Bot Mar 12, 2026

ozontech deleted a comment from seqbenchbot Mar 12, 2026

ozontech deleted a comment from github-actions Bot Mar 12, 2026

small fixes

b04a41d

cheb0 force-pushed the 312-bitpack-encoding branch from bf8f3d8 to 985f11f Compare March 12, 2026 10:45

less copying for little endian hosts

22a3e36

dkharms reviewed Mar 30, 2026

View reviewed changes

Comment thread util/algorithms.go Outdated

Comment thread util/algorithms.go Outdated

Comment thread frac/sealed/lids/block.go Outdated

dkharms reviewed Mar 31, 2026

View reviewed changes

Comment thread util/algorithms.go Outdated

forshev approved these changes Mar 31, 2026

View reviewed changes

cheb0 added 2 commits April 10, 2026 12:02

PR review: delete premature optimizations

812dbac

Merge branch 'main' into 312-bitpack-encoding

e60aabc

cheb0 requested a review from dkharms April 10, 2026 09:08

eguguchkin modified the milestones: v0.72.0, v0.71.0 Apr 13, 2026

Merge branch 'main' into 312-bitpack-encoding

757588c

eguguchkin removed this from the v0.71.0 milestone Apr 20, 2026

remove IsLastLID from storage format

f3827f1

cheb0 force-pushed the 312-bitpack-encoding branch from 3a567c4 to f3827f1 Compare April 21, 2026 10:28

dkharms approved these changes Apr 30, 2026

View reviewed changes

cheb0 added 3 commits May 12, 2026 11:12

Merge branch 'main' into 312-bitpack-encoding

1ea101c

# Conflicts: # cmd/index_analyzer/main.go # config/frac_version.go # frac/fraction_test.go # frac/remote.go # frac/sealed.go # frac/sealed/sealing/index.go

fix test

0dd2560

linter

333e8b6

use V4 version in block decoding

d94c7a1

eguguchkin added this to the v0.72.0 milestone May 18, 2026

Conversation

cheb0 commented Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Measurements

Uh oh!

cheb0 commented Mar 12, 2026

Uh oh!

seqbenchbot commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Mar 12, 2026

🔴 Performance Degradation

Uh oh!

cheb0 commented Mar 12, 2026

Uh oh!

seqbenchbot commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Mar 12, 2026

🔴 Performance Degradation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Apr 10, 2026

🔴 Performance Degradation

Uh oh!

github-actions Bot commented Apr 15, 2026

🔴 Performance Degradation

Uh oh!

github-actions Bot commented Apr 21, 2026

🔴 Performance Degradation

Uh oh!

github-actions Bot commented Apr 21, 2026

🔴 Performance Degradation

Uh oh!

github-actions Bot commented May 12, 2026

🔴 Performance Degradation

Uh oh!

github-actions Bot commented May 12, 2026

🔴 Performance Degradation

Uh oh!

cheb0 commented May 12, 2026

Uh oh!

seqbenchbot commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cheb0 commented May 12, 2026

Uh oh!

seqbenchbot commented May 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented May 12, 2026

🔴 Performance Degradation

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

cheb0 commented Jan 26, 2026 •

edited

Loading

seqbenchbot commented Mar 12, 2026 •

edited

Loading

seqbenchbot commented Mar 12, 2026 •

edited

Loading

seqbenchbot commented May 12, 2026 •

edited

Loading

seqbenchbot commented May 12, 2026 •

edited

Loading