Skip to content

[fix](inverted index) fix BM25 LENGTH_TABLE using byte4_to_int for correct norm decoding #59713

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
zzzxl1993 merged 1 commit into from
Jan 13, 2026
Merged

[fix](inverted index) fix BM25 LENGTH_TABLE using byte4_to_int for correct norm decoding #59713

zzzxl1993 merged 1 commit into from
Jan 13, 2026

Conversation

@zzzxl1993
Copy link
Contributor

@zzzxl1993 zzzxl1993 commented Jan 9, 2026 •
edited
Loading

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 9, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zzzxl1993
Copy link
Contributor Author

zzzxl1993 commented Jan 10, 2026

run buildall

@doris-robot
Copy link

doris-robot commented Jan 10, 2026

TPC-H: Total hot run time: 31406 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0eb7656a2c589ceb3665148e07cb7bee57a425e4, data reload: false

------ Round 1 ----------------------------------
q1	17608	4254	4070	4070
q2	2022	351	268	268
q3	10146	1274	719	719
q4	10218	816	313	313
q5	7550	2056	1836	1836
q6	190	169	141	141
q7	901	803	653	653
q8	9269	1372	1155	1155
q9	4917	4505	4586	4505
q10	6785	1783	1405	1405
q11	518	293	280	280
q12	672	745	575	575
q13	17767	3815	3077	3077
q14	297	296	284	284
q15	578	515	509	509
q16	681	665	633	633
q17	653	767	540	540
q18	6746	6378	6301	6301
q19	1109	963	596	596
q20	403	378	257	257
q21	3028	2518	2315	2315
q22	1045	1006	974	974
Total cold run time: 103103 ms
Total hot run time: 31406 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4090	4014	4028	4014
q2	328	397	325	325
q3	2097	2617	2198	2198
q4	1320	1772	1308	1308
q5	4065	3918	4085	3918
q6	208	173	136	136
q7	1863	1847	1626	1626
q8	2822	2426	2398	2398
q9	7378	7089	7190	7089
q10	2471	2673	2353	2353
q11	565	471	448	448
q12	756	792	637	637
q13	3693	4119	3408	3408
q14	378	316	292	292
q15	556	511	488	488
q16	668	697	662	662
q17	1175	1318	1352	1318
q18	7910	7658	7888	7658
q19	883	884	873	873
q20	2021	2152	1896	1896
q21	4756	4674	4325	4325
q22	1120	1085	1062	1062
Total cold run time: 51123 ms
Total hot run time: 48432 ms

@doris-robot
Copy link

doris-robot commented Jan 10, 2026

TPC-DS: Total hot run time: 173301 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0eb7656a2c589ceb3665148e07cb7bee57a425e4, data reload: false

query5	4390	587	431	431
query6	333	240	223	223
query7	4221	474	270	270
query8	331	255	240	240
query9	8739	2662	2653	2653
query10	496	373	339	339
query11	15352	15185	14931	14931
query12	177	117	112	112
query13	1258	491	386	386
query14	6343	3015	2744	2744
query14_1	2735	2683	2689	2683
query15	200	196	173	173
query16	990	474	436	436
query17	1115	685	579	579
query18	2515	446	348	348
query19	228	232	200	200
query20	129	118	117	117
query21	212	143	126	126
query22	3897	3928	4038	3928
query23	16003	15814	15215	15215
query23_1	15326	15343	15275	15275
query24	7501	1574	1203	1203
query24_1	1212	1190	1224	1190
query25	590	442	390	390
query26	1237	270	149	149
query27	2771	468	281	281
query28	4514	2137	2128	2128
query29	823	546	429	429
query30	312	230	211	211
query31	798	643	551	551
query32	75	66	68	66
query33	528	344	283	283
query34	898	891	523	523
query35	745	775	668	668
query36	832	842	854	842
query37	134	96	81	81
query38	2719	2719	2710	2710
query39	762	732	749	732
query39_1	710	724	723	723
query40	220	133	121	121
query41	105	74	64	64
query42	104	108	102	102
query43	488	466	464	464
query44	1361	723	719	719
query45	188	184	176	176
query46	883	980	597	597
query47	1377	1418	1390	1390
query48	310	329	247	247
query49	610	408	329	329
query50	665	276	211	211
query51	3768	3832	3867	3832
query52	103	106	96	96
query53	293	338	279	279
query54	298	259	250	250
query55	76	78	71	71
query56	283	301	285	285
query57	1019	1032	955	955
query58	264	252	248	248
query59	2003	2169	2161	2161
query60	326	312	294	294
query61	161	148	158	148
query62	398	349	309	309
query63	302	266	276	266
query64	5071	1302	997	997
query65	3845	3721	3725	3721
query66	1410	422	295	295
query67	14871	14959	14860	14860
query68	3398	1039	715	715
query69	460	357	321	321
query70	1055	910	936	910
query71	349	304	292	292
query72	6316	3691	3477	3477
query73	768	735	302	302
query74	8834	8811	8595	8595
query75	2841	2811	2468	2468
query76	3373	1077	661	661
query77	540	385	277	277
query78	9798	9783	9202	9202
query79	1496	926	596	596
query80	667	596	480	480
query81	518	262	242	242
query82	255	142	117	117
query83	263	266	242	242
query84	257	125	105	105
query85	905	547	464	464
query86	380	322	327	322
query87	2885	2853	2720	2720
query88	3270	2242	2220	2220
query89	393	366	336	336
query90	2100	161	153	153
query91	196	164	146	146
query92	73	70	61	61
query93	1355	947	527	527
query94	567	306	300	300
query95	575	326	296	296
query96	596	471	210	210
query97	2360	2371	2355	2355
query98	243	208	199	199
query99	593	570	508	508
Total cold run time: 249931 ms
Total hot run time: 173301 ms

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 10, 2026

BE UT Coverage Report

Increment line coverage 3.45% (1/29) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.02% (18974/35789)
Line Coverage 39.08% (175799/449882)
Region Coverage 33.66% (136184/404555)
Branch Coverage 34.67% (58843/169702)

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 10, 2026

BE Regression && UT Coverage Report

Increment line coverage 6.90% (2/29) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.85% (25835/34984)
Line Coverage 61.28% (274933/448649)
Region Coverage 56.12% (229360/408671)
Branch Coverage 58.06% (98846/170251)

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 10, 2026

BE Regression && UT Coverage Report

Increment line coverage 6.90% (2/29) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 73.85% (25835/34984)
Line Coverage 61.28% (274910/448649)
Region Coverage 56.12% (229347/408671)
Branch Coverage 58.05% (98834/170251)

csun5285
csun5285 approved these changes Jan 13, 2026
View reviewed changes
Copy link
Contributor

@csun5285 csun5285 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Jan 13, 2026

PR approved by anyone and no changes requested.

airborne12
airborne12 approved these changes Jan 13, 2026
View reviewed changes
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Jan 13, 2026

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 13, 2026
@zzzxl1993 zzzxl1993 merged commit 0e80bf4 into apache:master Jan 13, 2026
30 of 32 checks passed
github-actions bot pushed a commit that referenced this pull request Jan 13, 2026
@zzzxl1993
[fix](inverted index) fix BM25 LENGTH_TABLE using byte4_to_int for co…
76bbf75 
…rrect norm decoding (#59713)
@github-actions github-actions bot mentioned this pull request Jan 13, 2026
Merged
yiguolei pushed a commit that referenced this pull request Jan 13, 2026
@github-actions @zzzxl1993
branch-4.0: [fix](inverted index) fix BM25 LENGTH_TABLE using byte4_t…
2852d84 
…o_int for correct norm decoding #59713 (#59812)

Cherry-picked from #59713

Co-authored-by: zzzxl <yangsiyu@selectdb.com>
zzzxl1993 added a commit to zzzxl1993/doris that referenced this pull request Jan 13, 2026
@zzzxl1993
[fix](inverted index) fix BM25 LENGTH_TABLE using byte4_to_int for co…
868e3ca 
…rrect norm decoding (apache#59713)
@yiguolei yiguolei mentioned this pull request Jan 23, 2026
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@airborne12 airborne12 airborne12 approved these changes

+1 more reviewer

@csun5285 csun5285 csun5285 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/4.0.3-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@zzzxl1993 @hello-stephen @doris-robot @airborne12 @csun5285 @yiguolei