Skip to content

[Fix](orc-reader) Fix merge range not sorted in new merge io facility of orc-reader. #51102

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
May 24, 2025
Merged

[Fix](orc-reader) Fix merge range not sorted in new merge io facility of orc-reader. #51102

morningman merged 1 commit into from
May 24, 2025

Conversation

@kaka11chen
Copy link
Contributor

@kaka11chen kaka11chen commented May 21, 2025 •
edited by morningman
Loading

What problem does this PR solve?

Followup: #45966

Problem Summary:

Release note

Fix merge range not sorted in new merge io facility of orc-reader.
Because the ranges taken from std::unordered_map<orc::StreamId, io::PrefetchRange>&ranges are not sorted, merging adjacent ranges will have a very poor effect.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented May 21, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@kaka11chen
Copy link
Contributor Author

kaka11chen commented May 21, 2025

run buildall

@doris-robot
Copy link

doris-robot commented May 21, 2025

TPC-H: Total hot run time: 34225 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 70391ccec59e7ab02db0313728c2dbb05d58ad95, data reload: false

------ Round 1 ----------------------------------
q1	26668	5027	5050	5027
q2	2098	293	190	190
q3	10375	1293	719	719
q4	10238	1038	546	546
q5	7539	2395	2356	2356
q6	186	166	132	132
q7	930	740	631	631
q8	9336	1299	1151	1151
q9	6925	5176	5093	5093
q10	6876	2306	1913	1913
q11	503	307	287	287
q12	358	361	220	220
q13	17776	3667	3124	3124
q14	239	230	216	216
q15	544	491	482	482
q16	436	430	382	382
q17	610	883	369	369
q18	7454	7114	7063	7063
q19	1389	979	617	617
q20	341	327	227	227
q21	4078	3546	2483	2483
q22	1055	1009	997	997
Total cold run time: 115954 ms
Total hot run time: 34225 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5229	5120	5089	5089
q2	245	322	237	237
q3	2160	2678	2288	2288
q4	1443	1815	1465	1465
q5	4496	4402	4388	4388
q6	213	171	130	130
q7	2070	1950	1781	1781
q8	2610	2593	2541	2541
q9	7248	7148	7029	7029
q10	3064	3165	2800	2800
q11	565	523	491	491
q12	696	765	621	621
q13	3545	3872	3367	3367
q14	283	297	268	268
q15	507	486	477	477
q16	442	490	457	457
q17	1181	1572	1398	1398
q18	7702	7675	7405	7405
q19	878	891	994	891
q20	1988	1982	1869	1869
q21	5026	4454	4553	4454
q22	1085	1056	1010	1010
Total cold run time: 52676 ms
Total hot run time: 50456 ms

@doris-robot
Copy link

doris-robot commented May 21, 2025

TPC-DS: Total hot run time: 192895 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 70391ccec59e7ab02db0313728c2dbb05d58ad95, data reload: false

query1	1442	1093	1072	1072
query2	6176	1871	1878	1871
query3	10944	4543	4374	4374
query4	53439	25362	23010	23010
query5	5117	495	467	467
query6	361	211	194	194
query7	5072	498	306	306
query8	335	262	233	233
query9	6411	2659	2653	2653
query10	455	322	280	280
query11	15007	15086	14901	14901
query12	161	119	103	103
query13	1141	522	435	435
query14	10166	6408	6247	6247
query15	194	204	194	194
query16	7098	677	529	529
query17	1114	749	590	590
query18	1560	428	325	325
query19	207	201	181	181
query20	133	134	132	132
query21	212	144	112	112
query22	4387	4409	4269	4269
query23	34336	33607	33744	33607
query24	6557	2412	2477	2412
query25	479	465	401	401
query26	725	275	154	154
query27	2244	535	375	375
query28	2984	2187	2165	2165
query29	563	576	429	429
query30	272	217	192	192
query31	894	886	773	773
query32	75	60	63	60
query33	437	383	311	311
query34	781	874	553	553
query35	801	843	748	748
query36	947	973	923	923
query37	110	102	78	78
query38	4323	4334	4314	4314
query39	1537	1453	1472	1453
query40	207	128	116	116
query41	80	53	55	53
query42	125	115	111	111
query43	523	508	471	471
query44	1350	836	851	836
query45	187	180	167	167
query46	860	1014	662	662
query47	1833	1877	1807	1807
query48	401	437	325	325
query49	673	553	467	467
query50	685	700	424	424
query51	4249	4217	4182	4182
query52	122	116	104	104
query53	235	264	187	187
query54	610	585	531	531
query55	91	93	89	89
query56	306	308	324	308
query57	1193	1232	1117	1117
query58	267	261	271	261
query59	2728	2863	2682	2682
query60	342	320	312	312
query61	135	142	134	134
query62	743	737	679	679
query63	236	201	194	194
query64	1958	1067	664	664
query65	4343	4283	4279	4279
query66	718	402	305	305
query67	16033	15695	15316	15316
query68	7100	884	526	526
query69	551	306	270	270
query70	1252	1053	1137	1053
query71	499	323	304	304
query72	6026	4795	4761	4761
query73	1286	642	357	357
query74	8966	9169	9021	9021
query75	3801	3206	2684	2684
query76	4298	1211	755	755
query77	613	386	298	298
query78	10056	10196	9356	9356
query79	2459	779	582	582
query80	660	561	467	467
query81	483	256	226	226
query82	432	124	96	96
query83	352	253	242	242
query84	293	102	83	83
query85	809	351	310	310
query86	370	327	281	281
query87	4425	4489	4449	4449
query88	3155	2315	2323	2315
query89	403	328	283	283
query90	1960	213	210	210
query91	141	146	116	116
query92	71	65	57	57
query93	1181	931	590	590
query94	681	401	292	292
query95	368	298	282	282
query96	511	583	285	285
query97	2727	2760	2653	2653
query98	249	230	200	200
query99	1427	1396	1282	1282
Total cold run time: 298035 ms
Total hot run time: 192895 ms

@doris-robot
Copy link

doris-robot commented May 21, 2025

ClickBench: Total hot run time: 29.65 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 70391ccec59e7ab02db0313728c2dbb05d58ad95, data reload: false

query1	0.04	0.04	0.03
query2	0.13	0.10	0.12
query3	0.26	0.19	0.19
query4	1.58	0.20	0.18
query5	0.46	0.43	0.44
query6	1.17	0.66	0.66
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.61	0.52	0.53
query10	0.58	0.57	0.57
query11	0.16	0.11	0.10
query12	0.15	0.12	0.12
query13	0.62	0.61	0.60
query14	0.80	0.80	0.81
query15	0.89	0.88	0.87
query16	0.37	0.37	0.38
query17	1.02	1.03	1.06
query18	0.23	0.21	0.21
query19	1.95	1.82	1.90
query20	0.02	0.01	0.01
query21	15.40	0.89	0.55
query22	0.75	1.27	0.62
query23	14.88	1.39	0.60
query24	6.72	1.58	1.26
query25	0.52	0.25	0.12
query26	0.60	0.16	0.15
query27	0.06	0.04	0.04
query28	10.17	0.94	0.48
query29	12.63	4.11	3.36
query30	0.25	0.09	0.07
query31	2.83	0.61	0.39
query32	3.23	0.55	0.49
query33	3.09	3.03	3.08
query34	15.67	5.12	4.50
query35	4.54	4.56	4.53
query36	0.66	0.52	0.49
query37	0.09	0.06	0.07
query38	0.06	0.04	0.04
query39	0.03	0.03	0.02
query40	0.17	0.15	0.13
query41	0.08	0.03	0.03
query42	0.03	0.03	0.02
query43	0.04	0.03	0.02
Total cold run time: 103.6 s
Total hot run time: 29.65 s

@hello-stephen
Copy link
Contributor

hello-stephen commented May 21, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/2) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 55.94% (14926/26680)
Line Coverage 44.76% (132412/295809)
Region Coverage 43.85% (66617/151904)
Branch Coverage 38.45% (34137/88786)

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 21, 2025
@github-actions
Copy link
Contributor

github-actions bot commented May 21, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented May 21, 2025

PR approved by anyone and no changes requested.

@hello-stephen
Copy link
Contributor

hello-stephen commented May 21, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (1/1) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.49% (20878/26265)
Line Coverage 72.63% (214840/295807)
Region Coverage 70.82% (126377/178460)
Branch Coverage 64.55% (65471/101422)

@hello-stephen
Copy link
Contributor

hello-stephen commented May 24, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (2/2) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 79.49% (20878/26265)
Line Coverage 72.63% (214840/295807)
Region Coverage 70.82% (126377/178460)
Branch Coverage 64.55% (65471/101422)

@morningman morningman merged commit ef003cd into apache:master May 24, 2025
27 of 29 checks passed
koarz pushed a commit to koarz/doris that referenced this pull request Jun 4, 2025
@kaka11chen @koarz
[Fix](orc-reader) Fix merge range not sorted in new merge io facility…
32c44ad 
… of orc-reader. (apache#51102)

### What problem does this PR solve?

Related PR: apache#45966

Fix merge range not sorted in new merge io facility of orc-reader.
Because the ranges taken from std::unordered_map<orc::StreamId, io::PrefetchRange>&ranges are not sorted, merging adjacent ranges will have a very poor effect.
kaka11chen added a commit to kaka11chen/doris that referenced this pull request Jun 24, 2025
@kaka11chen
[Fix](orc-reader) Fix merge range not sorted in new merge io facility…
d46c0fe 
… of orc-reader. (apache#51102)

### What problem does this PR solve?

Related PR: apache#45966

Fix merge range not sorted in new merge io facility of orc-reader.
Because the ranges taken from std::unordered_map<orc::StreamId, io::PrefetchRange>&ranges are not sorted, merging adjacent ranges will have a very poor effect.
@kaka11chen kaka11chen mentioned this pull request Jun 24, 2025
Merged
morrySnow pushed a commit that referenced this pull request Jun 25, 2025
@kaka11chen
branch-3.1: [Feature](orc-reader) Implement new merge io facility for…
2684cc0 
… orc reader. (#52085)

Cherry-pick main PR: #45966, Fix bugs PR: #50185 #51102
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morningman morningman morningman approved these changes

@CalvinKirs CalvinKirs CalvinKirs approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@kaka11chen @hello-stephen @doris-robot @morningman @CalvinKirs @morrySnow