Skip to content

Conversation

@zhiqiang-hhhh
Copy link
Contributor

@zhiqiang-hhhh zhiqiang-hhhh commented Dec 26, 2024

What problem does this PR solve?

Related PR: #39927

#39927 introduced a new session var adaptive_pipeline_task_serial_read_on_limit, its implementation has flaw, reduce scanner parallelism should only happen when there is not conjunction. Since we can not estimate the filtration efficiency.

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zhiqiang-hhhh
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32539 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cf36520de967135eca4e4278833d5d0679b92226, data reload: false

------ Round 1 ----------------------------------
q1	17629	7059	5995	5995
q2	2048	297	172	172
q3	10414	1251	727	727
q4	10212	868	457	457
q5	7525	2160	1915	1915
q6	207	181	151	151
q7	868	764	613	613
q8	9228	1379	1199	1199
q9	5160	4932	4876	4876
q10	6756	2309	1874	1874
q11	467	287	262	262
q12	351	355	222	222
q13	17764	3511	2964	2964
q14	232	227	224	224
q15	561	502	497	497
q16	634	611	575	575
q17	556	843	319	319
q18	7280	6640	6436	6436
q19	1230	958	555	555
q20	314	321	200	200
q21	2766	2185	1979	1979
q22	357	331	327	327
Total cold run time: 102559 ms
Total hot run time: 32539 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6217	6257	6223	6223
q2	234	323	230	230
q3	2257	2714	2330	2330
q4	1460	1866	1402	1402
q5	4351	4771	4805	4771
q6	185	179	149	149
q7	2119	1997	1840	1840
q8	2625	2774	2697	2697
q9	7253	7166	7248	7166
q10	3082	3329	2817	2817
q11	584	524	515	515
q12	713	766	612	612
q13	3360	3763	3144	3144
q14	285	316	293	293
q15	586	512	509	509
q16	643	704	647	647
q17	1202	1733	1272	1272
q18	7788	7492	7077	7077
q19	805	1010	1083	1010
q20	1951	1949	1863	1863
q21	5607	5177	4857	4857
q22	617	634	588	588
Total cold run time: 53924 ms
Total hot run time: 52012 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 189898 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cf36520de967135eca4e4278833d5d0679b92226, data reload: false

query1	973	372	367	367
query2	6520	2252	2353	2252
query3	6712	214	207	207
query4	33921	23893	23515	23515
query5	4398	641	477	477
query6	274	184	173	173
query7	4649	493	296	296
query8	296	234	245	234
query9	9434	2738	2749	2738
query10	483	319	261	261
query11	18291	15542	15126	15126
query12	163	109	107	107
query13	1663	536	443	443
query14	10309	7036	7073	7036
query15	260	190	185	185
query16	8217	590	430	430
query17	1583	748	597	597
query18	2132	403	302	302
query19	210	188	157	157
query20	121	109	108	108
query21	213	128	108	108
query22	4239	4466	4313	4313
query23	34630	33045	33422	33045
query24	6605	2337	2231	2231
query25	485	444	381	381
query26	1205	267	149	149
query27	2325	470	319	319
query28	5561	2456	2449	2449
query29	751	561	407	407
query30	224	181	155	155
query31	1004	899	794	794
query32	101	59	55	55
query33	506	369	284	284
query34	749	861	514	514
query35	852	785	748	748
query36	1015	1032	959	959
query37	114	97	72	72
query38	4372	4176	4176	4176
query39	1493	1433	1447	1433
query40	208	111	104	104
query41	47	43	46	43
query42	116	103	105	103
query43	506	516	488	488
query44	1342	814	798	798
query45	178	174	175	174
query46	848	1029	628	628
query47	1925	1951	1899	1899
query48	392	408	339	339
query49	769	464	389	389
query50	615	632	394	394
query51	7140	7315	6966	6966
query52	102	103	94	94
query53	220	251	184	184
query54	483	480	400	400
query55	86	74	76	74
query56	260	261	260	260
query57	1218	1189	1130	1130
query58	228	234	240	234
query59	3081	3216	3114	3114
query60	281	258	247	247
query61	140	109	124	109
query62	870	800	725	725
query63	226	188	189	188
query64	4613	1000	641	641
query65	3298	3207	3192	3192
query66	863	409	304	304
query67	16020	15770	15427	15427
query68	9737	747	571	571
query69	449	291	254	254
query70	1270	1155	1158	1155
query71	432	284	268	268
query72	5806	3774	3878	3774
query73	661	740	360	360
query74	9931	9439	8955	8955
query75	4537	3153	2636	2636
query76	5500	1192	761	761
query77	1019	364	278	278
query78	10096	10218	9397	9397
query79	2478	883	616	616
query80	678	510	439	439
query81	467	280	236	236
query82	365	154	126	126
query83	207	170	146	146
query84	286	89	75	75
query85	758	359	308	308
query86	365	304	323	304
query87	4645	4621	4363	4363
query88	3223	2257	2220	2220
query89	406	344	307	307
query90	1996	190	186	186
query91	133	131	106	106
query92	67	54	61	54
query93	938	891	543	543
query94	646	395	320	320
query95	330	257	256	256
query96	475	602	280	280
query97	2763	2826	2691	2691
query98	220	204	203	203
query99	1724	1558	1459	1459
Total cold run time: 295736 ms
Total hot run time: 189898 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.26 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cf36520de967135eca4e4278833d5d0679b92226, data reload: false

query1	0.04	0.04	0.05
query2	0.07	0.04	0.04
query3	0.23	0.07	0.08
query4	1.60	0.10	0.10
query5	0.40	0.41	0.41
query6	1.14	0.65	0.65
query7	0.02	0.02	0.01
query8	0.04	0.04	0.03
query9	0.57	0.51	0.49
query10	0.54	0.59	0.53
query11	0.14	0.10	0.11
query12	0.14	0.10	0.11
query13	0.61	0.62	0.59
query14	2.72	2.72	2.83
query15	0.92	0.83	0.84
query16	0.38	0.38	0.38
query17	1.03	1.08	1.05
query18	0.23	0.21	0.21
query19	1.96	1.76	1.96
query20	0.02	0.01	0.02
query21	15.38	0.95	0.59
query22	0.76	0.71	0.62
query23	15.38	1.49	0.65
query24	3.02	0.71	1.46
query25	0.12	0.18	0.18
query26	0.28	0.15	0.14
query27	0.06	0.06	0.05
query28	13.87	1.54	1.06
query29	12.59	3.97	3.27
query30	0.25	0.09	0.07
query31	2.83	0.58	0.39
query32	3.22	0.55	0.46
query33	3.09	3.05	3.11
query34	16.61	5.17	4.54
query35	4.53	4.56	4.54
query36	0.66	0.50	0.50
query37	0.09	0.06	0.06
query38	0.04	0.04	0.04
query39	0.04	0.02	0.02
query40	0.18	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 105.96 s
Total hot run time: 31.26 s

@zhiqiang-hhhh
Copy link
Contributor Author

run beut

@zhiqiang-hhhh
Copy link
Contributor Author

run external

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.83% (10104/26020)
Line Coverage: 29.85% (85375/286050)
Region Coverage: 28.98% (43626/150520)
Branch Coverage: 25.52% (22246/87184)
Coverage Report: http://coverage.selectdb-in.cc/coverage/cf36520de967135eca4e4278833d5d0679b92226_cf36520de967135eca4e4278833d5d0679b92226/report/index.html

@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Dec 27, 2024
@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@yiguolei yiguolei added the usercase Important user case type label label Dec 27, 2024
@yiguolei yiguolei merged commit 4389600 into apache:master Dec 27, 2024
28 of 30 checks passed
@zhiqiang-hhhh zhiqiang-hhhh deleted the fix-scan-limit branch December 27, 2024 05:03
zhiqiang-hhhh added a commit to zhiqiang-hhhh/doris that referenced this pull request Dec 27, 2024
Related PR: apache#39927

apache#39927 introduced a new session var
`adaptive_pipeline_task_serial_read_on_limit`, its implementation has
flaw, reduce scanner parallelism should only happen when there is not
conjunction. Since we can not estimate the filtration efficiency.
zhiqiang-hhhh added a commit to zhiqiang-hhhh/doris that referenced this pull request Dec 27, 2024
Related PR: apache#39927

apache#39927 introduced a new session var
`adaptive_pipeline_task_serial_read_on_limit`, its implementation has
flaw, reduce scanner parallelism should only happen when there is not
conjunction. Since we can not estimate the filtration efficiency.
yiguolei pushed a commit that referenced this pull request Dec 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.8-merged dev/3.0.4-merged reviewed usercase Important user case type label

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants