Skip to content

[refactor](scanner) Delete unused metrics and refactor scan operators #42857

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
Gabriel39 merged 1 commit into from
Oct 31, 2024
Merged

[refactor](scanner) Delete unused metrics and refactor scan operators #42857

Gabriel39 merged 1 commit into from
Oct 31, 2024

Conversation

@Gabriel39
Copy link
Contributor

@Gabriel39 Gabriel39 commented Oct 29, 2024

Proposed changes

Issue Number: close #xxx

@doris-robot
Copy link

doris-robot commented Oct 29, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions
Copy link
Contributor

github-actions bot commented Oct 29, 2024

clang-tidy review says "All clean, LGTM! 👍"

@Gabriel39
Copy link
Contributor Author

Gabriel39 commented Oct 29, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Oct 29, 2024

TPC-H: Total hot run time: 41318 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 92b9bc983cd2a8822b2472c4f4b4c6717364cf12, data reload: false

------ Round 1 ----------------------------------
q1	17837	7522	7276	7276
q2	2059	164	152	152
q3	10718	1095	1194	1095
q4	10570	851	779	779
q5	7746	3077	3078	3077
q6	233	148	145	145
q7	1037	628	603	603
q8	9368	1963	2048	1963
q9	6611	6469	6486	6469
q10	7075	2422	2421	2421
q11	441	241	250	241
q12	403	217	210	210
q13	17797	2990	2981	2981
q14	246	214	208	208
q15	569	511	510	510
q16	642	580	587	580
q17	1010	583	603	583
q18	7377	6685	6707	6685
q19	1335	1070	1010	1010
q20	468	179	189	179
q21	4071	3252	3161	3161
q22	1100	1010	990	990
Total cold run time: 108713 ms
Total hot run time: 41318 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7301	7243	7227	7227
q2	318	219	221	219
q3	3022	2953	2963	2953
q4	2140	1878	1794	1794
q5	5805	5822	5772	5772
q6	231	147	144	144
q7	2225	1814	1791	1791
q8	3369	3570	3440	3440
q9	8971	8938	8912	8912
q10	3575	3550	3584	3550
q11	594	477	477	477
q12	863	652	646	646
q13	9823	3158	3196	3158
q14	304	272	272	272
q15	566	523	510	510
q16	675	621	642	621
q17	1866	1638	1609	1609
q18	8267	7940	7633	7633
q19	1736	1504	1506	1504
q20	2129	1911	1836	1836
q21	5619	5466	5450	5450
q22	1137	1014	1043	1014
Total cold run time: 70536 ms
Total hot run time: 60532 ms

@doris-robot
Copy link

doris-robot commented Oct 29, 2024

TeamCity be ut coverage result:
Function Coverage: 37.49% (9721/25929)
Line Coverage: 28.74% (80637/280545)
Region Coverage: 28.16% (41678/148016)
Branch Coverage: 24.72% (21173/85668)
Coverage Report: http://coverage.selectdb-in.cc/coverage/92b9bc983cd2a8822b2472c4f4b4c6717364cf12_92b9bc983cd2a8822b2472c4f4b4c6717364cf12/report/index.html

@doris-robot
Copy link

doris-robot commented Oct 29, 2024

TPC-DS: Total hot run time: 196911 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 92b9bc983cd2a8822b2472c4f4b4c6717364cf12, data reload: false

query1	1258	1013	1026	1013
query2	6259	2080	2097	2080
query3	11444	4901	4920	4901
query4	33551	23623	23505	23505
query5	4831	461	420	420
query6	267	179	175	175
query7	3976	296	300	296
query8	295	235	239	235
query9	9600	2642	2647	2642
query10	490	253	244	244
query11	18166	15440	15324	15324
query12	154	105	101	101
query13	1571	424	403	403
query14	9078	6893	7410	6893
query15	240	187	186	186
query16	8007	493	505	493
query17	1434	584	576	576
query18	2075	331	307	307
query19	358	156	162	156
query20	124	119	123	119
query21	214	103	109	103
query22	4930	4721	4281	4281
query23	34843	34046	34088	34046
query24	10829	2854	2772	2772
query25	594	411	414	411
query26	725	156	158	156
query27	1993	285	287	285
query28	6636	2428	2425	2425
query29	728	418	427	418
query30	259	161	162	161
query31	1023	813	786	786
query32	83	55	56	55
query33	691	278	270	270
query34	897	502	529	502
query35	1065	879	883	879
query36	1124	960	955	955
query37	119	74	71	71
query38	4438	4296	4342	4296
query39	1452	1428	1440	1428
query40	198	100	99	99
query41	47	45	47	45
query42	113	98	94	94
query43	535	481	514	481
query44	1232	808	817	808
query45	182	166	165	165
query46	1108	701	703	701
query47	1978	1876	1854	1854
query48	453	340	331	331
query49	914	409	412	409
query50	807	383	384	383
query51	7146	7044	6989	6989
query52	98	117	84	84
query53	243	173	171	171
query54	1136	387	384	384
query55	77	75	73	73
query56	243	252	230	230
query57	1313	1198	1170	1170
query58	218	201	199	199
query59	3253	3120	2930	2930
query60	272	247	240	240
query61	98	98	97	97
query62	833	671	675	671
query63	215	184	182	182
query64	3759	629	610	610
query65	3279	3184	3216	3184
query66	804	300	307	300
query67	15925	15564	15821	15564
query68	4358	544	571	544
query69	426	247	256	247
query70	1192	1175	1182	1175
query71	323	263	261	261
query72	6255	3967	3966	3966
query73	755	358	353	353
query74	10196	8987	9062	8987
query75	3406	2672	2675	2672
query76	2711	956	972	956
query77	378	258	274	258
query78	10637	9603	9566	9566
query79	1235	599	586	586
query80	801	418	410	410
query81	553	249	241	241
query82	1337	108	116	108
query83	212	142	132	132
query84	229	72	64	64
query85	991	290	278	278
query86	309	303	301	301
query87	4854	4758	4671	4671
query88	3260	2193	2241	2193
query89	401	285	293	285
query90	1931	179	182	179
query91	139	105	102	102
query92	55	50	48	48
query93	1149	526	533	526
query94	744	296	287	287
query95	345	246	238	238
query96	617	286	279	279
query97	2932	2707	2695	2695
query98	211	198	191	191
query99	1873	1295	1305	1295
Total cold run time: 297115 ms
Total hot run time: 196911 ms

@doris-robot
Copy link

doris-robot commented Oct 29, 2024

ClickBench: Total hot run time: 31.56 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 92b9bc983cd2a8822b2472c4f4b4c6717364cf12, data reload: false

query1	0.03	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.64	0.10	0.09
query5	0.42	0.40	0.38
query6	1.15	0.66	0.66
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.57	0.51	0.50
query10	0.55	0.56	0.55
query11	0.14	0.10	0.11
query12	0.14	0.11	0.10
query13	0.60	0.59	0.59
query14	2.73	2.86	2.73
query15	0.90	0.84	0.83
query16	0.38	0.37	0.38
query17	1.04	1.01	1.00
query18	0.20	0.19	0.21
query19	1.79	1.75	1.78
query20	0.02	0.01	0.01
query21	15.42	0.60	0.59
query22	2.64	1.61	2.00
query23	17.12	0.86	0.75
query24	2.91	0.45	1.48
query25	0.24	0.10	0.13
query26	0.49	0.15	0.14
query27	0.04	0.05	0.04
query28	11.10	1.10	1.06
query29	12.56	3.33	3.29
query30	0.24	0.07	0.06
query31	2.89	0.38	0.38
query32	3.26	0.46	0.45
query33	2.95	2.97	3.07
query34	17.02	4.50	4.41
query35	4.45	4.50	4.48
query36	0.65	0.48	0.47
query37	0.08	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.16	0.14	0.12
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.1 s
Total hot run time: 31.56 s

yiguolei
yiguolei reviewed Oct 30, 2024
View reviewed changes
yiguolei
yiguolei reviewed Oct 30, 2024
View reviewed changes
yiguolei
yiguolei reviewed Oct 30, 2024
View reviewed changes
yiguolei
yiguolei reviewed Oct 30, 2024
View reviewed changes
yiguolei
yiguolei reviewed Oct 30, 2024
View reviewed changes
yiguolei
yiguolei reviewed Oct 30, 2024
View reviewed changes
@Gabriel39
Copy link
Contributor Author

Gabriel39 commented Oct 30, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Oct 30, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Oct 30, 2024

TPC-H: Total hot run time: 41587 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 35bf6561e83151cdb1aef9383ee6608718908956, data reload: false

------ Round 1 ----------------------------------
q1	17797	7526	7357	7357
q2	2067	169	162	162
q3	10709	1088	1180	1088
q4	10391	819	892	819
q5	7744	3070	3115	3070
q6	235	144	143	143
q7	1014	610	596	596
q8	9344	1999	1981	1981
q9	6642	6484	6503	6484
q10	7116	2409	2403	2403
q11	461	256	243	243
q12	411	222	211	211
q13	17791	3033	2996	2996
q14	256	213	213	213
q15	575	511	524	511
q16	671	589	589	589
q17	980	565	588	565
q18	7547	6801	6719	6719
q19	1415	1032	1086	1032
q20	484	177	182	177
q21	4061	3205	3257	3205
q22	1138	1028	1023	1023
Total cold run time: 108849 ms
Total hot run time: 41587 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7376	7343	7312	7312
q2	313	223	220	220
q3	3090	3012	2972	2972
q4	2223	1791	1782	1782
q5	5750	5781	5780	5780
q6	222	143	141	141
q7	2223	1795	1829	1795
q8	3414	3424	3592	3424
q9	8972	8979	8960	8960
q10	3676	3606	3594	3594
q11	596	506	483	483
q12	818	615	649	615
q13	10230	3191	3204	3191
q14	307	280	279	279
q15	576	533	532	532
q16	704	648	651	648
q17	1842	1646	1637	1637
q18	8348	7943	7657	7657
q19	1727	1569	1585	1569
q20	2124	1901	1873	1873
q21	5615	5481	5552	5481
q22	1180	1072	1046	1046
Total cold run time: 71326 ms
Total hot run time: 60991 ms

@doris-robot
Copy link

doris-robot commented Oct 30, 2024

TeamCity be ut coverage result:
Function Coverage: 37.96% (9847/25940)
Line Coverage: 29.22% (81969/280563)
Region Coverage: 28.57% (42293/148029)
Branch Coverage: 25.09% (21491/85662)
Coverage Report: http://coverage.selectdb-in.cc/coverage/35bf6561e83151cdb1aef9383ee6608718908956_35bf6561e83151cdb1aef9383ee6608718908956/report/index.html

@doris-robot
Copy link

doris-robot commented Oct 30, 2024

TPC-DS: Total hot run time: 197732 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 35bf6561e83151cdb1aef9383ee6608718908956, data reload: false

query1	1222	979	972	972
query2	6215	2110	2057	2057
query3	11578	5012	5193	5012
query4	33721	23805	23704	23704
query5	4814	450	442	442
query6	265	171	195	171
query7	4005	296	299	296
query8	295	222	212	212
query9	9569	2652	2640	2640
query10	497	243	264	243
query11	18341	15327	15451	15327
query12	152	108	102	102
query13	1564	435	409	409
query14	9505	7538	7084	7084
query15	260	191	185	185
query16	8118	503	520	503
query17	1599	609	578	578
query18	2132	305	313	305
query19	364	169	163	163
query20	120	112	114	112
query21	209	111	118	111
query22	4953	4678	4388	4388
query23	34729	34033	34101	34033
query24	10983	2829	2778	2778
query25	584	400	395	395
query26	1053	158	164	158
query27	2548	283	294	283
query28	7918	2435	2451	2435
query29	732	430	416	416
query30	260	160	158	158
query31	1000	801	812	801
query32	88	54	58	54
query33	758	270	265	265
query34	948	504	518	504
query35	1063	896	870	870
query36	1088	959	968	959
query37	123	77	75	75
query38	4396	4166	4328	4166
query39	1462	1455	1439	1439
query40	202	100	97	97
query41	47	45	48	45
query42	106	104	99	99
query43	522	508	482	482
query44	1302	818	815	815
query45	186	168	168	168
query46	1152	693	715	693
query47	1941	1842	1872	1842
query48	416	314	322	314
query49	899	409	408	408
query50	812	397	391	391
query51	7197	7033	6969	6969
query52	100	87	88	87
query53	255	179	180	179
query54	1207	409	418	409
query55	75	75	77	75
query56	264	249	252	249
query57	1275	1204	1199	1199
query58	225	217	212	212
query59	3307	3061	3265	3061
query60	270	248	240	240
query61	99	101	103	101
query62	851	689	687	687
query63	222	189	178	178
query64	3983	633	612	612
query65	3267	3204	3211	3204
query66	793	318	307	307
query67	16248	15685	15807	15685
query68	4518	566	542	542
query69	432	260	257	257
query70	1212	1147	1123	1123
query71	369	252	254	252
query72	6239	3997	3980	3980
query73	754	364	355	355
query74	10124	8967	9036	8967
query75	3452	2694	2650	2650
query76	2743	1173	978	978
query77	376	278	269	269
query78	10555	9609	9623	9609
query79	1148	600	589	589
query80	778	415	439	415
query81	544	255	245	245
query82	1274	117	119	117
query83	214	134	157	134
query84	217	72	75	72
query85	1095	288	285	285
query86	322	305	300	300
query87	4833	4724	4614	4614
query88	3353	2243	2177	2177
query89	418	297	287	287
query90	1912	191	188	188
query91	133	101	99	99
query92	60	48	50	48
query93	1139	535	539	535
query94	763	291	292	291
query95	349	252	251	251
query96	616	288	290	288
query97	2850	2746	2690	2690
query98	211	199	208	199
query99	1560	1339	1313	1313
Total cold run time: 301017 ms
Total hot run time: 197732 ms

@doris-robot
Copy link

doris-robot commented Oct 30, 2024

ClickBench: Total hot run time: 32.47 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 35bf6561e83151cdb1aef9383ee6608718908956, data reload: false

query1	0.04	0.04	0.04
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.64	0.10	0.10
query5	0.40	0.39	0.40
query6	1.15	0.65	0.66
query7	0.02	0.01	0.02
query8	0.04	0.04	0.03
query9	0.54	0.49	0.49
query10	0.54	0.55	0.55
query11	0.15	0.11	0.11
query12	0.15	0.11	0.12
query13	0.62	0.60	0.59
query14	2.83	2.71	2.77
query15	0.88	0.82	0.82
query16	0.39	0.38	0.38
query17	1.06	0.98	0.99
query18	0.20	0.20	0.21
query19	1.96	1.76	1.77
query20	0.01	0.01	0.01
query21	15.38	0.58	0.59
query22	2.29	2.58	1.44
query23	16.88	1.25	0.94
query24	2.84	1.22	1.96
query25	0.33	0.08	0.05
query26	0.53	0.14	0.14
query27	0.04	0.04	0.04
query28	10.05	1.13	1.07
query29	12.63	3.31	3.30
query30	0.24	0.06	0.05
query31	2.88	0.38	0.39
query32	3.25	0.47	0.46
query33	3.00	3.07	3.03
query34	16.89	4.54	4.48
query35	4.49	4.55	4.52
query36	0.68	0.47	0.48
query37	0.08	0.05	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.15	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.02	0.03
Total cold run time: 105.76 s
Total hot run time: 32.47 s

@github-actions
Copy link
Contributor

github-actions bot commented Oct 30, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Oct 30, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Oct 30, 2024

PR approved by anyone and no changes requested.

@Gabriel39 Gabriel39 merged commit 298ddb5 into apache:master Oct 31, 2024
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Nov 1, 2024
@Gabriel39
[refactor](scanner) Delete unused metrics and refactor scan operators (…
da3dfa5 
…apache#42857)
Gabriel39 added a commit that referenced this pull request Nov 1, 2024
@Gabriel39
[branch-3.0](pick) pick #42857 #42898 #42909 #42992 #43002 (#43039)
2abe20b 
### What problem does this PR solve?
<!--
You need to clearly describe your PR in this part:

1. What problem was fixed (it's best to include specific error reporting
information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what
is it now, why was it modified, and what possible impacts might there
be.
3. What features were added. Why this function was added.
4. Which codes were refactored and why this part of the code was
refactored.
5. Which functions were optimized and what is the difference before and
after the optimization.

The description of the PR needs to enable reviewers to quickly and
clearly understand the logic of the code modification.
-->

<!--
If there are related issues, please fill in the issue number.
- If you want the issue to be closed after the PR is merged, please use
"close #12345". Otherwise, use "ref #12345"
-->
Issue Number: close #xxx

<!--
If this PR is followup a preivous PR, for example, fix the bug that
introduced by a related PR,
link the PR here
-->
Related PR: #xxx

Problem Summary:

### Check List (For Committer)

- Test <!-- At least one of them must be included. -->

    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No colde files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:

    - [x] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?

    - [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

- Release note

    <!-- bugfix, feat, behavior changed need a release note -->
    <!-- Add one line release note for this PR. -->
    None

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguolei yiguolei yiguolei approved these changes

+1 more reviewer

@Mryange Mryange Mryange approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.3-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@Gabriel39 @doris-robot @yiguolei @Mryange