Skip to content

[improvement](statistics)Use real base index id to fetch stats cache. #36914

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
Jibing-Li merged 1 commit into from
Jun 28, 2024
Merged

[improvement](statistics)Use real base index id to fetch stats cache. #36914

Jibing-Li merged 1 commit into from
Jun 28, 2024

Conversation

@Jibing-Li
Copy link
Contributor

@Jibing-Li Jibing-Li commented Jun 27, 2024 •
edited
Loading

For historical reason, statistics tables use -1 for OlapTable base index id. This brings many if/else branch for stats calculate. This pr is to screen the -1 for Nereids. The stats user could use the real base index id to fetch stats cache. Will do the id translation inside the get cache api.

@doris-robot
Copy link

doris-robot commented Jun 27, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Jibing-Li Jibing-Li marked this pull request as ready for review June 27, 2024 06:46
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 27, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jun 27, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jun 27, 2024

PR approved by anyone and no changes requested.

@Jibing-Li
Copy link
Contributor Author

Jibing-Li commented Jun 27, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Jun 27, 2024

TPC-H: Total hot run time: 39579 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f148fab59c9480e30e1ca7e37f0e5f54366baf27, data reload: false

------ Round 1 ----------------------------------
q1	17627	4398	4288	4288
q2	2023	192	201	192
q3	10425	1225	1043	1043
q4	10187	732	801	732
q5	7487	2646	2568	2568
q6	217	142	136	136
q7	948	596	626	596
q8	9228	2072	2070	2070
q9	8999	6526	6471	6471
q10	8974	3689	3698	3689
q11	473	232	235	232
q12	533	239	227	227
q13	17763	3030	2994	2994
q14	259	234	226	226
q15	517	467	502	467
q16	521	375	388	375
q17	962	680	728	680
q18	7919	7546	7280	7280
q19	5625	1503	1540	1503
q20	679	329	330	329
q21	4928	3143	3275	3143
q22	407	338	338	338
Total cold run time: 116701 ms
Total hot run time: 39579 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4373	4273	4255	4255
q2	372	275	291	275
q3	2978	2849	2917	2849
q4	1992	1812	1726	1726
q5	5718	5556	5547	5547
q6	224	129	135	129
q7	2232	1865	1901	1865
q8	3271	3474	3475	3474
q9	8751	8719	8919	8719
q10	4132	4015	3808	3808
q11	608	488	515	488
q12	848	675	651	651
q13	18308	3203	3262	3203
q14	319	304	300	300
q15	556	493	503	493
q16	521	476	444	444
q17	1830	1525	1546	1525
q18	9812	8452	8411	8411
q19	2744	1667	1621	1621
q20	2208	1898	1806	1806
q21	11399	5135	5058	5058
q22	657	609	550	550
Total cold run time: 83853 ms
Total hot run time: 57197 ms

@doris-robot
Copy link

doris-robot commented Jun 27, 2024

TPC-DS: Total hot run time: 175131 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f148fab59c9480e30e1ca7e37f0e5f54366baf27, data reload: false

query1	924	392	380	380
query2	6483	2483	2425	2425
query3	6641	214	219	214
query4	20867	17382	17316	17316
query5	3649	486	487	486
query6	262	175	186	175
query7	4587	295	292	292
query8	297	305	292	292
query9	8474	2373	2346	2346
query10	605	317	299	299
query11	10694	10056	10014	10014
query12	123	89	85	85
query13	1641	383	380	380
query14	9993	7923	7967	7923
query15	280	196	197	196
query16	8075	276	272	272
query17	1895	557	544	544
query18	2117	286	283	283
query19	308	157	161	157
query20	90	82	83	82
query21	210	138	128	128
query22	4270	4085	3980	3980
query23	33773	33689	33592	33592
query24	10682	3006	2861	2861
query25	616	425	411	411
query26	714	164	161	161
query27	2169	319	329	319
query28	6046	2121	2099	2099
query29	900	686	636	636
query30	267	164	161	161
query31	963	753	777	753
query32	89	57	62	57
query33	679	305	315	305
query34	870	502	500	500
query35	770	656	636	636
query36	1159	996	1020	996
query37	152	75	73	73
query38	2941	2844	2815	2815
query39	921	865	835	835
query40	220	130	138	130
query41	57	54	58	54
query42	111	107	112	107
query43	606	574	558	558
query44	1135	753	747	747
query45	199	171	174	171
query46	1083	734	732	732
query47	1855	1772	1781	1772
query48	383	312	329	312
query49	869	430	442	430
query50	782	393	399	393
query51	7026	6738	6809	6738
query52	111	91	100	91
query53	362	298	299	298
query54	867	448	442	442
query55	76	76	73	73
query56	286	255	274	255
query57	1132	1037	1063	1037
query58	249	249	245	245
query59	3356	2956	3313	2956
query60	312	299	278	278
query61	96	88	109	88
query62	599	470	449	449
query63	322	293	298	293
query64	8501	2254	1769	1769
query65	3197	3071	3121	3071
query66	745	334	321	321
query67	15666	14899	15176	14899
query68	8300	543	566	543
query69	673	468	421	421
query70	1204	1103	1162	1103
query71	519	281	281	281
query72	8743	5432	5543	5432
query73	1380	336	326	326
query74	5821	5497	5533	5497
query75	5120	2639	2661	2639
query76	4697	1013	903	903
query77	791	303	303	303
query78	10476	9974	9734	9734
query79	7481	515	508	508
query80	2053	486	479	479
query81	551	227	224	224
query82	262	107	109	107
query83	294	170	169	169
query84	314	88	86	86
query85	861	279	262	262
query86	344	302	304	302
query87	3335	3140	3075	3075
query88	4472	2447	2452	2447
query89	521	379	381	379
query90	1927	189	190	189
query91	129	97	99	97
query92	59	51	48	48
query93	5807	502	496	496
query94	1301	188	185	185
query95	404	322	320	320
query96	606	306	269	269
query97	3173	3082	3095	3082
query98	210	200	193	193
query99	1137	844	855	844
Total cold run time: 289364 ms
Total hot run time: 175131 ms

@doris-robot
Copy link

doris-robot commented Jun 27, 2024

ClickBench: Total hot run time: 30.59 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f148fab59c9480e30e1ca7e37f0e5f54366baf27, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.04
query4	1.69	0.09	0.07
query5	0.49	0.48	0.50
query6	1.13	0.73	0.73
query7	0.02	0.02	0.02
query8	0.05	0.04	0.05
query9	0.55	0.50	0.50
query10	0.54	0.54	0.55
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.60	0.60	0.60
query14	0.80	0.80	0.78
query15	0.83	0.81	0.81
query16	0.35	0.35	0.37
query17	1.00	1.04	0.99
query18	0.21	0.24	0.26
query19	1.87	1.76	1.68
query20	0.02	0.01	0.01
query21	15.46	0.77	0.67
query22	4.22	7.38	1.80
query23	18.30	1.37	1.30
query24	2.12	0.23	0.21
query25	0.14	0.08	0.08
query26	0.27	0.18	0.17
query27	0.09	0.07	0.07
query28	13.26	1.01	1.00
query29	12.59	3.28	3.27
query30	0.26	0.07	0.06
query31	2.88	0.39	0.40
query32	3.24	0.47	0.48
query33	2.89	2.91	2.91
query34	17.24	4.52	4.47
query35	4.53	4.47	4.50
query36	0.65	0.46	0.49
query37	0.18	0.15	0.16
query38	0.16	0.14	0.14
query39	0.04	0.04	0.04
query40	0.17	0.14	0.15
query41	0.09	0.05	0.06
query42	0.05	0.06	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.68 s
Total hot run time: 30.59 s

@Jibing-Li
Copy link
Contributor Author

Jibing-Li commented Jun 27, 2024

run p0

@Jibing-Li Jibing-Li merged commit 3f3a6eb into apache:master Jun 28, 2024
@Jibing-Li Jibing-Li deleted the baseindex branch June 28, 2024 02:29
Jibing-Li added a commit to Jibing-Li/incubator-doris that referenced this pull request Jun 28, 2024
@Jibing-Li
[improvement](statistics)Use real base index id to fetch stats cache. (…
5dd5006 
…apache#36914)

For historical reason, statistics tables use -1 for OlapTable base index
id. This brings many if/else branch for stats calculate. This pr is to
screen the -1 for Nereids. The stats user could use the real base index
id to fetch stats cache. Will do the id translation inside the get cache
api.
@Jibing-Li Jibing-Li mentioned this pull request Jun 28, 2024
Merged
Jibing-Li added a commit that referenced this pull request Jun 28, 2024
@Jibing-Li
[improvement](statistics)Use real base index id to fetch stats cache. (…
816899d 
…#36914) (#36992)

For historical reason, statistics tables use -1 for OlapTable base index
id. This brings many if/else branch for stats calculate. This pr is to
screen the -1 for Nereids. The stats user could use the real base index
id to fetch stats cache. Will do the id translation inside the get cache
api.

backport: #36914
dataroaring pushed a commit that referenced this pull request Jun 28, 2024
@Jibing-Li @dataroaring
[improvement](statistics)Use real base index id to fetch stats cache. (…
7f80190 
…#36914)

For historical reason, statistics tables use -1 for OlapTable base index
id. This brings many if/else branch for stats calculate. This pr is to
screen the -1 for Nereids. The stats user could use the real base index
id to fetch stats cache. Will do the id translation inside the get cache
api.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@englefly englefly englefly approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.5-merged dev/3.0.0-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@Jibing-Li @doris-robot @englefly @morrySnow @dataroaring