Skip to content

[cherry-pick](branch-2.0) fix inverted index file size #37838

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
airborne12 merged 1 commit into from
Jul 16, 2024
Merged

[cherry-pick](branch-2.0) fix inverted index file size #37838

airborne12 merged 1 commit into from
Jul 16, 2024

Conversation

@csun5285
Copy link
Contributor

@csun5285 csun5285 commented Jul 15, 2024 •
edited
Loading

Proposed changes

pick from master #37232

Issue Number: close #xxx

@doris-robot
Copy link

doris-robot commented Jul 15, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@csun5285
Copy link
Contributor Author

csun5285 commented Jul 15, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jul 15, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Jul 15, 2024

TPC-H: Total hot run time: 49764 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 520a2834bd1b63ef57e2357b0df86a37875187b8, data reload: false

------ Round 1 ----------------------------------
q1	17536	4365	4287	4287
q2	2068	158	145	145
q3	10494	1870	1912	1870
q4	10380	1252	1349	1252
q5	8415	3827	3911	3827
q6	241	125	128	125
q7	2030	1659	1612	1612
q8	9498	2687	2696	2687
q9	13786	11026	10381	10381
q10	8672	3543	3518	3518
q11	418	241	257	241
q12	464	300	298	298
q13	18344	3990	4007	3990
q14	363	334	324	324
q15	496	461	460	460
q16	669	585	580	580
q17	1142	961	903	903
q18	7303	6908	6890	6890
q19	1767	1654	1598	1598
q20	511	295	305	295
q21	4354	4039	4077	4039
q22	535	442	448	442
Total cold run time: 119486 ms
Total hot run time: 49764 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4316	4309	4272	4272
q2	324	227	219	219
q3	4177	4135	4119	4119
q4	2753	2733	2745	2733
q5	7144	7090	7108	7090
q6	235	121	122	121
q7	3245	2914	2819	2819
q8	4351	4476	4518	4476
q9	16949	16802	16757	16757
q10	4197	4281	4241	4241
q11	743	684	700	684
q12	1029	881	855	855
q13	5344	3775	3737	3737
q14	443	429	422	422
q15	509	459	457	457
q16	732	672	678	672
q17	3856	3864	3905	3864
q18	8702	8660	8751	8660
q19	1749	1658	1672	1658
q20	2347	2097	2068	2068
q21	8399	8469	8387	8387
q22	1117	1004	1001	1001
Total cold run time: 82661 ms
Total hot run time: 79312 ms

@doris-robot
Copy link

doris-robot commented Jul 15, 2024

TeamCity be ut coverage result:
Function Coverage: 37.80% (8115/21466)
Line Coverage: 29.47% (66512/225670)
Region Coverage: 28.94% (34275/118428)
Branch Coverage: 24.82% (17610/70940)
Coverage Report: http://coverage.selectdb-in.cc/coverage/520a2834bd1b63ef57e2357b0df86a37875187b8_520a2834bd1b63ef57e2357b0df86a37875187b8/report/index.html

@doris-robot
Copy link

doris-robot commented Jul 15, 2024

TPC-DS: Total hot run time: 203396 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 520a2834bd1b63ef57e2357b0df86a37875187b8, data reload: false

query1	927	423	378	378
query2	6525	2805	2616	2616
query3	6916	216	202	202
query4	20104	18129	18089	18089
query5	19736	6479	6484	6479
query6	290	214	221	214
query7	4163	311	304	304
query8	411	426	419	419
query9	3130	2671	2615	2615
query10	418	312	283	283
query11	11464	10844	10962	10844
query12	126	75	73	73
query13	5598	685	693	685
query14	17645	13401	13266	13266
query15	368	246	246	246
query16	6467	293	264	264
query17	1692	1463	849	849
query18	2332	413	422	413
query19	197	154	151	151
query20	82	78	80	78
query21	193	99	93	93
query22	5321	5213	4916	4916
query23	32628	31792	32158	31792
query24	6950	6539	6553	6539
query25	526	432	427	427
query26	535	159	163	159
query27	1896	294	300	294
query28	6241	2379	2347	2347
query29	2881	2775	2800	2775
query30	241	169	168	168
query31	893	760	720	720
query32	70	62	58	58
query33	386	265	262	262
query34	827	465	469	465
query35	1103	900	974	900
query36	1289	1125	1314	1125
query37	89	60	61	60
query38	3036	2926	2963	2926
query39	1379	1316	1321	1316
query40	206	100	99	99
query41	46	44	45	44
query42	81	84	90	84
query43	803	671	711	671
query44	1127	726	716	716
query45	250	238	239	238
query46	1236	958	950	950
query47	1766	1762	1691	1691
query48	1010	706	699	699
query49	611	379	388	379
query50	848	621	616	616
query51	4795	4621	4631	4621
query52	94	79	88	79
query53	450	322	328	322
query54	2655	2474	2462	2462
query55	80	79	81	79
query56	233	224	203	203
query57	1232	1187	1187	1187
query58	214	208	206	206
query59	4252	3976	3953	3953
query60	223	192	212	192
query61	95	93	103	93
query62	788	442	517	442
query63	491	346	345	345
query64	2605	1538	1481	1481
query65	3604	3560	3577	3560
query66	813	367	375	367
query67	15311	16003	15037	15037
query68	10102	652	638	638
query69	568	356	355	355
query70	1978	1341	1473	1341
query71	389	305	316	305
query72	6480	3475	3442	3442
query73	727	312	314	312
query74	6264	5911	5837	5837
query75	5261	3743	3701	3701
query76	6500	1189	1183	1183
query77	1143	258	254	254
query78	12944	12134	11655	11655
query79	12045	671	639	639
query80	1436	398	398	398
query81	493	240	230	230
query82	1675	96	98	96
query83	165	134	133	133
query84	263	69	69	69
query85	896	333	332	332
query86	335	289	294	289
query87	3297	3011	3041	3011
query88	5200	2331	2302	2302
query89	484	276	296	276
query90	1971	209	213	209
query91	185	138	141	138
query92	56	55	55	55
query93	7039	578	599	578
query94	734	206	212	206
query95	1093	1079	1038	1038
query96	644	325	324	324
query97	6418	6359	6381	6359
query98	189	190	172	172
query99	3037	893	942	893
Total cold run time: 321005 ms
Total hot run time: 203396 ms

@doris-robot
Copy link

doris-robot commented Jul 15, 2024

ClickBench: Total hot run time: 30.83 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 520a2834bd1b63ef57e2357b0df86a37875187b8, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.03	0.02
query3	0.25	0.04	0.04
query4	1.78	0.06	0.07
query5	0.53	0.53	0.53
query6	1.35	0.62	0.61
query7	0.01	0.00	0.01
query8	0.03	0.03	0.02
query9	0.51	0.49	0.48
query10	0.53	0.53	0.53
query11	0.12	0.08	0.09
query12	0.11	0.10	0.09
query13	0.62	0.61	0.62
query14	0.80	0.79	0.79
query15	0.78	0.76	0.76
query16	0.36	0.39	0.37
query17	1.01	1.00	1.04
query18	0.23	0.24	0.24
query19	1.90	1.89	1.90
query20	0.02	0.01	0.00
query21	15.49	0.57	0.54
query22	2.29	2.11	1.53
query23	17.50	1.02	0.88
query24	4.25	1.40	1.14
query25	0.29	0.14	0.05
query26	0.55	0.15	0.18
query27	0.03	0.04	0.04
query28	8.41	0.79	0.72
query29	12.65	2.32	2.34
query30	0.54	0.52	0.52
query31	2.82	0.39	0.38
query32	3.37	0.50	0.49
query33	3.09	3.06	3.08
query34	15.25	4.81	4.77
query35	4.86	4.83	4.84
query36	1.05	1.02	1.02
query37	0.06	0.04	0.04
query38	0.04	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.14	0.14
query41	0.07	0.02	0.02
query42	0.03	0.01	0.02
query43	0.02	0.01	0.02
Total cold run time: 103.87 s
Total hot run time: 30.83 s

@doris-robot
Copy link

doris-robot commented Jul 15, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 520a2834bd1b63ef57e2357b0df86a37875187b8 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.1 seconds inserted 10000000 Rows, about 473K ops/s

@yiguolei yiguolei changed the title [cherry-pick](branch-2.1) fix inverted index file size [cherry-pick](branch-2.0) fix inverted index file size Jul 16, 2024
airborne12
airborne12 approved these changes Jul 16, 2024
View reviewed changes
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Jul 16, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 16, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jul 16, 2024

PR approved by anyone and no changes requested.

@airborne12 airborne12 merged commit 2ddaab3 into apache:branch-2.0 Jul 16, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@csun5285
[cherry-pick](branch-2.0) fix inverted index file size (apache#37838)
219403e 
## Proposed changes

pick from master apache#37232
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@airborne12 airborne12 airborne12 approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. kind/test reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@csun5285 @doris-robot @airborne12