Skip to content

[opt](inverted index) opt value extraction from column to string #37395

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
airborne12 merged 1 commit into from
Jul 12, 2024
Merged

[opt](inverted index) opt value extraction from column to string #37395

airborne12 merged 1 commit into from
Jul 12, 2024

Conversation

@zzzxl1993
Copy link
Contributor

@zzzxl1993 zzzxl1993 commented Jul 7, 2024

Proposed changes

Issue Number: close #xxx

@github-actions
Copy link
Contributor

github-actions bot commented Jul 7, 2024

clang-tidy review says "All clean, LGTM! 👍"

xiaokang
xiaokang approved these changes Jul 8, 2024
View reviewed changes
Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 8, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jul 8, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jul 8, 2024

PR approved by anyone and no changes requested.

@xiaokang
Copy link
Contributor

xiaokang commented Jul 8, 2024

@zzzxl1993 pls describe the optimization result

@zzzxl1993
Copy link
Contributor Author

zzzxl1993 commented Jul 12, 2024

run buildall

airborne12
airborne12 approved these changes Jul 12, 2024
View reviewed changes
Copy link
Member

@airborne12 airborne12 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@doris-robot
Copy link

doris-robot commented Jul 12, 2024

TPC-H: Total hot run time: 39929 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b2ba18b1094591e8b32dc0b804419b4b523fd6ad, data reload: false

------ Round 1 ----------------------------------
q1	17638	4720	4280	4280
q2	2040	186	183	183
q3	10454	1158	1133	1133
q4	10182	777	789	777
q5	7541	2680	2657	2657
q6	223	140	137	137
q7	962	601	586	586
q8	9212	2067	2081	2067
q9	8888	6563	6535	6535
q10	8882	3790	3771	3771
q11	456	234	239	234
q12	446	233	222	222
q13	17756	2972	2947	2947
q14	267	239	243	239
q15	514	479	465	465
q16	473	384	379	379
q17	948	648	696	648
q18	7986	7484	7408	7408
q19	4001	1402	1391	1391
q20	659	326	323	323
q21	4984	3262	3309	3262
q22	354	288	285	285
Total cold run time: 114866 ms
Total hot run time: 39929 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4378	4299	4253	4253
q2	374	266	272	266
q3	2989	2754	2916	2754
q4	1979	1716	1766	1716
q5	5659	5557	5502	5502
q6	223	134	139	134
q7	2141	1793	1896	1793
q8	3290	3407	3381	3381
q9	8822	8706	8840	8706
q10	4046	3879	3797	3797
q11	580	470	475	470
q12	784	648	665	648
q13	15837	3146	3159	3146
q14	313	291	276	276
q15	528	471	502	471
q16	492	449	415	415
q17	1831	1498	1514	1498
q18	8057	7835	7786	7786
q19	1768	1543	1611	1543
q20	2063	1862	1933	1862
q21	5082	4897	4857	4857
q22	595	492	510	492
Total cold run time: 71831 ms
Total hot run time: 55766 ms

@doris-robot
Copy link

doris-robot commented Jul 12, 2024

TPC-DS: Total hot run time: 173371 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b2ba18b1094591e8b32dc0b804419b4b523fd6ad, data reload: false

query1	907	388	369	369
query2	6450	1831	1767	1767
query3	6636	215	217	215
query4	28265	17489	17568	17489
query5	3610	478	469	469
query6	282	167	167	167
query7	4595	299	295	295
query8	251	190	192	190
query9	8497	2463	2448	2448
query10	432	305	265	265
query11	10728	10076	9986	9986
query12	111	83	87	83
query13	1638	369	363	363
query14	10196	7602	7609	7602
query15	228	171	164	164
query16	7587	310	314	310
query17	1817	555	514	514
query18	1656	271	270	270
query19	203	145	145	145
query20	88	79	81	79
query21	202	130	123	123
query22	4397	4116	4001	4001
query23	34035	33376	33467	33376
query24	10917	2871	2885	2871
query25	620	400	390	390
query26	1121	157	147	147
query27	2393	277	279	277
query28	7091	2083	2082	2082
query29	916	636	624	624
query30	251	158	158	158
query31	998	802	781	781
query32	100	53	55	53
query33	762	304	321	304
query34	1022	482	508	482
query35	696	576	572	572
query36	1084	959	972	959
query37	148	89	89	89
query38	3036	2908	2871	2871
query39	888	798	866	798
query40	215	126	121	121
query41	48	47	45	45
query42	110	104	104	104
query43	508	453	463	453
query44	1203	737	726	726
query45	185	161	158	158
query46	1097	715	706	706
query47	1873	1781	1769	1769
query48	399	307	295	295
query49	865	404	416	404
query50	772	389	389	389
query51	6943	6884	6770	6770
query52	102	105	89	89
query53	352	290	280	280
query54	882	448	451	448
query55	74	79	80	79
query56	315	270	270	270
query57	1154	1026	1067	1026
query58	242	241	248	241
query59	2910	2668	2678	2668
query60	310	277	281	277
query61	115	116	113	113
query62	792	672	644	644
query63	322	289	285	285
query64	9463	2203	1681	1681
query65	3178	3120	3099	3099
query66	747	327	325	325
query67	15246	15220	15167	15167
query68	4518	563	556	556
query69	493	359	333	333
query70	1207	1151	1042	1042
query71	409	280	277	277
query72	7125	5766	5485	5485
query73	752	332	335	332
query74	6015	5563	5527	5527
query75	3351	2722	2698	2698
query76	2210	937	832	832
query77	429	305	308	305
query78	9502	9010	8918	8918
query79	2646	522	529	522
query80	2448	470	529	470
query81	579	218	217	217
query82	949	141	129	129
query83	300	174	166	166
query84	273	87	90	87
query85	2027	307	309	307
query86	473	332	318	318
query87	3302	3100	3131	3100
query88	4241	2475	2481	2475
query89	471	384	370	370
query90	1726	190	190	190
query91	130	101	99	99
query92	63	51	48	48
query93	2760	524	515	515
query94	1119	215	212	212
query95	415	314	310	310
query96	620	283	273	273
query97	3168	3094	3023	3023
query98	235	198	198	198
query99	1670	1257	1271	1257
Total cold run time: 280902 ms
Total hot run time: 173371 ms

@doris-robot
Copy link

doris-robot commented Jul 12, 2024

ClickBench: Total hot run time: 30.95 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b2ba18b1094591e8b32dc0b804419b4b523fd6ad, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.03
query3	0.23	0.05	0.06
query4	1.68	0.07	0.07
query5	0.49	0.48	0.48
query6	1.13	0.73	0.72
query7	0.02	0.01	0.01
query8	0.04	0.04	0.04
query9	0.55	0.48	0.48
query10	0.55	0.54	0.54
query11	0.14	0.11	0.11
query12	0.14	0.12	0.12
query13	0.59	0.59	0.58
query14	0.76	0.77	0.79
query15	0.85	0.81	0.81
query16	0.35	0.36	0.38
query17	1.00	0.96	0.99
query18	0.22	0.22	0.22
query19	1.77	1.73	1.67
query20	0.02	0.01	0.01
query21	15.38	0.74	0.65
query22	3.65	7.65	2.45
query23	18.27	1.43	1.24
query24	2.17	0.22	0.22
query25	0.15	0.10	0.08
query26	0.30	0.21	0.20
query27	0.46	0.23	0.22
query28	13.25	1.01	0.99
query29	12.64	3.25	3.28
query30	0.25	0.06	0.05
query31	2.89	0.38	0.40
query32	3.26	0.46	0.47
query33	2.94	2.91	2.94
query34	17.12	4.37	4.35
query35	4.47	4.41	4.46
query36	0.67	0.47	0.48
query37	0.18	0.16	0.15
query38	0.15	0.14	0.15
query39	0.04	0.04	0.04
query40	0.15	0.12	0.12
query41	0.10	0.04	0.04
query42	0.06	0.04	0.05
query43	0.04	0.03	0.04
Total cold run time: 109.24 s
Total hot run time: 30.95 s

@zzzxl1993
Copy link
Contributor Author

zzzxl1993 commented Jul 12, 2024

run p0

@zzzxl1993
Copy link
Contributor Author

zzzxl1993 commented Jul 12, 2024

run external

1 similar comment
@zzzxl1993
Copy link
Contributor Author

zzzxl1993 commented Jul 12, 2024

run external

@zzzxl1993
Copy link
Contributor Author

zzzxl1993 commented Jul 12, 2024

run p0

@airborne12 airborne12 merged commit f6afbf7 into apache:master Jul 12, 2024
@xiaokang xiaokang added the doing label Jul 15, 2024
seawinde pushed a commit to seawinde/doris that referenced this pull request Jul 17, 2024
@zzzxl1993 @seawinde
[opt](inverted index) opt value extraction from column to string (apa…
010e70a 
…che#37395)

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@zzzxl1993 @dataroaring
[opt](inverted index) opt value extraction from column to string (#37395
622cc4f 
)

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
zzzxl1993 added a commit to zzzxl1993/doris that referenced this pull request Jul 21, 2024
@zzzxl1993
[opt](inverted index) opt value extraction from column to string (apa…
d0f247f 
…che#37395)

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
@zzzxl1993 zzzxl1993 mentioned this pull request Jul 21, 2024
Merged
airborne12 pushed a commit that referenced this pull request Jul 24, 2024
@zzzxl1993
[opt](inverted index) opt value extraction from column to string (#38179
c6cd1e2 
)

## Proposed changes

#37395
zzzxl1993 added a commit to zzzxl1993/doris that referenced this pull request Aug 6, 2024
@zzzxl1993
[opt](inverted index) opt value extraction from column to string (apa…
e3714eb 
…che#37395)

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
@zzzxl1993 zzzxl1993 mentioned this pull request Aug 6, 2024
Merged
xiaokang pushed a commit that referenced this pull request Aug 13, 2024
@zzzxl1993
[opt](inverted index) opt value extraction from column to string (#37395
cf868f9 
) (#38937)
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Closed
GoGoWen pushed a commit to GoGoWen/incubator-doris that referenced this pull request Aug 27, 2024
@zzzxl1993
[opt](inverted index) opt value extraction from column to string (apa…
0b5ee9f 
…che#37395) (apache#38937)
@yiguolei yiguolei mentioned this pull request Sep 5, 2024
Closed
3 tasks
@xiaokang xiaokang mentioned this pull request Sep 14, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@xiaokang xiaokang xiaokang approved these changes

@airborne12 airborne12 airborne12 approved these changes

@qidaye qidaye qidaye approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.15-merged dev/2.1.6-merged dev/3.0.1-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@zzzxl1993 @xiaokang @doris-robot @airborne12 @qidaye @dataroaring