Skip to content

Conversation

@eldenmoon
Copy link
Member

… column name and column path

Both could be reference to related field in TabletColumn.And use shared_ptr for TabletColumn in TabletSchema for later memory reuse

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@eldenmoon
Copy link
Member Author

run buildall

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

}

Status RowCursor::init(const std::vector<TabletColumn>& schema) {
Status RowCursor::init(const std::vector<TabletColumnPtr>& schema) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: method 'init' can be made static [readability-convert-member-functions-to-static]

Suggested change
Status RowCursor::init(const std::vector<TabletColumnPtr>& schema) {
static Status RowCursor::init(const std::vector<TabletColumnPtr>& schema) {

}

Status RowCursor::init(const std::vector<TabletColumn>& schema, size_t column_count) {
Status RowCursor::init(const std::vector<TabletColumnPtr>& schema, size_t column_count) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: method 'init' can be made static [readability-convert-member-functions-to-static]

Suggested change
Status RowCursor::init(const std::vector<TabletColumnPtr>& schema, size_t column_count) {
static Status RowCursor::init(const std::vector<TabletColumnPtr>& schema, size_t column_count) {

@doris-robot
Copy link

TPC-H: Total hot run time: 41543 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2c71b31a5148746e8f154e4c58369e8135c033be, data reload: false

------ Round 1 ----------------------------------
q1	17636	4970	4891	4891
q2	2047	147	138	138
q3	10569	1025	1004	1004
q4	4659	970	997	970
q5	7721	3200	3306	3200
q6	199	138	135	135
q7	1261	776	774	774
q8	9271	2098	2106	2098
q9	7546	6767	6710	6710
q10	8318	2655	2641	2641
q11	403	227	207	207
q12	769	330	325	325
q13	18005	3681	3662	3662
q14	289	261	257	257
q15	627	525	500	500
q16	482	425	421	421
q17	931	885	878	878
q18	7457	6676	6614	6614
q19	1543	1479	1508	1479
q20	618	345	351	345
q21	6736	3955	3981	3955
q22	881	356	339	339
Total cold run time: 107968 ms
Total hot run time: 41543 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4872	4861	4869	4861
q2	295	181	188	181
q3	3611	3607	3598	3598
q4	2574	2555	2549	2549
q5	5782	5778	5794	5778
q6	218	132	132	132
q7	2268	1677	1647	1647
q8	3069	3128	3139	3128
q9	8795	8740	8744	8740
q10	6807	4262	4236	4236
q11	534	389	391	389
q12	779	547	554	547
q13	4331	3409	3422	3409
q14	261	239	233	233
q15	612	501	496	496
q16	519	438	440	438
q17	1708	1629	1630	1629
q18	8265	7755	7559	7559
q19	1669	1630	1639	1630
q20	2108	1849	1853	1849
q21	6608	6223	6222	6222
q22	582	512	517	512
Total cold run time: 66267 ms
Total hot run time: 59763 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 177764 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2c71b31a5148746e8f154e4c58369e8135c033be, data reload: false

query1	920	353	346	346
query2	6524	1612	1756	1612
query3	6704	204	204	204
query4	23190	21241	21140	21140
query5	4240	458	449	449
query6	273	176	171	171
query7	4605	286	287	286
query8	247	190	195	190
query9	8436	2764	2782	2764
query10	417	223	223	223
query11	14973	14566	14604	14566
query12	144	78	82	78
query13	1694	435	413	413
query14	9092	7557	7835	7557
query15	208	183	187	183
query16	7417	259	250	250
query17	1400	558	530	530
query18	1965	293	258	258
query19	192	140	146	140
query20	84	81	85	81
query21	186	125	119	119
query22	4991	4746	4867	4746
query23	32582	31589	31728	31589
query24	12715	3454	3408	3408
query25	652	372	355	355
query26	1898	156	173	156
query27	3035	311	314	311
query28	6615	1843	1823	1823
query29	1140	616	608	608
query30	283	135	142	135
query31	913	753	755	753
query32	93	61	55	55
query33	726	236	238	236
query34	1062	482	496	482
query35	963	855	879	855
query36	1000	948	896	896
query37	169	60	66	60
query38	3276	3142	3185	3142
query39	1387	1331	1319	1319
query40	283	105	103	103
query41	37	36	37	36
query42	99	97	96	96
query43	469	442	443	442
query44	1086	675	689	675
query45	196	184	180	180
query46	1039	760	760	760
query47	1693	1556	1541	1541
query48	419	349	350	349
query49	1213	304	292	292
query50	766	374	382	374
query51	5306	5145	5183	5145
query52	104	97	86	86
query53	410	307	293	293
query54	289	217	216	216
query55	89	80	85	80
query56	220	201	197	197
query57	1032	929	970	929
query58	220	192	195	192
query59	2267	2199	2201	2199
query60	237	213	219	213
query61	86	91	83	83
query62	590	401	378	378
query63	328	290	275	275
query64	6404	3053	3145	3053
query65	3301	3274	3230	3230
query66	1339	331	324	324
query67	14551	14476	14532	14476
query68	5141	547	560	547
query69	515	353	354	353
query70	1270	1252	1187	1187
query71	431	255	264	255
query72	6466	2784	2628	2628
query73	689	305	312	305
query74	6829	6486	6592	6486
query75	3194	2576	2565	2565
query76	3330	1108	1205	1108
query77	357	235	232	232
query78	9450	8851	8819	8819
query79	976	502	499	499
query80	502	348	334	334
query81	443	205	205	205
query82	160	88	85	85
query83	140	119	119	119
query84	220	77	76	76
query85	1034	349	340	340
query86	295	296	308	296
query87	3506	3300	3239	3239
query88	2666	2287	2278	2278
query89	432	352	361	352
query90	1982	162	160	160
query91	160	128	133	128
query92	54	47	48	47
query93	956	487	497	487
query94	1115	187	179	179
query95	8614	365	350	350
query96	566	269	262	262
query97	4475	4256	4241	4241
query98	230	208	195	195
query99	1076	719	701	701
Total cold run time: 277758 ms
Total hot run time: 177764 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.34 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2c71b31a5148746e8f154e4c58369e8135c033be, data reload: false

query1	0.03	0.02	0.02
query2	0.06	0.02	0.03
query3	0.22	0.08	0.07
query4	1.66	0.08	0.09
query5	0.50	0.49	0.48
query6	1.36	0.61	0.61
query7	0.02	0.01	0.01
query8	0.04	0.02	0.02
query9	0.53	0.46	0.45
query10	0.50	0.49	0.49
query11	0.13	0.10	0.10
query12	0.12	0.10	0.10
query13	0.59	0.59	0.58
query14	0.76	0.78	0.78
query15	0.82	0.81	0.80
query16	0.33	0.34	0.34
query17	0.92	0.86	0.90
query18	0.19	0.19	0.16
query19	1.80	1.72	1.70
query20	0.02	0.01	0.01
query21	15.40	0.67	0.59
query22	2.82	3.76	2.86
query23	17.44	1.05	0.97
query24	1.98	0.22	0.36
query25	0.63	0.07	0.06
query26	0.16	0.15	0.14
query27	0.06	0.05	0.06
query28	12.39	0.85	0.84
query29	12.53	3.32	3.30
query30	0.49	0.49	0.48
query31	2.78	0.36	0.37
query32	3.35	0.47	0.47
query33	3.13	3.16	3.10
query34	15.33	4.49	4.52
query35	4.54	4.50	4.54
query36	1.07	0.96	0.97
query37	0.07	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.01	0.01
query40	0.17	0.14	0.14
query41	0.07	0.01	0.02
query42	0.02	0.02	0.01
query43	0.03	0.02	0.02
Total cold run time: 105.12 s
Total hot run time: 31.34 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 2c71b31a5148746e8f154e4c58369e8135c033be with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          60 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.5 seconds inserted 10000000 Rows, about 740K ops/s

@eldenmoon
Copy link
Member Author

run buildall

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

#include <memory>
#include <string>

#include "olap/olap_common.h"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: 'olap/olap_common.h' file not found [clang-diagnostic-error]

#include "olap/olap_common.h"
         ^

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.74% (8552/23930)
Line Coverage: 27.71% (69413/250455)
Region Coverage: 26.83% (36031/134275)
Branch Coverage: 23.65% (18434/77946)
Coverage Report: http://coverage.selectdb-in.cc/coverage/8c3ae42aa37f498ff6ecfb7146a64505b8ca36ad_8c3ae42aa37f498ff6ecfb7146a64505b8ca36ad/report/index.html

@xiaokang xiaokang changed the title [Improve](TabletSchemaCache) reduce duplicated memory consumption for… [Improve](TabletSchemaCache) reduce duplicated memory consumption for column name and column path Feb 27, 2024
TabletColumn& mutable_column_by_uid(int32_t col_unique_id);
const std::vector<TabletColumn>& columns() const;
std::vector<TabletColumn>& mutable_columns();
void relace_column(size_t pos, TabletColumn new_col);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

replace_column

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

… column name and column path

Both could be reference to related field in TabletColumn.And use shared_ptr for TabletColumn in TabletSchema for later memory reuse
@eldenmoon
Copy link
Member Author

run buildall

@eldenmoon
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37815 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d3b6852f7ea5a27c5bd70b55e6fddab023baad4b, data reload: false

------ Round 1 ----------------------------------
q1	17722	3997	4052	3997
q2	2021	146	141	141
q3	10567	931	938	931
q4	4658	923	948	923
q5	7598	2890	3025	2890
q6	178	124	122	122
q7	1304	823	812	812
q8	9484	2046	2093	2046
q9	7253	6529	6461	6461
q10	8202	2549	2539	2539
q11	420	224	219	219
q12	776	315	313	313
q13	17965	3019	2958	2958
q14	282	255	261	255
q15	487	449	442	442
q16	492	397	396	396
q17	955	898	848	848
q18	6759	5837	5814	5814
q19	1589	1526	1516	1516
q20	551	285	278	278
q21	7564	3622	3707	3622
q22	802	295	292	292
Total cold run time: 107629 ms
Total hot run time: 37815 ms

----- Round 2, with runtime_filter_mode=off -----
q1	3997	3964	4005	3964
q2	319	227	233	227
q3	2980	2931	2897	2897
q4	1832	1819	1828	1819
q5	5248	5244	5247	5244
q6	208	115	117	115
q7	2240	1847	1849	1847
q8	3212	3292	3268	3268
q9	8546	8515	8544	8515
q10	6202	3739	3723	3723
q11	523	439	443	439
q12	702	526	563	526
q13	14093	2855	2788	2788
q14	268	249	256	249
q15	481	430	432	430
q16	448	424	426	424
q17	1682	1682	1669	1669
q18	7932	7573	7277	7277
q19	3498	1455	1618	1455
q20	1905	1746	1710	1710
q21	4927	4892	4879	4879
q22	541	475	485	475
Total cold run time: 71784 ms
Total hot run time: 53940 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 178474 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d3b6852f7ea5a27c5bd70b55e6fddab023baad4b, data reload: false

query1	934	359	339	339
query2	7483	2190	2070	2070
query3	6714	222	214	214
query4	26533	21302	21154	21154
query5	4337	936	1369	936
query6	279	183	180	180
query7	4624	306	297	297
query8	248	174	170	170
query9	8419	2265	2236	2236
query10	432	228	227	227
query11	14929	14301	14277	14277
query12	131	96	86	86
query13	1636	464	428	428
query14	8933	7052	6733	6733
query15	236	191	187	187
query16	7238	265	271	265
query17	931	572	564	564
query18	1917	296	288	288
query19	205	167	163	163
query20	94	94	88	88
query21	204	129	124	124
query22	4612	4473	4463	4463
query23	31231	31232	30720	30720
query24	11193	3096	3004	3004
query25	658	410	408	408
query26	1282	170	162	162
query27	2763	368	367	367
query28	6297	1843	1840	1840
query29	1083	611	648	611
query30	306	151	149	149
query31	945	744	758	744
query32	100	62	59	59
query33	769	260	254	254
query34	1038	466	494	466
query35	928	803	821	803
query36	930	814	808	808
query37	136	73	71	71
query38	3202	3073	3107	3073
query39	1412	1363	1405	1363
query40	216	149	119	119
query41	58	56	56	56
query42	113	110	105	105
query43	443	408	429	408
query44	1096	723	709	709
query45	213	199	194	194
query46	1050	806	795	795
query47	1630	1542	1552	1542
query48	426	359	357	357
query49	1115	365	358	358
query50	798	383	392	383
query51	6671	6591	6571	6571
query52	101	98	93	93
query53	350	290	283	283
query54	290	245	242	242
query55	92	88	87	87
query56	243	235	227	227
query57	1079	1017	1003	1003
query58	238	225	218	218
query59	2666	2379	2415	2379
query60	263	276	248	248
query61	117	116	116	116
query62	644	424	418	418
query63	306	283	284	283
query64	5118	3347	3320	3320
query65	2998	2987	2990	2987
query66	991	320	326	320
query67	15020	14390	14258	14258
query68	14124	580	588	580
query69	702	395	380	380
query70	1915	1078	1078	1078
query71	575	280	286	280
query72	10103	2590	2472	2472
query73	3275	336	346	336
query74	7076	6763	6850	6763
query75	6104	2717	2705	2705
query76	7247	1132	1164	1132
query77	759	259	263	259
query78	10287	9683	9577	9577
query79	9095	519	525	519
query80	1090	503	382	382
query81	497	215	216	215
query82	254	89	92	89
query83	268	146	144	144
query84	289	79	80	79
query85	1107	333	333	333
query86	383	313	281	281
query87	3495	3180	3172	3172
query88	3268	2384	2358	2358
query89	528	363	352	352
query90	2225	174	176	174
query91	159	130	129	129
query92	62	53	50	50
query93	3199	532	534	532
query94	1396	187	191	187
query95	471	353	349	349
query96	600	271	268	268
query97	4003	3923	3867	3867
query98	223	216	208	208
query99	1096	775	754	754
Total cold run time: 305562 ms
Total hot run time: 178474 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.52 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d3b6852f7ea5a27c5bd70b55e6fddab023baad4b, data reload: false

query1	0.04	0.04	0.04
query2	0.06	0.03	0.03
query3	0.24	0.06	0.06
query4	1.68	0.09	0.09
query5	0.52	0.52	0.49
query6	1.27	0.67	0.67
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.57	0.52	0.51
query10	0.57	0.57	0.58
query11	0.13	0.09	0.10
query12	0.12	0.11	0.10
query13	0.58	0.57	0.56
query14	0.74	0.75	0.76
query15	0.83	0.80	0.82
query16	0.40	0.38	0.38
query17	0.95	0.99	0.97
query18	0.26	0.26	0.26
query19	1.80	1.71	1.69
query20	0.02	0.01	0.01
query21	15.43	0.62	0.62
query22	3.55	4.13	2.06
query23	17.48	0.95	0.98
query24	2.18	0.80	0.17
query25	0.15	0.09	0.05
query26	0.20	0.15	0.13
query27	0.04	0.03	0.04
query28	12.17	0.84	0.84
query29	12.58	3.59	3.43
query30	0.62	0.58	0.58
query31	2.80	0.33	0.35
query32	3.38	0.44	0.43
query33	2.94	2.93	2.92
query34	15.49	4.29	4.30
query35	4.31	4.30	4.28
query36	1.08	1.01	1.02
query37	0.07	0.05	0.06
query38	0.04	0.03	0.04
query39	0.02	0.02	0.02
query40	0.18	0.13	0.13
query41	0.08	0.02	0.02
query42	0.02	0.01	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.68 s
Total hot run time: 30.52 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit d3b6852f7ea5a27c5bd70b55e6fddab023baad4b with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       18.2 seconds inserted 10000000 Rows, about 549K ops/s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.40% (8550/24155)
Line Coverage: 27.18% (69436/255435)
Region Coverage: 26.41% (36038/136476)
Branch Coverage: 23.25% (18420/79240)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d3b6852f7ea5a27c5bd70b55e6fddab023baad4b_d3b6852f7ea5a27c5bd70b55e6fddab023baad4b/report/index.html

Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 7, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Mar 7, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Mar 7, 2024

PR approved by anyone and no changes requested.

@eldenmoon
Copy link
Member Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.36% (8553/24187)
Line Coverage: 27.13% (69479/256057)
Region Coverage: 26.36% (36049/136775)
Branch Coverage: 23.20% (18427/79440)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a5f04dea124497295c21b3bd0a2cc425e3d108af_a5f04dea124497295c21b3bd0a2cc425e3d108af/report/index.html

@eldenmoon
Copy link
Member Author

run cloud_p0

@eldenmoon
Copy link
Member Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.36% (8552/24187)
Line Coverage: 27.14% (69485/256057)
Region Coverage: 26.36% (36058/136775)
Branch Coverage: 23.20% (18432/79440)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1d5b2d19d525b3f427dd327c2ad7fbf8d42b430a_1d5b2d19d525b3f427dd327c2ad7fbf8d42b430a/report/index.html

Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eldenmoon eldenmoon merged commit 991fdff into apache:master Mar 7, 2024
@eldenmoon eldenmoon deleted the mem-opt2_1 branch March 7, 2024 12:32
yiguolei pushed a commit that referenced this pull request Mar 9, 2024
… column name and column path (#31141)

Both could be reference to related field in TabletColumn.And use shared_ptr for TabletColumn in TabletSchema for later memory reuse
@yiguolei yiguolei mentioned this pull request Mar 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.0 reviewed variant

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants