Skip to content

[Improve](Tablet Schema) Use deterministic way to serialize protobuf (#30906) #31947

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
xiaokang merged 1 commit into from
Mar 7, 2024
Merged

[Improve](Tablet Schema) Use deterministic way to serialize protobuf (#30906) #31947

xiaokang merged 1 commit into from
Mar 7, 2024

Conversation

@eldenmoon
Copy link
Member

@eldenmoon eldenmoon commented Mar 7, 2024 •
edited by xiaokang
Loading

#30906

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

doris-robot commented Mar 7, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@eldenmoon
Copy link
Member Author

eldenmoon commented Mar 7, 2024

run buildall

@eldenmoon
Copy link
Member Author

eldenmoon commented Mar 7, 2024

run buildall

@eldenmoon
Copy link
Member Author

eldenmoon commented Mar 7, 2024

run buildall

github-actions[bot]
github-actions bot reviewed Mar 7, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/olap/tablet_schema.h
}

// Dump [(name, type, is_nullable), ...]
string dump_structure() const {
Copy link
Contributor

@github-actions github-actions bot Mar 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'dump_structure' should be marked [[nodiscard]] [modernize-use-nodiscard]

Suggested change
string dump_structure() const {
[[nodiscard]] string dump_structure() const {

@doris-robot
Copy link

doris-robot commented Mar 7, 2024

TPC-H: Total hot run time: 50236 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 65a45ae02440ccdec8c227a2cfe5cceedf98164e, data reload: false

------ Round 1 ----------------------------------
q1	20190	4561	4497	4497
q2	2828	164	158	158
q3	11600	1940	1890	1890
q4	10106	1235	1303	1235
q5	8473	3991	4036	3991
q6	229	124	124	124
q7	2008	1606	1609	1606
q8	9278	2739	2734	2734
q9	10805	10527	10495	10495
q10	8652	3498	3520	3498
q11	427	247	243	243
q12	468	298	318	298
q13	18352	3988	4036	3988
q14	357	323	320	320
q15	517	460	453	453
q16	693	598	585	585
q17	1142	1021	975	975
q18	7384	6821	6794	6794
q19	1686	1568	1564	1564
q20	532	315	313	313
q21	4485	4142	4078	4078
q22	508	397	398	397
Total cold run time: 120720 ms
Total hot run time: 50236 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4342	4295	4293	4293
q2	316	229	213	213
q3	4163	4177	4130	4130
q4	2746	2739	2761	2739
q5	7309	7205	7236	7205
q6	235	122	116	116
q7	3228	2808	2822	2808
q8	4360	4456	4529	4456
q9	17204	17087	16916	16916
q10	4234	4321	4260	4260
q11	744	675	687	675
q12	1041	878	883	878
q13	6462	3701	3765	3701
q14	456	411	425	411
q15	502	449	459	449
q16	771	702	704	702
q17	3788	3868	3837	3837
q18	8769	8764	8937	8764
q19	1737	1715	1649	1649
q20	2380	2122	2088	2088
q21	8494	8524	8534	8524
q22	1055	953	963	953
Total cold run time: 84336 ms
Total hot run time: 79767 ms

@doris-robot
Copy link

doris-robot commented Mar 7, 2024

TeamCity be ut coverage result:
Function Coverage: 37.80% (8038/21265)
Line Coverage: 29.48% (65587/222447)
Region Coverage: 28.95% (33749/116571)
Branch Coverage: 24.79% (17314/69846)
Coverage Report: http://coverage.selectdb-in.cc/coverage/65a45ae02440ccdec8c227a2cfe5cceedf98164e_65a45ae02440ccdec8c227a2cfe5cceedf98164e/report/index.html

@doris-robot
Copy link

doris-robot commented Mar 7, 2024

TPC-DS: Total hot run time: 201573 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 65a45ae02440ccdec8c227a2cfe5cceedf98164e, data reload: false

query1	911	398	376	376
query2	6520	2403	2032	2032
query3	6916	199	203	199
query4	20143	17986	17930	17930
query5	19726	6573	6474	6474
query6	288	213	230	213
query7	4153	301	298	298
query8	241	274	247	247
query9	3026	2630	2548	2548
query10	409	300	297	297
query11	11295	10654	10742	10654
query12	116	73	73	73
query13	5568	636	605	605
query14	17731	13723	13451	13451
query15	368	250	232	232
query16	6431	255	257	255
query17	1749	1449	860	860
query18	2321	395	393	393
query19	199	139	135	135
query20	68	79	73	73
query21	186	98	98	98
query22	5351	5082	5056	5056
query23	32450	31766	31802	31766
query24	6920	6602	6459	6459
query25	514	431	419	419
query26	523	156	152	152
query27	1892	288	289	288
query28	6129	2199	2167	2167
query29	2943	2782	2729	2729
query30	238	158	159	158
query31	883	732	746	732
query32	66	56	61	56
query33	390	258	243	243
query34	847	460	467	460
query35	1107	885	938	885
query36	1820	1561	1467	1467
query37	84	63	58	58
query38	3075	2911	2864	2864
query39	1363	1312	1326	1312
query40	198	94	93	93
query41	35	34	31	31
query42	87	79	80	79
query43	600	627	633	627
query44	1109	704	713	704
query45	236	229	233	229
query46	1232	983	984	983
query47	1738	1754	1630	1630
query48	963	661	658	658
query49	617	368	368	368
query50	845	581	581	581
query51	4790	4660	4688	4660
query52	82	77	69	69
query53	436	321	309	309
query54	2634	2450	2442	2442
query55	86	80	78	78
query56	211	203	203	203
query57	1224	1123	1008	1008
query58	210	209	216	209
query59	3575	3394	3277	3277
query60	214	191	221	191
query61	85	82	82	82
query62	813	502	467	467
query63	468	328	333	328
query64	2497	1407	1348	1348
query65	3610	3582	3559	3559
query66	824	368	360	360
query67	17807	17016	15862	15862
query68	5415	616	667	616
query69	546	348	347	347
query70	2042	1720	1659	1659
query71	377	300	308	300
query72	6287	3349	3421	3349
query73	727	310	308	308
query74	6305	5872	5767	5767
query75	4428	3647	3676	3647
query76	2970	1186	1209	1186
query77	460	245	233	233
query78	12588	11488	11583	11488
query79	8367	615	663	615
query80	1456	388	381	381
query81	497	229	227	227
query82	1619	97	100	97
query83	166	134	128	128
query84	254	69	65	65
query85	989	276	272	272
query86	374	365	395	365
query87	3237	3006	3023	3006
query88	5165	2270	2272	2270
query89	384	282	335	282
query90	1881	200	202	200
query91	151	120	113	113
query92	56	54	51	51
query93	5150	548	555	548
query94	805	204	206	204
query95	1108	1081	1054	1054
query96	641	324	311	311
query97	6547	6333	6373	6333
query98	184	187	169	169
query99	2976	833	887	833
Total cold run time: 306338 ms
Total hot run time: 201573 ms

@doris-robot
Copy link

doris-robot commented Mar 7, 2024

ClickBench: Total hot run time: 30.06 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 65a45ae02440ccdec8c227a2cfe5cceedf98164e, data reload: false

query1	0.02	0.02	0.02
query2	0.06	0.02	0.02
query3	0.25	0.05	0.04
query4	1.81	0.07	0.06
query5	0.53	0.52	0.53
query6	1.45	0.63	0.61
query7	0.02	0.01	0.02
query8	0.04	0.02	0.02
query9	0.53	0.48	0.48
query10	0.53	0.55	0.54
query11	0.11	0.08	0.09
query12	0.12	0.09	0.09
query13	0.62	0.62	0.61
query14	0.77	0.80	0.79
query15	0.79	0.76	0.76
query16	0.37	0.37	0.37
query17	1.01	1.03	1.02
query18	0.23	0.25	0.25
query19	1.94	1.86	1.79
query20	0.01	0.01	0.01
query21	15.49	0.56	0.55
query22	2.08	1.97	1.13
query23	17.02	1.00	1.03
query24	5.09	0.77	1.36
query25	0.38	0.13	0.04
query26	0.59	0.16	0.17
query27	0.04	0.04	0.04
query28	7.96	0.71	0.73
query29	12.60	2.35	2.25
query30	0.55	0.53	0.55
query31	2.81	0.39	0.37
query32	3.40	0.51	0.49
query33	3.08	3.08	3.07
query34	15.26	4.80	4.79
query35	4.88	4.83	4.83
query36	1.07	1.01	1.02
query37	0.06	0.04	0.05
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.13	0.14
query41	0.06	0.02	0.01
query42	0.02	0.01	0.02
query43	0.02	0.01	0.02
Total cold run time: 103.88 s
Total hot run time: 30.06 s

@doris-robot
Copy link

doris-robot commented Mar 7, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 65a45ae02440ccdec8c227a2cfe5cceedf98164e with default session variables
Stream load json:         21 seconds loaded 2358488459 Bytes, about 107 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       20.0 seconds inserted 10000000 Rows, about 500K ops/s

@xiaokang xiaokang changed the title [Improve](Tablet Schema) Use deterministic way to serialize protobuf … [Improve](Tablet Schema) Use deterministic way to serialize protobuf (#30906) Mar 7, 2024
@xiaokang xiaokang merged commit f3064a8 into apache:branch-2.0 Mar 7, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@eldenmoon
[Improve](Tablet Schema) Use deterministic way to serialize protobuf (a…
47960c0 
…pache#30906) (apache#31947)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@eldenmoon @doris-robot @xiaokang