Skip to content

Conversation

@eldenmoon
Copy link
Member

@eldenmoon eldenmoon commented Jun 25, 2024

Proposed changes

  1. Variant use serialize_one_row_to_string to string and then parse to jsonb as row store.Since we could not get the original string after [Refactor](Variant) refactor flush logic to support partial update #34925
  2. Remove redundant code
    Issue Number: close #xxx

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@eldenmoon
Copy link
Member Author

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 42960 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1f06617115998f710c3f307c73832e58ddebd923, data reload: false

------ Round 1 ----------------------------------
q1	17699	5139	4750	4750
q2	2009	185	178	178
q3	10666	1199	1168	1168
q4	11069	987	779	779
q5	8650	3049	2982	2982
q6	246	136	138	136
q7	1060	620	635	620
q8	9283	2315	2307	2307
q9	9800	7132	7045	7045
q10	9088	3911	3947	3911
q11	468	248	250	248
q12	495	230	230	230
q13	17793	3008	2964	2964
q14	290	231	213	213
q15	538	481	491	481
q16	518	393	382	382
q17	1020	776	729	729
q18	8390	7760	7502	7502
q19	3689	1602	1568	1568
q20	667	319	319	319
q21	5300	4105	4359	4105
q22	431	345	343	343
Total cold run time: 119169 ms
Total hot run time: 42960 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4844	4705	4883	4705
q2	401	292	292	292
q3	3315	3071	3033	3033
q4	2094	1686	1711	1686
q5	5747	5723	5740	5723
q6	257	130	131	130
q7	2393	1752	1761	1752
q8	3501	3711	3696	3696
q9	8830	8810	8728	8728
q10	4265	3957	4061	3957
q11	645	515	506	506
q12	872	634	645	634
q13	16704	3212	3201	3201
q14	336	287	290	287
q15	564	506	493	493
q16	515	443	435	435
q17	1974	1597	1579	1579
q18	8415	8319	7625	7625
q19	1948	1634	1749	1634
q20	2164	1880	1834	1834
q21	5425	5022	5214	5022
q22	709	544	599	544
Total cold run time: 75918 ms
Total hot run time: 57496 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172992 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1f06617115998f710c3f307c73832e58ddebd923, data reload: false

query1	929	368	379	368
query2	6177	2490	2465	2465
query3	6637	198	200	198
query4	19394	17259	17334	17259
query5	3794	473	478	473
query6	260	168	169	168
query7	4602	300	294	294
query8	312	289	322	289
query9	8502	2391	2375	2375
query10	569	302	287	287
query11	10485	9982	9979	9979
query12	131	80	83	80
query13	1620	366	369	366
query14	9346	7771	7048	7048
query15	225	185	178	178
query16	7876	296	263	263
query17	1881	541	511	511
query18	2000	266	264	264
query19	191	152	146	146
query20	86	79	86	79
query21	208	126	130	126
query22	4441	4080	4022	4022
query23	33815	33692	33351	33351
query24	11865	3014	2990	2990
query25	648	374	380	374
query26	1666	162	162	162
query27	2979	332	325	325
query28	7756	2126	2083	2083
query29	1021	622	610	610
query30	226	150	150	150
query31	967	763	717	717
query32	99	52	64	52
query33	755	277	283	277
query34	893	460	468	460
query35	729	631	606	606
query36	1111	955	935	935
query37	147	71	69	69
query38	2893	2730	2756	2730
query39	874	786	836	786
query40	256	127	123	123
query41	55	51	50	50
query42	120	95	99	95
query43	612	530	554	530
query44	1205	725	729	725
query45	202	169	168	168
query46	1078	696	698	696
query47	1834	1764	1758	1758
query48	374	301	305	301
query49	923	444	430	430
query50	768	385	395	385
query51	7077	6816	6712	6712
query52	109	92	95	92
query53	372	298	296	296
query54	945	445	434	434
query55	80	71	72	71
query56	271	264	255	255
query57	1111	1032	1038	1032
query58	264	288	245	245
query59	3476	3151	3175	3151
query60	286	264	266	264
query61	93	89	92	89
query62	630	444	448	444
query63	334	300	293	293
query64	9683	2235	1736	1736
query65	3258	3107	3104	3104
query66	1326	330	332	330
query67	15477	15000	14870	14870
query68	4502	548	543	543
query69	527	482	373	373
query70	1183	1143	1156	1143
query71	388	274	266	266
query72	6882	5322	5081	5081
query73	744	332	328	328
query74	5899	5453	5490	5453
query75	3427	2739	2676	2676
query76	2550	906	872	872
query77	437	310	312	310
query78	10378	9847	9729	9729
query79	2559	510	508	508
query80	1318	469	474	469
query81	596	218	225	218
query82	1256	104	102	102
query83	291	172	174	172
query84	241	88	85	85
query85	1627	287	275	275
query86	482	306	303	303
query87	3338	3109	3068	3068
query88	3923	2363	2351	2351
query89	499	390	390	390
query90	1888	198	188	188
query91	130	97	97	97
query92	64	50	51	50
query93	2366	521	523	521
query94	1331	188	187	187
query95	403	317	318	317
query96	592	270	268	268
query97	3253	3085	3041	3041
query98	228	223	197	197
query99	1229	817	845	817
Total cold run time: 274860 ms
Total hot run time: 172992 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.92 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1f06617115998f710c3f307c73832e58ddebd923, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.06	0.05
query4	1.66	0.09	0.08
query5	0.50	0.48	0.48
query6	1.13	0.73	0.72
query7	0.03	0.01	0.01
query8	0.04	0.04	0.04
query9	0.55	0.49	0.49
query10	0.54	0.54	0.53
query11	0.16	0.11	0.12
query12	0.15	0.12	0.11
query13	0.60	0.58	0.59
query14	0.75	0.77	0.80
query15	0.83	0.81	0.81
query16	0.35	0.37	0.37
query17	0.99	0.96	0.97
query18	0.20	0.26	0.23
query19	1.75	1.70	1.70
query20	0.02	0.01	0.01
query21	15.45	0.66	0.66
query22	4.21	7.05	2.26
query23	18.28	1.32	1.32
query24	2.13	0.22	0.23
query25	0.14	0.09	0.09
query26	0.27	0.19	0.18
query27	0.08	0.08	0.08
query28	13.22	1.02	1.01
query29	12.61	3.28	3.28
query30	0.26	0.06	0.05
query31	2.87	0.39	0.38
query32	3.27	0.47	0.46
query33	2.92	2.92	2.94
query34	17.25	4.38	4.45
query35	4.46	4.46	4.46
query36	0.65	0.45	0.46
query37	0.18	0.15	0.16
query38	0.16	0.14	0.14
query39	0.04	0.03	0.04
query40	0.18	0.14	0.14
query41	0.09	0.05	0.04
query42	0.07	0.06	0.06
query43	0.05	0.04	0.05
Total cold run time: 109.43 s
Total hot run time: 30.92 s

@eldenmoon eldenmoon force-pushed the master branch 3 times, most recently from 505c62c to 4ac1288 Compare June 26, 2024 02:19
@eldenmoon
Copy link
Member Author

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39912 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4ac1288f23c9f1c8fd370a224d270f6955c318e7, data reload: false

------ Round 1 ----------------------------------
q1	17741	4433	4303	4303
q2	2002	190	180	180
q3	10533	1196	1092	1092
q4	10214	814	730	730
q5	7456	2709	2651	2651
q6	220	134	135	134
q7	957	599	603	599
q8	9230	2087	2085	2085
q9	9106	6546	6514	6514
q10	8991	3698	3723	3698
q11	473	232	238	232
q12	463	227	229	227
q13	18717	2957	2973	2957
q14	267	221	209	209
q15	529	470	489	470
q16	538	382	382	382
q17	971	683	755	683
q18	8122	7523	7493	7493
q19	8857	1455	1516	1455
q20	656	317	326	317
q21	4961	3166	4046	3166
q22	399	335	341	335
Total cold run time: 121403 ms
Total hot run time: 39912 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4428	4278	4244	4244
q2	374	279	275	275
q3	3152	2945	2880	2880
q4	2058	1698	1712	1698
q5	5477	5516	5565	5516
q6	229	128	138	128
q7	2233	1838	1891	1838
q8	3294	3446	3427	3427
q9	8721	8842	8847	8842
q10	4026	3781	3909	3781
q11	593	527	516	516
q12	812	656	650	650
q13	16186	3171	3178	3171
q14	316	283	280	280
q15	523	479	496	479
q16	502	443	421	421
q17	1805	1513	1508	1508
q18	8288	8149	8080	8080
q19	1948	1628	1596	1596
q20	3255	1882	1831	1831
q21	5126	4825	4981	4825
q22	642	571	573	571
Total cold run time: 73988 ms
Total hot run time: 56557 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174674 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4ac1288f23c9f1c8fd370a224d270f6955c318e7, data reload: false

query1	914	385	386	385
query2	6360	2467	2345	2345
query3	6627	205	201	201
query4	19582	17434	17334	17334
query5	3729	469	461	461
query6	237	161	182	161
query7	4589	295	303	295
query8	311	278	291	278
query9	8558	2473	2429	2429
query10	565	307	268	268
query11	10559	9997	10092	9997
query12	117	83	103	83
query13	1629	375	368	368
query14	10005	7623	7623	7623
query15	247	185	186	185
query16	7725	269	268	268
query17	1890	543	515	515
query18	1952	272	276	272
query19	191	147	154	147
query20	91	85	83	83
query21	211	132	122	122
query22	4330	4050	4049	4049
query23	33626	33534	33634	33534
query24	10853	2925	2862	2862
query25	592	404	392	392
query26	718	166	165	165
query27	2380	359	330	330
query28	6099	2205	2191	2191
query29	906	646	655	646
query30	294	156	158	156
query31	970	789	748	748
query32	105	57	55	55
query33	696	329	310	310
query34	908	474	485	474
query35	758	658	657	657
query36	1144	1016	971	971
query37	146	73	73	73
query38	2979	2854	2807	2807
query39	887	859	824	824
query40	209	132	129	129
query41	53	52	54	52
query42	123	94	99	94
query43	568	557	562	557
query44	1081	740	732	732
query45	210	157	163	157
query46	1070	735	747	735
query47	1862	1754	1774	1754
query48	372	293	299	293
query49	892	408	411	408
query50	776	393	413	393
query51	6934	6918	6791	6791
query52	103	97	92	92
query53	361	296	293	293
query54	883	446	440	440
query55	73	77	74	74
query56	282	265	260	260
query57	1143	1059	1026	1026
query58	256	236	240	236
query59	3472	3228	3361	3228
query60	293	268	277	268
query61	91	93	93	93
query62	603	463	437	437
query63	328	291	294	291
query64	8808	2233	1755	1755
query65	3202	3133	3102	3102
query66	740	327	337	327
query67	15260	15044	14901	14901
query68	4584	525	534	525
query69	563	380	336	336
query70	1173	1076	1101	1076
query71	406	279	275	275
query72	7931	5314	5398	5314
query73	747	334	332	332
query74	5947	5555	5512	5512
query75	3648	2663	2696	2663
query76	2408	937	1017	937
query77	695	312	298	298
query78	10414	9898	9656	9656
query79	2352	516	517	516
query80	1138	462	465	462
query81	564	219	223	219
query82	781	106	103	103
query83	326	174	174	174
query84	264	90	84	84
query85	1790	292	278	278
query86	462	324	341	324
query87	3321	3109	3066	3066
query88	4484	2474	2483	2474
query89	486	381	384	381
query90	1813	191	191	191
query91	128	100	101	100
query92	63	51	49	49
query93	2404	511	507	507
query94	1212	193	189	189
query95	413	323	314	314
query96	605	274	275	274
query97	3211	3065	3046	3046
query98	218	200	191	191
query99	1107	838	826	826
Total cold run time: 270186 ms
Total hot run time: 174674 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.91 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4ac1288f23c9f1c8fd370a224d270f6955c318e7, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.05
query3	0.22	0.04	0.05
query4	1.67	0.08	0.09
query5	0.50	0.49	0.50
query6	1.15	0.73	0.72
query7	0.02	0.01	0.02
query8	0.04	0.04	0.04
query9	0.53	0.49	0.48
query10	0.55	0.54	0.54
query11	0.15	0.11	0.11
query12	0.14	0.12	0.12
query13	0.60	0.58	0.59
query14	0.76	0.80	0.80
query15	0.82	0.80	0.81
query16	0.35	0.37	0.37
query17	0.99	0.99	1.00
query18	0.19	0.26	0.24
query19	1.80	1.73	1.68
query20	0.01	0.01	0.01
query21	15.45	0.65	0.64
query22	4.53	8.04	1.22
query23	18.26	1.49	1.32
query24	2.11	0.23	0.22
query25	0.16	0.09	0.09
query26	0.28	0.18	0.18
query27	0.07	0.08	0.08
query28	13.24	1.03	0.99
query29	12.63	3.29	3.29
query30	0.26	0.07	0.06
query31	2.85	0.38	0.38
query32	3.30	0.48	0.47
query33	2.90	2.85	2.96
query34	17.19	4.44	4.52
query35	4.64	4.48	4.45
query36	0.66	0.46	0.46
query37	0.18	0.16	0.15
query38	0.15	0.14	0.14
query39	0.04	0.04	0.04
query40	0.17	0.15	0.14
query41	0.10	0.05	0.05
query42	0.05	0.04	0.04
query43	0.05	0.04	0.05
Total cold run time: 109.88 s
Total hot run time: 29.91 s

Copy link
Contributor

@qidaye qidaye left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 28, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xiaokang xiaokang added dev/2.0.0-alpha1 should be merged into branch-2.0-alpha dev/2.1.x and removed dev/2.0.0-alpha1 should be merged into branch-2.0-alpha labels Jun 28, 2024
@eldenmoon eldenmoon merged commit 5765ffa into apache:master Jun 28, 2024
dataroaring pushed a commit that referenced this pull request Jun 28, 2024
)

1. Variant use serialize_one_row_to_string to string and then parse to
jsonb as row store.Since we could not get the original string after
#34925
2. Remove redundant code
eldenmoon added a commit to eldenmoon/incubator-doris that referenced this pull request Jul 10, 2024
…che#36793)

1. Variant use serialize_one_row_to_string to string and then parse to
jsonb as row store.Since we could not get the original string after
2. Remove redundant code
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.5-merged dev/3.0.0-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants