Skip to content

[feature](function) support hll functions hll_from_base64, hll_to_base64 #32089

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 9 commits into from
Apr 16, 2024
Merged

[feature](function) support hll functions hll_from_base64, hll_to_base64 #32089

morningman merged 9 commits into from
Apr 16, 2024

Conversation

@superdiaodiao
Copy link
Contributor

@superdiaodiao superdiaodiao commented Mar 12, 2024 •
edited
Loading

Proposed changes

Issue Number: #31320

Support two hll functions:

  • hll_from_base64
    Convert a base64 string(result of function hll_to_base64) into a hll.
  • hll_to_base64
    Convert an input hll to a base64 string.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

doris-robot commented Mar 12, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@superdiaodiao
Copy link
Contributor Author

superdiaodiao commented Mar 12, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Mar 12, 2024

TPC-H: Total hot run time: 36886 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f0ff7da6656625fd77d0ee700ac0f4d48a53469e, data reload: false

------ Round 1 ----------------------------------
q1	18344	4611	4286	4286
q2	2514	164	152	152
q3	11209	939	938	938
q4	5295	744	721	721
q5	8360	2875	2774	2774
q6	192	130	132	130
q7	1243	844	826	826
q8	9765	2081	2000	2000
q9	7415	6458	6454	6454
q10	8170	2450	2473	2450
q11	416	213	206	206
q12	768	280	298	280
q13	18301	3044	3042	3042
q14	294	265	270	265
q15	498	465	471	465
q16	475	406	415	406
q17	964	562	540	540
q18	6725	5864	5762	5762
q19	1579	1393	1466	1393
q20	547	289	297	289
q21	7400	3269	3211	3211
q22	780	302	296	296
Total cold run time: 111254 ms
Total hot run time: 36886 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4144	4091	4090	4090
q2	330	227	234	227
q3	2971	2839	2817	2817
q4	1824	1545	1527	1527
q5	5258	5223	5213	5213
q6	197	114	118	114
q7	2242	1798	1823	1798
q8	3156	3297	3296	3296
q9	8563	8598	8578	8578
q10	6235	3649	3682	3649
q11	532	450	438	438
q12	731	545	528	528
q13	12381	2862	2883	2862
q14	265	250	252	250
q15	483	450	439	439
q16	459	413	420	413
q17	1732	1500	1449	1449
q18	7570	7416	7192	7192
q19	1652	1492	1512	1492
q20	1896	1714	1716	1714
q21	4873	4682	4736	4682
q22	549	448	446	446
Total cold run time: 68043 ms
Total hot run time: 53214 ms

@doris-robot
Copy link

doris-robot commented Mar 12, 2024

TeamCity be ut coverage result:
Function Coverage: 34.98% (8559/24469)
Line Coverage: 26.81% (69539/259404)
Region Coverage: 26.06% (36079/138461)
Branch Coverage: 23.03% (18445/80092)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f0ff7da6656625fd77d0ee700ac0f4d48a53469e_f0ff7da6656625fd77d0ee700ac0f4d48a53469e/report/index.html

@doris-robot
Copy link

doris-robot commented Mar 12, 2024

TPC-DS: Total hot run time: 186690 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f0ff7da6656625fd77d0ee700ac0f4d48a53469e, data reload: false

query1	915	352	339	339
query2	7434	2064	1944	1944
query3	6705	211	211	211
query4	31096	20482	20586	20482
query5	4355	456	468	456
query6	258	171	165	165
query7	4621	299	280	280
query8	234	163	169	163
query9	9954	2298	2282	2282
query10	462	222	234	222
query11	17069	13965	14041	13965
query12	134	89	92	89
query13	1641	404	441	404
query14	12090	10397	10409	10397
query15	248	199	189	189
query16	7981	262	258	258
query17	1917	565	547	547
query18	2056	299	288	288
query19	215	154	147	147
query20	89	83	80	80
query21	196	123	123	123
query22	4594	4412	4414	4412
query23	31787	31016	31160	31016
query24	10838	2805	2805	2805
query25	613	358	356	356
query26	1482	157	154	154
query27	3139	369	365	365
query28	8090	1895	1898	1895
query29	905	631	606	606
query30	300	146	149	146
query31	960	700	716	700
query32	102	57	53	53
query33	752	327	236	236
query34	1013	451	481	451
query35	827	593	585	585
query36	991	877	894	877
query37	123	80	78	78
query38	3260	3160	3151	3151
query39	1407	1396	1382	1382
query40	211	109	112	109
query41	50	48	46	46
query42	102	95	94	94
query43	479	447	455	447
query44	1261	700	688	688
query45	272	250	253	250
query46	1075	682	675	675
query47	1706	1617	1585	1585
query48	438	358	357	357
query49	1148	331	336	331
query50	759	367	370	367
query51	6718	6705	6611	6611
query52	110	91	121	91
query53	337	273	269	269
query54	320	243	241	241
query55	85	80	82	80
query56	252	229	240	229
query57	1107	998	1021	998
query58	240	203	208	203
query59	2741	2630	2643	2630
query60	275	270	331	270
query61	114	137	129	129
query62	597	392	410	392
query63	297	271	265	265
query64	5823	3980	4009	3980
query65	3136	3039	3038	3038
query66	1442	332	327	327
query67	14758	14386	14531	14386
query68	5898	530	527	527
query69	579	380	383	380
query70	1218	1116	1143	1116
query71	482	290	282	282
query72	6617	2752	2474	2474
query73	707	318	331	318
query74	7107	6514	6494	6494
query75	9238	7915	8076	7915
query76	4243	927	962	927
query77	631	256	252	252
query78	10279	9716	9838	9716
query79	8936	526	527	526
query80	1909	368	389	368
query81	539	216	208	208
query82	1624	208	196	196
query83	282	149	143	143
query84	283	78	74	74
query85	1647	324	316	316
query86	486	303	293	293
query87	3370	3265	3289	3265
query88	5134	2365	2366	2365
query89	535	350	353	350
query90	1998	171	172	171
query91	157	125	128	125
query92	63	46	45	45
query93	7225	525	491	491
query94	1216	172	171	171
query95	413	322	324	322
query96	604	268	272	268
query97	4424	4243	4281	4243
query98	238	205	200	200
query99	1249	762	762	762
Total cold run time: 316037 ms
Total hot run time: 186690 ms

@doris-robot
Copy link

doris-robot commented Mar 12, 2024

ClickBench: Total hot run time: 30.54 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f0ff7da6656625fd77d0ee700ac0f4d48a53469e, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.06	0.06
query4	1.68	0.10	0.10
query5	0.51	0.51	0.49
query6	1.22	0.64	0.64
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.55	0.51	0.50
query10	0.55	0.56	0.54
query11	0.12	0.09	0.09
query12	0.11	0.09	0.09
query13	0.58	0.57	0.56
query14	0.72	0.75	0.76
query15	0.81	0.80	0.77
query16	0.40	0.40	0.43
query17	1.00	0.98	1.06
query18	0.20	0.25	0.23
query19	1.83	1.76	1.74
query20	0.01	0.01	0.01
query21	15.41	0.56	0.57
query22	2.28	1.78	1.72
query23	16.82	0.85	0.80
query24	3.31	1.83	1.05
query25	0.22	0.32	0.09
query26	0.53	0.12	0.13
query27	0.04	0.03	0.03
query28	10.91	0.89	0.84
query29	12.57	3.26	3.27
query30	0.71	0.62	0.58
query31	2.81	0.34	0.33
query32	3.38	0.44	0.45
query33	2.81	2.77	2.83
query34	16.43	4.34	4.32
query35	4.29	4.30	4.32
query36	1.06	1.01	0.99
query37	0.06	0.05	0.06
query38	0.05	0.03	0.03
query39	0.02	0.01	0.02
query40	0.18	0.14	0.14
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.02	0.03
Total cold run time: 104.72 s
Total hot run time: 30.54 s

@doris-robot
Copy link

doris-robot commented Mar 12, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit f0ff7da6656625fd77d0ee700ac0f4d48a53469e with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       16.5 seconds inserted 10000000 Rows, about 606K ops/s

morrySnow
morrySnow previously approved these changes Mar 13, 2024
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 13, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Mar 13, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Mar 13, 2024

PR approved by anyone and no changes requested.

HappenLee
HappenLee reviewed Mar 13, 2024
View reviewed changes
HappenLee
HappenLee reviewed Mar 13, 2024
View reviewed changes
HappenLee
HappenLee reviewed Mar 13, 2024
View reviewed changes
HappenLee
HappenLee reviewed Mar 13, 2024
View reviewed changes
HappenLee
HappenLee reviewed Mar 13, 2024
View reviewed changes
HappenLee
HappenLee reviewed Mar 13, 2024
View reviewed changes
@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Mar 13, 2024
@superdiaodiao
Copy link
Contributor Author

superdiaodiao commented Mar 13, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Mar 13, 2024

TPC-H: Total hot run time: 36779 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9f87b2184ff3615b9b35120724277c5dea28c4c1, data reload: false

------ Round 1 ----------------------------------
q1	17913	4268	4179	4179
q2	2339	153	153	153
q3	11396	1017	919	919
q4	4991	713	708	708
q5	7643	2855	2830	2830
q6	191	127	128	127
q7	1231	813	817	813
q8	9482	2088	2002	2002
q9	7266	6490	6495	6490
q10	8123	2511	2473	2473
q11	420	218	214	214
q12	817	293	303	293
q13	17969	2984	2995	2984
q14	279	240	247	240
q15	493	460	450	450
q16	481	390	390	390
q17	946	540	576	540
q18	6713	5915	5775	5775
q19	1583	1381	1459	1381
q20	544	288	284	284
q21	7408	3257	3233	3233
q22	792	301	309	301
Total cold run time: 109020 ms
Total hot run time: 36779 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4119	4058	4059	4058
q2	322	221	230	221
q3	2952	2781	2840	2781
q4	1814	1547	1506	1506
q5	5182	5238	5233	5233
q6	207	114	117	114
q7	2262	1870	1865	1865
q8	3143	3269	3268	3268
q9	8606	8571	8553	8553
q10	6144	3703	3621	3621
q11	532	446	439	439
q12	729	542	553	542
q13	16522	2846	2823	2823
q14	275	248	249	248
q15	485	444	435	435
q16	448	409	413	409
q17	1726	1477	1467	1467
q18	7575	7077	7080	7077
q19	5270	1501	1519	1501
q20	1917	1715	1697	1697
q21	4755	4546	4706	4546
q22	554	473	454	454
Total cold run time: 75539 ms
Total hot run time: 52858 ms

@doris-robot
Copy link

doris-robot commented Mar 13, 2024

TPC-DS: Total hot run time: 187629 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9f87b2184ff3615b9b35120724277c5dea28c4c1, data reload: false

query1	904	358	356	356
query2	7577	2023	1967	1967
query3	6708	222	216	216
query4	31191	20607	20837	20607
query5	4368	425	500	425
query6	258	175	168	168
query7	4616	323	285	285
query8	228	169	168	168
query9	9855	2288	2270	2270
query10	453	239	249	239
query11	15279	14172	14665	14172
query12	144	90	83	83
query13	1626	412	422	412
query14	11519	10962	10364	10364
query15	281	217	194	194
query16	8019	269	260	260
query17	1947	537	527	527
query18	2062	267	280	267
query19	258	161	155	155
query20	91	100	78	78
query21	198	128	128	128
query22	4636	4522	4468	4468
query23	31772	31226	31203	31203
query24	10786	2908	2892	2892
query25	604	353	351	351
query26	1184	156	157	156
query27	2772	349	350	349
query28	7834	1893	1892	1892
query29	941	635	622	622
query30	305	155	152	152
query31	979	706	727	706
query32	94	65	54	54
query33	751	261	248	248
query34	1063	476	484	476
query35	856	618	617	617
query36	1028	845	860	845
query37	110	80	75	75
query38	3289	3186	3117	3117
query39	1421	1369	1379	1369
query40	212	112	111	111
query41	50	46	46	46
query42	104	96	99	96
query43	506	447	458	447
query44	1184	730	701	701
query45	281	260	248	248
query46	1184	708	704	704
query47	1725	1597	1619	1597
query48	430	367	355	355
query49	1126	346	322	322
query50	816	375	374	374
query51	6772	6616	6574	6574
query52	101	100	87	87
query53	395	283	280	280
query54	305	231	238	231
query55	87	80	82	80
query56	253	255	228	228
query57	1110	1015	1000	1000
query58	233	209	208	208
query59	2847	2562	2612	2562
query60	263	250	246	246
query61	109	110	112	110
query62	625	411	399	399
query63	306	276	282	276
query64	5314	4253	4169	4169
query65	3110	3044	2995	2995
query66	1430	334	338	334
query67	14923	14552	14655	14552
query68	6910	538	529	529
query69	621	381	379	379
query70	1293	1168	1082	1082
query71	531	295	279	279
query72	6795	2742	2453	2453
query73	773	315	326	315
query74	7057	6601	6621	6601
query75	9608	8241	8173	8173
query76	4777	1060	930	930
query77	635	253	269	253
query78	10522	9649	9578	9578
query79	11947	529	537	529
query80	1701	402	378	378
query81	515	209	203	203
query82	837	205	192	192
query83	212	148	144	144
query84	279	79	74	74
query85	1462	328	306	306
query86	417	303	303	303
query87	3472	3186	3209	3186
query88	5274	2310	2295	2295
query89	515	379	367	367
query90	2006	177	173	173
query91	158	130	134	130
query92	60	48	46	46
query93	7266	510	494	494
query94	1211	177	174	174
query95	425	323	330	323
query96	674	277	280	277
query97	4469	4285	4252	4252
query98	236	206	198	198
query99	1253	756	748	748
Total cold run time: 317719 ms
Total hot run time: 187629 ms

@doris-robot
Copy link

doris-robot commented Mar 13, 2024

ClickBench: Total hot run time: 29.99 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9f87b2184ff3615b9b35120724277c5dea28c4c1, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.02	0.02
query3	0.23	0.06	0.06
query4	1.68	0.10	0.10
query5	0.53	0.51	0.49
query6	1.20	0.61	0.62
query7	0.02	0.01	0.02
query8	0.04	0.02	0.02
query9	0.55	0.49	0.50
query10	0.56	0.55	0.55
query11	0.12	0.09	0.09
query12	0.11	0.09	0.09
query13	0.57	0.56	0.54
query14	0.72	0.75	0.75
query15	0.79	0.76	0.77
query16	0.40	0.38	0.39
query17	1.01	1.03	1.00
query18	0.20	0.25	0.22
query19	1.79	1.68	1.64
query20	0.02	0.01	0.01
query21	15.41	0.56	0.56
query22	2.52	2.24	1.70
query23	17.20	0.79	0.67
query24	3.07	1.80	1.00
query25	0.11	0.21	0.06
query26	0.66	0.12	0.12
query27	0.04	0.03	0.03
query28	11.72	0.88	0.86
query29	12.54	3.14	3.13
query30	0.61	0.59	0.60
query31	2.81	0.34	0.33
query32	3.39	0.44	0.44
query33	2.84	2.81	2.76
query34	16.03	4.29	4.30
query35	4.33	4.32	4.31
query36	1.07	1.01	1.00
query37	0.07	0.05	0.05
query38	0.04	0.03	0.03
query39	0.03	0.01	0.01
query40	0.16	0.14	0.13
query41	0.07	0.02	0.02
query42	0.02	0.02	0.02
query43	0.02	0.02	0.02
Total cold run time: 105.4 s
Total hot run time: 29.99 s

github-actions[bot]
github-actions bot reviewed Apr 11, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

@superdiaodiao
Copy link
Contributor Author

superdiaodiao commented Apr 11, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Apr 11, 2024

TeamCity be ut coverage result:
Function Coverage: 35.60% (8907/25022)
Line Coverage: 27.32% (73135/267744)
Region Coverage: 26.45% (37828/143006)
Branch Coverage: 23.21% (19278/83070)
Coverage Report: http://coverage.selectdb-in.cc/coverage/bd8488535f792fd33172ab4cabd16519b69b717d_bd8488535f792fd33172ab4cabd16519b69b717d/report/index.html

@superdiaodiao superdiaodiao requested a review from morrySnow April 11, 2024 16:25
HappenLee
HappenLee reviewed Apr 11, 2024
View reviewed changes
HappenLee
HappenLee reviewed Apr 11, 2024
View reviewed changes
HappenLee
HappenLee reviewed Apr 11, 2024
View reviewed changes
be/src/vec/functions/hll_from_base64.cpp
doris::Slice decoded_slice(decode_buff.data(), outlen);
doris::HyperLogLog hll;
if (!hll.deserialize(decoded_slice)) {
return Status::RuntimeError(
Copy link
Contributor

@HappenLee HappenLee Apr 11, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not set NULL ? or return ERROR?

Copy link
Contributor Author

@superdiaodiao superdiaodiao Apr 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I saw bitmap_from_base64() return error, shouldn't we keep the same?

@superdiaodiao
Copy link
Contributor Author

superdiaodiao commented Apr 12, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Apr 12, 2024

TPC-H: Total hot run time: 38769 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit efae565c75100668269029886e08c19c77a0d8e1, data reload: false

------ Round 1 ----------------------------------
q1	17736	4490	4382	4382
q2	2460	193	186	186
q3	10779	1141	1182	1141
q4	10524	810	884	810
q5	7742	2817	2729	2729
q6	223	137	134	134
q7	950	567	555	555
q8	9423	2080	2059	2059
q9	7839	6619	6601	6601
q10	8655	3585	3513	3513
q11	466	226	229	226
q12	437	222	210	210
q13	18797	2901	2983	2901
q14	281	227	240	227
q15	511	487	482	482
q16	507	405	381	381
q17	980	704	669	669
q18	7398	6822	6731	6731
q19	5983	1546	1539	1539
q20	684	314	315	314
q21	3449	2681	2788	2681
q22	357	304	298	298
Total cold run time: 116181 ms
Total hot run time: 38769 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4370	4215	4183	4183
q2	364	280	276	276
q3	2994	2737	2735	2735
q4	1852	1590	1585	1585
q5	5328	5344	5345	5344
q6	215	122	120	120
q7	1764	1411	1395	1395
q8	3245	3332	3365	3332
q9	8512	8601	8591	8591
q10	3907	3772	3748	3748
q11	581	478	483	478
q12	760	590	592	590
q13	17671	2937	2944	2937
q14	309	261	277	261
q15	496	463	465	463
q16	473	416	424	416
q17	1782	1474	1467	1467
q18	7643	7536	7458	7458
q19	1666	1529	1584	1529
q20	1942	1752	1732	1732
q21	4911	4850	4970	4850
q22	523	457	491	457
Total cold run time: 71308 ms
Total hot run time: 53947 ms

@superdiaodiao
Copy link
Contributor Author

superdiaodiao commented Apr 12, 2024

run feut

@doris-robot
Copy link

doris-robot commented Apr 12, 2024

TPC-DS: Total hot run time: 183186 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit efae565c75100668269029886e08c19c77a0d8e1, data reload: false

query1	898	1132	1116	1116
query2	7246	2557	2358	2358
query3	6654	203	212	203
query4	37441	21793	21513	21513
query5	4153	386	398	386
query6	265	184	175	175
query7	4039	292	286	286
query8	219	171	175	171
query9	5746	2302	2301	2301
query10	534	243	245	243
query11	14840	14203	14378	14203
query12	137	91	89	89
query13	982	359	359	359
query14	9139	6870	6822	6822
query15	204	172	181	172
query16	7370	256	254	254
query17	1732	582	551	551
query18	1731	280	267	267
query19	190	157	146	146
query20	92	88	108	88
query21	198	127	126	126
query22	4960	4869	4893	4869
query23	33764	33050	33132	33050
query24	12822	2912	2872	2872
query25	554	376	376	376
query26	1793	148	158	148
query27	3052	302	303	302
query28	7741	2033	2016	2016
query29	868	598	589	589
query30	309	164	161	161
query31	886	735	718	718
query32	61	53	54	53
query33	566	258	233	233
query34	860	471	473	471
query35	838	690	715	690
query36	1054	908	936	908
query37	283	69	70	69
query38	3567	3458	3443	3443
query39	1585	1554	1547	1547
query40	273	124	126	124
query41	53	47	51	47
query42	111	101	119	101
query43	574	552	547	547
query44	1432	713	708	708
query45	280	262	267	262
query46	1065	720	733	720
query47	1965	1842	1870	1842
query48	362	290	281	281
query49	1137	369	361	361
query50	751	380	376	376
query51	6731	6632	6622	6622
query52	101	95	90	90
query53	345	285	283	283
query54	258	227	221	221
query55	77	71	72	71
query56	236	220	220	220
query57	1246	1125	1123	1123
query58	222	201	203	201
query59	3287	3068	3053	3053
query60	283	222	236	222
query61	96	96	91	91
query62	623	432	454	432
query63	301	281	273	273
query64	5056	3356	3923	3356
query65	3152	3049	3043	3043
query66	1319	334	323	323
query67	15320	15080	15125	15080
query68	4599	533	526	526
query69	490	305	302	302
query70	1255	1151	1125	1125
query71	394	275	266	266
query72	6386	2652	2428	2428
query73	728	317	320	317
query74	6882	6480	6571	6480
query75	3085	2319	2327	2319
query76	2968	1103	1106	1103
query77	629	254	255	254
query78	10783	10154	10158	10154
query79	3080	519	515	515
query80	2033	428	434	428
query81	526	243	233	233
query82	1218	102	97	97
query83	342	176	175	175
query84	267	90	87	87
query85	1692	371	257	257
query86	463	302	307	302
query87	3828	3546	3580	3546
query88	5284	2362	2435	2362
query89	478	381	377	377
query90	1986	174	178	174
query91	118	96	95	95
query92	56	47	47	47
query93	3877	519	501	501
query94	1265	177	179	177
query95	397	297	307	297
query96	588	267	255	255
query97	2724	2492	2482	2482
query98	232	216	214	214
query99	1242	874	882	874
Total cold run time: 295924 ms
Total hot run time: 183186 ms

@doris-robot
Copy link

doris-robot commented Apr 12, 2024

ClickBench: Total hot run time: 30.22 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit efae565c75100668269029886e08c19c77a0d8e1, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.04	0.04
query3	0.23	0.05	0.05
query4	1.66	0.07	0.07
query5	0.50	0.51	0.51
query6	1.12	0.65	0.64
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.54	0.49	0.50
query10	0.55	0.55	0.54
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.60	0.58	0.59
query14	0.77	0.79	0.75
query15	0.82	0.81	0.80
query16	0.36	0.35	0.38
query17	1.02	1.00	1.04
query18	0.22	0.23	0.24
query19	1.90	1.80	1.72
query20	0.01	0.01	0.01
query21	15.43	0.67	0.66
query22	4.38	7.65	1.77
query23	18.31	1.48	1.23
query24	1.94	0.24	0.21
query25	0.16	0.08	0.08
query26	0.26	0.16	0.16
query27	0.08	0.08	0.08
query28	13.32	1.00	0.98
query29	12.65	3.36	3.32
query30	0.26	0.07	0.05
query31	2.89	0.38	0.39
query32	3.26	0.45	0.46
query33	2.85	2.86	2.85
query34	17.10	4.40	4.37
query35	4.52	4.48	4.48
query36	0.64	0.46	0.46
query37	0.19	0.16	0.16
query38	0.16	0.14	0.14
query39	0.05	0.03	0.03
query40	0.17	0.14	0.14
query41	0.10	0.05	0.04
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.6 s
Total hot run time: 30.22 s

@doris-robot
Copy link

doris-robot commented Apr 12, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit efae565c75100668269029886e08c19c77a0d8e1 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      33 seconds loaded 861443392 Bytes, about 24 MB/s
Insert into select:       13.8 seconds inserted 10000000 Rows, about 724K ops/s

@superdiaodiao superdiaodiao mentioned this pull request Apr 13, 2024
Closed
3 tasks
HappenLee
HappenLee approved these changes Apr 14, 2024
View reviewed changes
Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Apr 14, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 14, 2024
@morningman morningman merged commit c7c8916 into apache:master Apr 16, 2024
@superdiaodiao superdiaodiao mentioned this pull request Apr 16, 2024
Merged
yiguolei pushed a commit that referenced this pull request Apr 16, 2024
@superdiaodiao @Doris-Extras
[feature](function) support hll functions hll_from_base64, hll_to_bas…
8640246 
…e64 (#32089)

Issue Number: #31320 

Support two hll functions:

- hll_from_base64
Convert a base64 string(result of function hll_to_base64) into a hll.
- hll_to_base64
Convert an input hll to a base64 string.
morningman pushed a commit to apache/doris-website that referenced this pull request Apr 17, 2024
@superdiaodiao
[Doc](function) Support hll_from_base64, hll_to_base64, group_array_i…
d7ed522 
…ntersect functions (#560)

- hll_from_base64, hll_to_base64:
master pr: apache/doris#32089

- group_array_intersect:
master pr: apache/doris#33265
yiguolei pushed a commit that referenced this pull request Apr 17, 2024
@superdiaodiao @Doris-Extras
[feature](function) support hll functions hll_from_base64, hll_to_bas…
22a6b1d 
…e64 (#32089)

Issue Number: #31320 

Support two hll functions:

- hll_from_base64
Convert a base64 string(result of function hll_to_base64) into a hll.
- hll_to_base64
Convert an input hll to a base64 string.
@yiguolei yiguolei mentioned this pull request Apr 26, 2024
Closed
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@morningman morningman morningman approved these changes

@HappenLee HappenLee HappenLee approved these changes

@morrySnow morrySnow Awaiting requested review from morrySnow

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@superdiaodiao @doris-robot @morningman @HappenLee @morrySnow