[enhance](parquet) support reading brotli compressed parquet file #41875

suxiaogang223 · 2024-10-15T07:29:22Z

Proposed changes

Impl BrotliBlockCompression to uncompressed brotli parquet data.
fix parquet case: group0/large_string_map.brotli.parquet

doris-robot · 2024-10-15T07:29:28Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions · 2024-10-15T07:33:46Z

clang-tidy review says "All clean, LGTM! 👍"

suxiaogang223 · 2024-10-15T08:56:59Z

run buildall

github-actions · 2024-10-15T08:57:57Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-10-15T09:00:15Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-10-16T07:30:38Z

clang-tidy review says "All clean, LGTM! 👍"

suxiaogang223 · 2024-10-18T09:20:52Z

run buildall

github-actions · 2024-10-18T09:32:18Z

clang-tidy review says "All clean, LGTM! 👍"

morningman · 2024-10-18T09:48:43Z

run buildall

morningman · 2024-10-19T00:53:05Z

run buildall

github-actions · 2024-10-19T00:57:28Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-10-19T01:32:14Z

TPC-H: Total hot run time: 41327 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 48b8947aaf4de36afa4fea2f28ec95470b88bfa2, data reload: false

------ Round 1 ----------------------------------
q1	17595	7386	7292	7292
q2	2019	279	294	279
q3	12045	1071	1186	1071
q4	10580	848	796	796
q5	7768	3105	3119	3105
q6	236	150	148	148
q7	1019	623	605	605
q8	9359	1933	1999	1933
q9	6586	6462	6383	6383
q10	7086	2428	2432	2428
q11	448	246	250	246
q12	419	219	217	217
q13	17779	3015	3003	3003
q14	241	216	222	216
q15	564	524	544	524
q16	680	580	591	580
q17	970	618	490	490
q18	7382	6750	6566	6566
q19	1375	1046	1109	1046
q20	494	192	185	185
q21	4211	3262	3209	3209
q22	1090	1012	1005	1005
Total cold run time: 109946 ms
Total hot run time: 41327 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7320	7249	7237	7237
q2	324	235	229	229
q3	3042	3016	2926	2926
q4	2078	1785	1781	1781
q5	5764	5806	5841	5806
q6	230	148	146	146
q7	2274	1856	1793	1793
q8	3435	3536	3480	3480
q9	8938	8966	8924	8924
q10	3576	3557	3587	3557
q11	591	492	492	492
q12	871	646	622	622
q13	8313	3221	3205	3205
q14	312	273	280	273
q15	593	522	515	515
q16	713	661	647	647
q17	1857	1600	1624	1600
q18	8295	7773	7611	7611
q19	1737	1548	1328	1328
q20	2170	1899	1879	1879
q21	5619	5359	5444	5359
q22	1125	1079	1043	1043
Total cold run time: 69177 ms
Total hot run time: 60453 ms

doris-robot · 2024-10-19T01:44:20Z

TPC-DS: Total hot run time: 193048 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 48b8947aaf4de36afa4fea2f28ec95470b88bfa2, data reload: false

query1	913	392	400	392
query2	6251	2107	2068	2068
query3	8689	197	194	194
query4	34334	23696	23672	23672
query5	3514	470	484	470
query6	279	165	170	165
query7	4189	295	294	294
query8	303	262	233	233
query9	9311	2745	2739	2739
query10	457	285	263	263
query11	18055	15317	15297	15297
query12	151	103	97	97
query13	1576	419	417	417
query14	10066	7797	7565	7565
query15	254	177	192	177
query16	7999	475	512	475
query17	1640	596	569	569
query18	1983	312	308	308
query19	373	168	176	168
query20	123	116	114	114
query21	217	114	112	112
query22	4809	4432	4565	4432
query23	35041	34142	33987	33987
query24	11100	2822	2804	2804
query25	632	403	408	403
query26	1326	163	168	163
query27	2574	287	288	287
query28	7792	2429	2447	2429
query29	851	418	420	418
query30	263	151	151	151
query31	1058	815	804	804
query32	99	55	60	55
query33	762	306	297	297
query34	926	502	516	502
query35	866	749	772	749
query36	1106	950	955	950
query37	165	95	93	93
query38	4196	3996	3957	3957
query39	1492	1424	1434	1424
query40	266	108	103	103
query41	51	47	48	47
query42	122	102	103	102
query43	544	506	502	502
query44	1222	814	800	800
query45	194	165	171	165
query46	1136	715	703	703
query47	1921	1832	1826	1826
query48	433	333	321	321
query49	925	429	438	429
query50	849	378	387	378
query51	7049	6990	6872	6872
query52	102	93	91	91
query53	263	183	181	181
query54	1175	441	436	436
query55	82	79	79	79
query56	274	274	299	274
query57	1290	1193	1197	1193
query58	239	230	239	230
query59	3236	3055	3116	3055
query60	296	257	260	257
query61	103	100	102	100
query62	853	681	674	674
query63	222	192	187	187
query64	4730	627	601	601
query65	3337	3266	3272	3266
query66	1339	303	315	303
query67	15962	15553	15639	15553
query68	4930	566	567	566
query69	534	284	280	280
query70	1238	1153	1194	1153
query71	355	284	279	279
query72	7231	4120	4039	4039
query73	794	360	359	359
query74	10364	9082	9150	9082
query75	3419	2709	2712	2709
query76	3143	872	873	872
query77	463	303	298	298
query78	10499	9609	9624	9609
query79	1569	601	595	595
query80	1173	455	455	455
query81	592	255	244	244
query82	773	145	142	142
query83	241	136	145	136
query84	255	76	69	69
query85	1392	300	296	296
query86	364	313	303	303
query87	4443	4333	4567	4333
query88	3120	2222	2198	2198
query89	407	291	291	291
query90	2218	191	190	190
query91	142	99	99	99
query92	63	50	51	50
query93	1086	550	549	549
query94	1070	296	269	269
query95	351	247	249	247
query96	619	279	286	279
query97	3324	3149	3142	3142
query98	221	196	194	194
query99	1591	1308	1299	1299
Total cold run time: 301934 ms
Total hot run time: 193048 ms

doris-robot · 2024-10-19T01:49:47Z

ClickBench: Total hot run time: 32.09 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 48b8947aaf4de36afa4fea2f28ec95470b88bfa2, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.03	0.03
query3	0.23	0.06	0.06
query4	1.65	0.10	0.09
query5	0.51	0.50	0.53
query6	1.12	0.73	0.72
query7	0.01	0.01	0.02
query8	0.04	0.03	0.03
query9	0.59	0.50	0.47
query10	0.55	0.55	0.55
query11	0.14	0.11	0.10
query12	0.13	0.11	0.11
query13	0.63	0.59	0.60
query14	2.70	2.81	2.82
query15	0.91	0.84	0.84
query16	0.38	0.38	0.38
query17	0.97	0.97	1.05
query18	0.20	0.19	0.20
query19	1.93	1.73	1.97
query20	0.01	0.01	0.01
query21	15.36	0.60	0.62
query22	2.51	2.06	1.89
query23	17.13	0.95	0.77
query24	3.63	0.46	0.81
query25	0.21	0.15	0.05
query26	0.54	0.14	0.14
query27	0.04	0.06	0.04
query28	10.95	1.09	1.07
query29	12.52	3.24	3.23
query30	0.25	0.06	0.06
query31	2.87	0.39	0.38
query32	3.27	0.46	0.44
query33	2.99	3.03	3.08
query34	16.90	4.46	4.43
query35	4.54	4.50	4.53
query36	0.65	0.49	0.47
query37	0.08	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.64 s
Total hot run time: 32.09 s

morningman · 2024-10-19T13:15:29Z

run buildall

github-actions · 2024-10-19T13:19:32Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-10-19T13:54:05Z

TeamCity be ut coverage result:
Function Coverage: 37.47% (9709/25912)
Line Coverage: 28.74% (80626/280519)
Region Coverage: 28.19% (41731/148039)
Branch Coverage: 24.76% (21212/85656)
Coverage Report: http://coverage.selectdb-in.cc/coverage/b8c0cdda38a548964ca75e359401abf53afef539_b8c0cdda38a548964ca75e359401abf53afef539/report/index.html

doris-robot · 2024-10-19T13:55:25Z

TPC-H: Total hot run time: 41376 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b8c0cdda38a548964ca75e359401abf53afef539, data reload: false

------ Round 1 ----------------------------------
q1	17627	7432	7266	7266
q2	2029	279	287	279
q3	12212	1075	1219	1075
q4	10562	875	800	800
q5	7751	3122	3042	3042
q6	236	149	148	148
q7	1024	599	604	599
q8	9348	1918	2102	1918
q9	6614	6466	6457	6457
q10	7060	2389	2423	2389
q11	448	253	248	248
q12	408	219	215	215
q13	17768	3040	3038	3038
q14	237	209	211	209
q15	571	538	529	529
q16	659	583	581	581
q17	974	487	527	487
q18	7271	6721	6730	6721
q19	1353	976	1051	976
q20	474	183	185	183
q21	4008	3241	3265	3241
q22	1130	975	1021	975
Total cold run time: 109764 ms
Total hot run time: 41376 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7308	7239	7258	7239
q2	334	232	247	232
q3	3075	2951	2927	2927
q4	2086	1790	1811	1790
q5	5750	5766	5822	5766
q6	227	143	144	143
q7	2260	1831	1803	1803
q8	3397	3564	3473	3473
q9	8963	8883	8870	8870
q10	3599	3555	3530	3530
q11	585	483	486	483
q12	846	655	649	649
q13	10911	3227	3200	3200
q14	320	287	273	273
q15	591	542	549	542
q16	674	646	639	639
q17	1850	1632	1615	1615
q18	8290	7878	7507	7507
q19	1731	1428	1555	1428
q20	2146	1874	1890	1874
q21	5405	5429	5486	5429
q22	1124	1062	1049	1049
Total cold run time: 71472 ms
Total hot run time: 60461 ms

doris-robot · 2024-10-19T14:07:32Z

TPC-DS: Total hot run time: 192880 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b8c0cdda38a548964ca75e359401abf53afef539, data reload: false

query1	921	384	406	384
query2	6257	2179	2091	2091
query3	8679	192	208	192
query4	34347	23607	23618	23607
query5	3543	508	477	477
query6	280	178	176	176
query7	4192	299	297	297
query8	301	244	234	234
query9	9187	2743	2734	2734
query10	457	287	277	277
query11	18113	15406	15552	15406
query12	166	106	104	104
query13	1566	455	429	429
query14	9481	7618	7489	7489
query15	248	172	184	172
query16	8073	456	456	456
query17	1648	602	590	590
query18	2241	312	314	312
query19	375	159	162	159
query20	127	121	114	114
query21	212	112	108	108
query22	4728	4672	4544	4544
query23	35479	34266	34306	34266
query24	11070	2779	2840	2779
query25	612	398	403	398
query26	1143	164	162	162
query27	2634	284	285	284
query28	7829	2453	2425	2425
query29	832	428	433	428
query30	297	159	149	149
query31	1041	801	816	801
query32	94	56	56	56
query33	772	299	300	299
query34	940	512	526	512
query35	905	741	792	741
query36	1095	955	941	941
query37	159	92	98	92
query38	4040	3952	3927	3927
query39	1487	1449	1415	1415
query40	215	98	101	98
query41	50	47	46	46
query42	118	103	98	98
query43	544	503	479	479
query44	1224	795	797	795
query45	200	172	167	167
query46	1133	710	694	694
query47	1928	1827	1813	1813
query48	434	321	318	318
query49	997	435	440	435
query50	836	387	403	387
query51	7142	7036	6969	6969
query52	99	87	87	87
query53	260	179	181	179
query54	1144	437	420	420
query55	81	76	74	74
query56	282	258	267	258
query57	1302	1164	1181	1164
query58	230	279	243	243
query59	3083	3023	3019	3019
query60	293	274	270	270
query61	105	104	102	102
query62	888	663	674	663
query63	226	188	183	183
query64	3964	639	638	638
query65	3311	3199	3217	3199
query66	835	316	304	304
query67	16086	15667	15576	15576
query68	4752	575	582	575
query69	504	302	286	286
query70	1152	1130	1125	1125
query71	372	282	290	282
query72	7307	3921	4171	3921
query73	787	354	361	354
query74	10293	9040	9030	9030
query75	3480	2772	2613	2613
query76	2961	873	897	873
query77	636	306	309	306
query78	10597	9621	9613	9613
query79	1493	599	599	599
query80	1372	461	453	453
query81	595	238	239	238
query82	723	141	148	141
query83	287	138	142	138
query84	280	69	71	69
query85	1609	306	293	293
query86	432	304	307	304
query87	4565	4232	4360	4232
query88	3058	2238	2160	2160
query89	416	287	293	287
query90	2073	190	190	190
query91	136	101	102	101
query92	59	48	48	48
query93	1282	554	543	543
query94	1198	293	296	293
query95	351	246	248	246
query96	612	274	280	274
query97	3355	3121	3130	3121
query98	217	197	195	195
query99	1781	1312	1272	1272
Total cold run time: 301316 ms
Total hot run time: 192880 ms

doris-robot · 2024-10-19T14:13:01Z

ClickBench: Total hot run time: 32.72 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b8c0cdda38a548964ca75e359401abf53afef539, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.02	0.03
query3	0.23	0.07	0.06
query4	1.64	0.10	0.10
query5	0.51	0.50	0.50
query6	1.13	0.73	0.73
query7	0.02	0.01	0.02
query8	0.05	0.03	0.03
query9	0.55	0.51	0.51
query10	0.53	0.55	0.54
query11	0.14	0.10	0.10
query12	0.14	0.11	0.11
query13	0.62	0.59	0.58
query14	2.87	2.87	2.81
query15	0.91	0.82	0.82
query16	0.38	0.40	0.39
query17	1.04	1.00	1.05
query18	0.20	0.20	0.20
query19	1.91	1.80	2.02
query20	0.01	0.00	0.01
query21	15.36	0.61	0.58
query22	2.56	2.90	1.58
query23	17.08	0.97	0.89
query24	3.01	1.56	1.19
query25	0.37	0.19	0.10
query26	0.40	0.13	0.14
query27	0.05	0.04	0.05
query28	10.30	1.09	1.07
query29	12.55	3.28	3.24
query30	0.24	0.06	0.06
query31	2.86	0.39	0.37
query32	3.28	0.45	0.46
query33	2.97	2.99	3.06
query34	17.11	4.42	4.44
query35	4.52	4.52	4.52
query36	0.65	0.47	0.48
query37	0.08	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.17	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.01
query43	0.04	0.03	0.02
Total cold run time: 106.78 s
Total hot run time: 32.72 s

morningman

LGTM

github-actions · 2024-10-20T09:51:46Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-10-20T09:51:48Z

PR approved by anyone and no changes requested.

kaka11chen

LGTM

…ache#41875) Impl BrotliBlockCompression to uncompressed brotli parquet data. fix parquet case: group0/large_string_map.brotli.parquet

…ache#41875) ## Proposed changes Impl BrotliBlockCompression to uncompressed brotli parquet data. fix parquet case: group0/large_string_map.brotli.parquet

(#42177) cherry pick from #41875 Co-authored-by: Socrates <suxiaogang223@icloud.com>

…ile (#42162) pick pr: #41875

suxiaogang223 force-pushed the support_brotli branch from b96e0e0 to 8c29e66 Compare October 15, 2024 08:55

suxiaogang223 force-pushed the support_brotli branch from 5a58b09 to d629082 Compare October 18, 2024 09:18

morningman force-pushed the support_brotli branch from d629082 to 48b8947 Compare October 19, 2024 00:52

suxiaogang223 added 3 commits October 19, 2024 21:15

build brotli and impl

0325432

add regression

32f3b0e

add regression case

b8c0cdd

morningman force-pushed the support_brotli branch from 48b8947 to b8c0cdd Compare October 19, 2024 13:15

morningman approved these changes Oct 20, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 20, 2024

github-actions bot added the reviewed label Oct 20, 2024

morningman added dev/2.1.x dev/3.0.x labels Oct 20, 2024

kaka11chen approved these changes Oct 20, 2024

View reviewed changes

morningman merged commit ff6d2ea into apache:master Oct 21, 2024

suxiaogang223 deleted the support_brotli branch October 21, 2024 06:23

suxiaogang223 mentioned this pull request Oct 21, 2024

[cherry-pick](branch-2.1) support reading brotli compressed parquet file #42162

Merged

morningman mentioned this pull request Oct 21, 2024

[enhance](parquet) support reading brotli compressed parquet file #41875 #42177

Merged

morningman added a commit that referenced this pull request Oct 21, 2024

[enhance](parquet) support reading brotli compressed parquet file #41875

78d38dd

(#42177) cherry pick from #41875 Co-authored-by: Socrates <suxiaogang223@icloud.com>

morningman added dev/2.1.7-merged dev/3.0.3-merged and removed dev/3.0.x dev/2.1.7-merged labels Oct 21, 2024

morningman pushed a commit that referenced this pull request Oct 21, 2024

[cherry-pick](branch-2.1) support reading brotli compressed parquet f…

a32ad0b

…ile (#42162) pick pr: #41875

morningman added dev/2.1.7-merged and removed dev/2.1.x labels Oct 21, 2024

[enhance](parquet) support reading brotli compressed parquet file #41875

[enhance](parquet) support reading brotli compressed parquet file #41875

Uh oh!

Conversation

suxiaogang223 commented Oct 15, 2024

Proposed changes

Uh oh!

doris-robot commented Oct 15, 2024

Uh oh!

github-actions bot commented Oct 15, 2024

Uh oh!

suxiaogang223 commented Oct 15, 2024

Uh oh!

github-actions bot commented Oct 15, 2024

Uh oh!

github-actions bot commented Oct 15, 2024

Uh oh!

github-actions bot commented Oct 16, 2024

Uh oh!

suxiaogang223 commented Oct 18, 2024

Uh oh!

github-actions bot commented Oct 18, 2024

Uh oh!

morningman commented Oct 18, 2024

Uh oh!

morningman commented Oct 19, 2024

Uh oh!

github-actions bot commented Oct 19, 2024

Uh oh!

doris-robot commented Oct 19, 2024

Uh oh!

doris-robot commented Oct 19, 2024

Uh oh!

doris-robot commented Oct 19, 2024

Uh oh!

morningman commented Oct 19, 2024

Uh oh!

github-actions bot commented Oct 19, 2024

Uh oh!

doris-robot commented Oct 19, 2024

Uh oh!

doris-robot commented Oct 19, 2024

Uh oh!

doris-robot commented Oct 19, 2024

Uh oh!

doris-robot commented Oct 19, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 20, 2024

Uh oh!

github-actions bot commented Oct 20, 2024

Uh oh!

kaka11chen left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants