Skip to content

Conversation

@morningman
Copy link
Contributor

cherry pick from #41860

We can create hive table for text format with `'file_format'='text'`,
and set related properties:
```sql
create table tb (
    id int,
    `name` string
) PROPERTIES (
    'file_format'='text',
    'compression'='gzip',
    'field.delim'='\t',
    'line.delim'='\n',
    'collection.delim'=';',
    'mapkey.delim'=':',
    'serialization.null.format'='\\N',
    'escape.delim'='\\'
);

```

---------

Co-authored-by: morningman <morningman@163.com>
@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@doris-robot
Copy link

TPC-H: Total hot run time: 40301 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit aa1cde49c13163489e38d3a27232a6d2b4bc7c6a, data reload: false

------ Round 1 ----------------------------------
q1	17604	7422	7247	7247
q2	2049	155	147	147
q3	10763	1068	1144	1068
q4	10543	783	774	774
q5	7742	2818	2792	2792
q6	237	149	152	149
q7	997	632	614	614
q8	9569	1853	1917	1853
q9	7573	6343	6378	6343
q10	6975	2263	2318	2263
q11	434	248	255	248
q12	404	223	211	211
q13	17778	2973	2940	2940
q14	239	219	212	212
q15	540	518	517	517
q16	653	592	603	592
q17	981	567	575	567
q18	7216	6511	6507	6507
q19	3104	987	918	918
q20	554	271	272	271
q21	3944	3243	3078	3078
q22	1088	997	990	990
Total cold run time: 110987 ms
Total hot run time: 40301 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7344	7183	7293	7183
q2	321	224	226	224
q3	2966	2840	2871	2840
q4	2071	1846	1799	1799
q5	5642	5640	5661	5640
q6	232	153	150	150
q7	2218	1772	1751	1751
q8	3305	3527	3364	3364
q9	8821	8781	8797	8781
q10	3535	3517	3513	3513
q11	583	477	480	477
q12	773	610	586	586
q13	16542	3115	3119	3115
q14	303	264	280	264
q15	567	520	521	520
q16	705	665	673	665
q17	1822	1609	1557	1557
q18	8116	7876	7578	7578
q19	4565	1476	1497	1476
q20	2208	1850	1874	1850
q21	5355	5071	5297	5071
q22	1130	1012	1016	1012
Total cold run time: 79124 ms
Total hot run time: 59416 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 194237 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit aa1cde49c13163489e38d3a27232a6d2b4bc7c6a, data reload: false

query1	1259	893	901	893
query2	6273	2106	1991	1991
query3	10775	3839	3605	3605
query4	68001	25604	23453	23453
query5	5500	464	441	441
query6	473	160	163	160
query7	6341	309	302	302
query8	302	203	200	200
query9	9422	2690	2667	2667
query10	496	290	270	270
query11	18100	15106	15774	15106
query12	153	101	108	101
query13	1612	465	446	446
query14	11380	7565	7379	7379
query15	218	179	177	177
query16	7173	540	495	495
query17	1060	559	559	559
query18	1817	301	310	301
query19	213	156	149	149
query20	117	106	104	104
query21	210	101	103	101
query22	4365	4396	4378	4378
query23	34290	33600	34080	33600
query24	5573	2916	2836	2836
query25	508	411	411	411
query26	668	159	165	159
query27	1693	292	303	292
query28	4256	2503	2483	2483
query29	668	433	433	433
query30	235	158	153	153
query31	976	796	825	796
query32	73	57	53	53
query33	424	301	289	289
query34	879	503	507	503
query35	866	709	735	709
query36	1073	958	942	942
query37	140	85	90	85
query38	3885	3893	3849	3849
query39	1499	1413	1411	1411
query40	210	100	101	100
query41	46	45	43	43
query42	122	96	104	96
query43	523	475	478	475
query44	1125	777	783	777
query45	200	166	166	166
query46	1135	737	716	716
query47	1926	1805	1818	1805
query48	430	353	336	336
query49	707	383	375	375
query50	812	419	423	419
query51	7051	6991	7013	6991
query52	102	92	92	92
query53	268	192	185	185
query54	584	470	469	469
query55	81	78	78	78
query56	281	244	264	244
query57	1248	1114	1140	1114
query58	219	228	244	228
query59	3255	2784	2744	2744
query60	283	271	264	264
query61	101	98	105	98
query62	767	662	659	659
query63	223	195	180	180
query64	1464	635	613	613
query65	3228	3174	3244	3174
query66	712	312	306	306
query67	15635	15382	15268	15268
query68	4056	561	546	546
query69	613	281	293	281
query70	1137	1158	1125	1125
query71	465	272	275	272
query72	7643	3940	3943	3940
query73	762	350	334	334
query74	10228	8979	8946	8946
query75	4211	2654	2615	2615
query76	3126	922	875	875
query77	766	289	319	289
query78	10060	9265	9256	9256
query79	7760	601	591	591
query80	1249	438	459	438
query81	555	242	243	242
query82	1104	135	141	135
query83	386	145	139	139
query84	295	80	83	80
query85	1660	308	299	299
query86	439	306	292	292
query87	4502	4253	4345	4253
query88	5221	2428	2389	2389
query89	411	293	291	291
query90	2179	189	189	189
query91	139	113	115	113
query92	61	48	49	48
query93	6310	526	531	526
query94	1129	292	299	292
query95	353	257	252	252
query96	620	294	276	276
query97	3325	3152	3196	3152
query98	217	204	197	197
query99	1809	1267	1285	1267
Total cold run time: 338495 ms
Total hot run time: 194237 ms

@morningman morningman merged commit df9fe45 into apache:branch-3.0 Oct 21, 2024
@gavinchou gavinchou mentioned this pull request Nov 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants