Skip to content

[fix](meta) fix catalog replay error (#35532) #35660

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
May 30, 2024
Merged

[fix](meta) fix catalog replay error (#35532) #35660

morningman merged 1 commit into from
May 30, 2024

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented May 30, 2024

bp #35532

@morningman
[fix](meta) fix catalog replay error (apache#35532)
b6dbc07 
```
Caused by: java.lang.NullPointerException
    at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936) ~[?:1.8.0_301]
    at org.apache.doris.catalog.ResourceMgr.getResource(ResourceMgr.java:166) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.datasource.CatalogProperty.catalogResource(CatalogProperty.java:67) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.datasource.CatalogProperty.getOrDefault(CatalogProperty.java:77) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.datasource.ExternalCatalog.setDefaultPropsIfMissing(ExternalCatalog.java:173) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.datasource.hive.HMSExternalCatalog.setDefaultPropsIfMissing(HMSExternalCatalog.java:238) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.datasource.ExternalCatalog.gsonPostProcess(ExternalCatalog.java:687) ~[doris-fe.jar:1.2-SNAPSHOT]
    at org.apache.doris.persist.gson.GsonUtils$PostProcessTypeAdapterFactory$1.read(GsonUtils.java:640) ~[doris-fe.jar:1.2-SNAPSHOT]
    at com.google.gson.TypeAdapter.fromJsonTree(TypeAdapter.java:299) ~[gson-2.10.1.jar:?]
    at org.apache.doris.persist.gson.RuntimeTypeAdapterFactory$1.read(RuntimeTypeAdapterFactory.java:289) ~[doris-fe.jar:1.2-SNAPSHOT]
```

Introduced from apache#33610.
When read meta image, the `resource` maybe null, we should ignore it.
@doris-robot
Copy link

doris-robot commented May 30, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@morningman
Copy link
Contributor Author

morningman commented May 30, 2024

run buildall

@doris-robot
Copy link

doris-robot commented May 30, 2024

TPC-H: Total hot run time: 49567 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b6dbc074f17852c2ca00188d585fa6dd2cb6a213, data reload: false

------ Round 1 ----------------------------------
q1	17954	4375	4371	4371
q2	2041	186	146	146
q3	10307	1912	1884	1884
q4	10370	1231	1296	1231
q5	8396	3914	3892	3892
q6	240	127	129	127
q7	2072	1620	1575	1575
q8	9540	2738	2714	2714
q9	13652	10253	10154	10154
q10	8661	3576	3512	3512
q11	424	254	243	243
q12	469	298	302	298
q13	18352	3961	4036	3961
q14	351	325	320	320
q15	512	459	465	459
q16	674	566	575	566
q17	1140	937	914	914
q18	7256	6866	7054	6866
q19	1704	1599	1536	1536
q20	546	314	313	313
q21	4422	4089	4096	4089
q22	503	396	404	396
Total cold run time: 119586 ms
Total hot run time: 49567 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4414	4317	4346	4317
q2	321	232	226	226
q3	4198	4158	4141	4141
q4	2763	2731	2764	2731
q5	7160	7049	7106	7049
q6	239	123	122	122
q7	3241	2817	2867	2817
q8	4369	4465	4476	4465
q9	16820	16808	16784	16784
q10	4205	4263	4323	4263
q11	732	690	709	690
q12	1034	850	856	850
q13	6991	3748	3719	3719
q14	460	421	423	421
q15	505	469	458	458
q16	738	681	693	681
q17	3883	3783	3835	3783
q18	8890	8705	8723	8705
q19	1698	1676	1663	1663
q20	2387	2160	2123	2123
q21	8575	8582	8447	8447
q22	1012	956	917	917
Total cold run time: 84635 ms
Total hot run time: 79372 ms

@doris-robot
Copy link

doris-robot commented May 30, 2024

TPC-DS: Total hot run time: 203104 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b6dbc074f17852c2ca00188d585fa6dd2cb6a213, data reload: false

query1	939	415	386	386
query2	6623	2717	2650	2650
query3	6922	217	212	212
query4	20004	17855	17913	17855
query5	19734	6499	6515	6499
query6	301	224	230	224
query7	4165	306	322	306
query8	249	274	244	244
query9	3125	2679	2605	2605
query10	432	292	310	292
query11	11325	10707	10618	10618
query12	128	78	74	74
query13	5598	673	671	671
query14	18068	13304	13583	13304
query15	385	229	241	229
query16	6444	292	269	269
query17	1693	1459	885	885
query18	2288	416	411	411
query19	220	150	146	146
query20	78	80	81	80
query21	203	101	93	93
query22	5194	4981	5069	4981
query23	32481	31807	32071	31807
query24	6881	6526	6463	6463
query25	515	435	423	423
query26	614	168	160	160
query27	2007	294	299	294
query28	6140	2348	2317	2317
query29	2830	2640	2863	2640
query30	244	163	172	163
query31	900	716	711	711
query32	69	56	65	56
query33	403	256	248	248
query34	844	477	481	477
query35	1126	925	930	925
query36	1291	1393	1185	1185
query37	92	59	58	58
query38	3065	2955	2933	2933
query39	1388	1342	1330	1330
query40	230	94	94	94
query41	41	38	38	38
query42	86	84	85	84
query43	704	790	709	709
query44	1173	724	726	724
query45	240	238	226	226
query46	1227	940	1005	940
query47	1796	1623	1901	1623
query48	1018	723	687	687
query49	621	380	369	369
query50	858	599	606	599
query51	4743	4635	4726	4635
query52	96	76	89	76
query53	444	321	310	310
query54	2653	2491	2451	2451
query55	92	76	85	76
query56	246	205	215	205
query57	1157	1161	1092	1092
query58	216	215	215	215
query59	4238	4077	4259	4077
query60	221	220	213	213
query61	101	101	105	101
query62	809	472	521	472
query63	486	339	339	339
query64	2651	1519	1486	1486
query65	3616	3556	3565	3556
query66	844	374	383	374
query67	16167	15300	15975	15300
query68	9915	660	658	658
query69	573	345	357	345
query70	1650	1473	1481	1473
query71	415	308	322	308
query72	6521	3496	3488	3488
query73	743	326	327	326
query74	6327	5824	5871	5824
query75	5402	3766	3649	3649
query76	6218	1160	1213	1160
query77	1044	258	258	258
query78	12666	11546	11510	11510
query79	9300	648	646	646
query80	1482	400	407	400
query81	502	240	230	230
query82	1593	102	100	100
query83	169	136	136	136
query84	265	72	72	72
query85	891	326	328	326
query86	345	291	330	291
query87	3212	3027	3022	3022
query88	4804	2368	2360	2360
query89	486	324	284	284
query90	1953	216	215	215
query91	180	141	146	141
query92	60	52	56	52
query93	6891	584	628	584
query94	724	214	206	206
query95	1108	1057	1075	1057
query96	648	333	337	333
query97	6520	6312	6417	6312
query98	185	182	178	178
query99	3017	945	865	865
Total cold run time: 317591 ms
Total hot run time: 203104 ms

@doris-robot
Copy link

doris-robot commented May 30, 2024

ClickBench: Total hot run time: 30.25 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b6dbc074f17852c2ca00188d585fa6dd2cb6a213, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.05	0.04
query4	1.79	0.07	0.08
query5	0.55	0.52	0.52
query6	1.23	0.62	0.59
query7	0.01	0.00	0.01
query8	0.03	0.02	0.02
query9	0.53	0.50	0.47
query10	0.53	0.54	0.52
query11	0.12	0.08	0.09
query12	0.12	0.10	0.09
query13	0.61	0.61	0.61
query14	0.79	0.80	0.78
query15	0.79	0.78	0.77
query16	0.40	0.37	0.38
query17	0.98	0.99	0.97
query18	0.24	0.23	0.27
query19	1.90	1.85	1.84
query20	0.01	0.02	0.02
query21	15.46	0.57	0.55
query22	2.13	2.26	1.50
query23	17.13	1.09	0.92
query24	6.69	0.62	1.58
query25	0.41	0.09	0.06
query26	0.69	0.15	0.15
query27	0.05	0.04	0.03
query28	6.27	0.83	0.70
query29	12.76	2.38	2.39
query30	0.63	0.57	0.54
query31	2.80	0.38	0.37
query32	3.40	0.49	0.49
query33	3.09	3.08	3.08
query34	15.30	4.83	4.78
query35	4.85	4.84	4.78
query36	1.06	1.00	1.02
query37	0.06	0.05	0.05
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.17	0.14	0.14
query41	0.06	0.01	0.02
query42	0.02	0.02	0.01
query43	0.02	0.01	0.01
Total cold run time: 104.07 s
Total hot run time: 30.25 s

@doris-robot
Copy link

doris-robot commented May 30, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit b6dbc074f17852c2ca00188d585fa6dd2cb6a213 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.9 seconds inserted 10000000 Rows, about 456K ops/s

@morningman morningman merged commit 3531e23 into apache:branch-2.0 May 30, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@morningman
[fix](meta) fix catalog replay error (apache#35532) (apache#35660)
9f53e8d 
bp apache#35532
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@morningman @doris-robot