Skip to content

[fix](restore) Reset index id for restore #33648

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
dataroaring merged 1 commit into from
Apr 16, 2024
Merged

[fix](restore) Reset index id for restore #33648

dataroaring merged 1 commit into from
Apr 16, 2024

Conversation

@w41ter
Copy link
Contributor

@w41ter w41ter commented Apr 15, 2024

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

doris-robot commented Apr 15, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@w41ter
Copy link
Contributor Author

w41ter commented Apr 15, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Apr 15, 2024

TPC-H: Total hot run time: 38431 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 827a814a1366befcdf456045122d3cbae9a3768f, data reload: false

------ Round 1 ----------------------------------
q1	17610	4360	4222	4222
q2	2022	188	182	182
q3	10463	1185	1222	1185
q4	10201	822	784	784
q5	7532	2702	2640	2640
q6	225	134	136	134
q7	1038	604	583	583
q8	9240	2040	2038	2038
q9	7947	6588	6526	6526
q10	8569	3544	3538	3538
q11	454	229	232	229
q12	505	220	220	220
q13	18353	2916	2922	2916
q14	283	224	228	224
q15	532	484	490	484
q16	523	395	375	375
q17	968	650	707	650
q18	7356	6749	6620	6620
q19	7072	1562	1530	1530
q20	692	323	292	292
q21	3656	2753	2765	2753
q22	356	306	310	306
Total cold run time: 115597 ms
Total hot run time: 38431 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4343	4207	4227	4207
q2	351	270	266	266
q3	2989	2746	2762	2746
q4	1879	1633	1577	1577
q5	5308	5281	5311	5281
q6	212	123	123	123
q7	2234	1856	1861	1856
q8	3196	3372	3335	3335
q9	8549	8576	8699	8576
q10	4039	3980	4044	3980
q11	594	526	489	489
q12	801	605	622	605
q13	16123	3242	3168	3168
q14	323	287	268	268
q15	525	475	491	475
q16	487	424	447	424
q17	1824	1569	1501	1501
q18	8009	7744	7788	7744
q19	1675	1611	1548	1548
q20	1948	1854	1851	1851
q21	9180	4969	4835	4835
q22	555	477	466	466
Total cold run time: 75144 ms
Total hot run time: 55321 ms

@doris-robot
Copy link

doris-robot commented Apr 15, 2024

TPC-DS: Total hot run time: 184423 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 827a814a1366befcdf456045122d3cbae9a3768f, data reload: false

query1	891	1113	1124	1113
query2	7730	2625	2305	2305
query3	6645	202	199	199
query4	37207	21582	21246	21246
query5	4129	417	388	388
query6	232	179	176	176
query7	4038	295	290	290
query8	217	176	168	168
query9	5783	2306	2314	2306
query10	368	236	255	236
query11	14848	14385	14185	14185
query12	143	90	88	88
query13	998	369	372	369
query14	9460	7002	6813	6813
query15	206	178	180	178
query16	7402	265	261	261
query17	1737	582	597	582
query18	1716	275	267	267
query19	198	152	155	152
query20	93	85	86	85
query21	201	136	130	130
query22	5027	4939	5045	4939
query23	33856	33266	33432	33266
query24	8590	2993	3058	2993
query25	505	389	387	387
query26	721	156	154	154
query27	2860	375	376	375
query28	6356	2076	2073	2073
query29	852	639	628	628
query30	309	182	174	174
query31	933	745	743	743
query32	60	55	55	55
query33	507	242	256	242
query34	922	489	502	489
query35	832	740	714	714
query36	1034	928	916	916
query37	104	72	70	70
query38	3652	3727	3682	3682
query39	1658	1568	1606	1568
query40	180	129	129	129
query41	46	45	47	45
query42	119	104	103	103
query43	569	566	541	541
query44	1289	731	760	731
query45	305	290	289	289
query46	1091	725	722	722
query47	2045	1952	1948	1948
query48	374	320	310	310
query49	810	361	375	361
query50	786	399	404	399
query51	6834	6757	6674	6674
query52	98	95	93	93
query53	355	299	282	282
query54	261	240	236	236
query55	81	74	77	74
query56	246	236	235	235
query57	1263	1229	1198	1198
query58	226	206	235	206
query59	3623	3221	3241	3221
query60	258	247	233	233
query61	100	92	88	88
query62	595	442	431	431
query63	304	280	276	276
query64	4151	3999	3673	3673
query65	3079	3041	3019	3019
query66	719	316	311	311
query67	15575	15271	14989	14989
query68	7040	551	552	551
query69	523	308	312	308
query70	1306	1116	1128	1116
query71	437	273	274	273
query72	6405	2682	2466	2466
query73	836	318	317	317
query74	6873	6504	6404	6404
query75	3091	2269	2370	2269
query76	4233	1128	1125	1125
query77	641	243	253	243
query78	10948	10190	10213	10190
query79	4344	526	528	526
query80	2299	435	413	413
query81	512	238	294	238
query82	1556	96	97	96
query83	363	171	167	167
query84	265	92	82	82
query85	1394	272	262	262
query86	471	312	291	291
query87	3759	3592	3498	3498
query88	5862	2376	2296	2296
query89	478	362	370	362
query90	1838	180	183	180
query91	120	95	97	95
query92	55	47	46	46
query93	5598	517	506	506
query94	1176	182	175	175
query95	377	301	292	292
query96	611	259	258	258
query97	2665	2476	2468	2468
query98	233	225	221	221
query99	1241	878	867	867
Total cold run time: 295952 ms
Total hot run time: 184423 ms

@doris-robot
Copy link

doris-robot commented Apr 15, 2024

ClickBench: Total hot run time: 30.69 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 827a814a1366befcdf456045122d3cbae9a3768f, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.69	0.07	0.07
query5	0.47	0.49	0.49
query6	1.47	0.66	0.66
query7	0.02	0.01	0.01
query8	0.05	0.04	0.03
query9	0.56	0.50	0.49
query10	0.54	0.55	0.55
query11	0.14	0.10	0.11
query12	0.15	0.11	0.12
query13	0.60	0.58	0.58
query14	0.75	0.78	0.77
query15	0.81	0.80	0.80
query16	0.37	0.37	0.36
query17	1.02	0.94	1.02
query18	0.21	0.23	0.25
query19	1.86	1.72	1.80
query20	0.01	0.01	0.01
query21	15.74	0.66	0.66
query22	4.08	6.92	2.19
query23	18.26	1.36	1.34
query24	1.45	0.32	0.25
query25	0.15	0.08	0.09
query26	0.25	0.16	0.16
query27	0.08	0.08	0.08
query28	13.36	1.00	0.99
query29	12.61	3.28	3.28
query30	0.25	0.06	0.06
query31	2.85	0.38	0.37
query32	3.30	0.47	0.46
query33	2.83	2.83	2.85
query34	17.09	4.39	4.40
query35	4.46	4.50	4.48
query36	0.65	0.46	0.46
query37	0.17	0.15	0.15
query38	0.16	0.14	0.14
query39	0.04	0.04	0.03
query40	0.18	0.14	0.14
query41	0.10	0.05	0.04
query42	0.06	0.04	0.05
query43	0.04	0.03	0.03
Total cold run time: 109.23 s
Total hot run time: 30.69 s

@doris-robot
Copy link

doris-robot commented Apr 15, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 827a814a1366befcdf456045122d3cbae9a3768f with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      33 seconds loaded 861443392 Bytes, about 24 MB/s
Insert into select:       13.7 seconds inserted 10000000 Rows, about 729K ops/s

dataroaring
dataroaring approved these changes Apr 15, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 15, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Apr 15, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Apr 15, 2024

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit d778aa5 into apache:master Apr 16, 2024
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Apr 17, 2024
@w41ter
[fix](restore) Reset index id for restore (apache#33648)
b96218a
@w41ter w41ter mentioned this pull request Apr 17, 2024
Merged
@w41ter w41ter deleted the fix_reset_id_for_restore branch April 17, 2024 08:04
dataroaring pushed a commit that referenced this pull request Apr 17, 2024
@w41ter
Cherry-pick fix about restore indexes and ids (#33772)
4adb907 
* [fix](restore) Reset index id for restore (#33648)

* [fix](restore) add indexes as part of table signature (#33650)
dataroaring pushed a commit that referenced this pull request Apr 17, 2024
@w41ter @dataroaring
[fix](restore) Reset index id for restore (#33648)
801e21c
yiguolei pushed a commit that referenced this pull request Apr 17, 2024
@w41ter @Doris-Extras
[fix](restore) Reset index id for restore (#33648)
6ce8267
yiguolei pushed a commit that referenced this pull request Apr 17, 2024
@w41ter @Doris-Extras
[fix](restore) Reset index id for restore (#33648)
b44fed8
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@w41ter
Cherry-pick fix about restore indexes and ids (apache#33772)
9c8e821 
* [fix](restore) Reset index id for restore (apache#33648)

* [fix](restore) add indexes as part of table signature (apache#33650)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gavinchou gavinchou gavinchou approved these changes

@dataroaring dataroaring dataroaring approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.9-merged dev/3.0.0-merged p0_b reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@w41ter @doris-robot @gavinchou @dataroaring @xiaokang @wm1581066