Skip to content

Conversation

@WinkerDu
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

Doris FE firstly launch with no root password for now. In my organization scene, we use Doris operator start a Doris cluster, then set password for root user via peripheral system, which increase system Complexity.

This pr introduces FE config initial_root_password to set root user initial 2-staged SHA-1 encrypted password, default as '', means no root password. Subsequent 'set password' operations for root user will overwrite the initial root password.

if initial_root_password is not a valid 2-staged SHA-1 encrypted password, no root password set as before.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@WinkerDu WinkerDu force-pushed the master-initial-root-pswd branch from 0509124 to da17119 Compare November 22, 2023 17:06
@WinkerDu
Copy link
Contributor Author

run buildall

@WinkerDu WinkerDu force-pushed the master-initial-root-pswd branch from da17119 to 2216cee Compare November 22, 2023 18:21
@WinkerDu
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 43.72 seconds
stream load tsv: 567 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 33 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 28.5 seconds inserted 10000000 Rows, about 350K ops/s
storage size: 17097849902 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit 2216ceeaeb422e259fbc52c17857b538d78cbbac, data reload: true

run tpch-sf100 query with default conf and session variables
q1	4840	4610	4620	4610
q2	386	122	124	122
q3	2072	1914	1895	1895
q4	1811	1270	1258	1258
q5	4085	3964	3976	3964
q6	245	121	125	121
q7	1395	873	882	873
q8	3818	2811	2752	2752
q9	9789	9641	9491	9491
q10	3456	3539	3535	3535
q11	404	255	243	243
q12	444	288	280	280
q13	18443	3765	3819	3765
q14	312	288	292	288
q15	593	517	538	517
q16	654	576	577	576
q17	1142	969	943	943
q18	7834	7420	7416	7416
q19	1703	1701	1681	1681
q20	577	311	297	297
q21	4441	3973	4007	3973
q22	461	370	383	370
Total cold run time: 68905 ms
Total hot run time: 48970 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4584	4577	4568	4568
q2	327	237	248	237
q3	4015	4005	4011	4005
q4	2695	2693	2694	2693
q5	9558	9589	9506	9506
q6	242	118	122	118
q7	3058	2513	2486	2486
q8	4427	4448	4460	4448
q9	12856	12897	13283	12897
q10	4093	4172	4174	4172
q11	790	653	645	645
q12	975	814	819	814
q13	4282	3560	3553	3553
q14	376	337	341	337
q15	560	521	530	521
q16	734	683	679	679
q17	3872	3856	3977	3856
q18	9536	8902	8895	8895
q19	1824	1795	1796	1795
q20	2379	2062	2070	2062
q21	8848	8553	8462	8462
q22	922	794	814	794
Total cold run time: 80953 ms
Total hot run time: 77543 ms

@WinkerDu
Copy link
Contributor Author

cc @morningman @chenlinzhong @luozenglin Please have a review, thx

@WinkerDu WinkerDu force-pushed the master-initial-root-pswd branch from 2216cee to f5fc04d Compare November 23, 2023 18:06
@WinkerDu
Copy link
Contributor Author

run buildall

@WinkerDu WinkerDu force-pushed the master-initial-root-pswd branch from f5fc04d to b646ce1 Compare November 23, 2023 18:09
@WinkerDu
Copy link
Contributor Author

run buildall

@WinkerDu
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit adc49342176c1acd979e46301d4cad0ec9285370, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4906	4696	4655	4655
q2	374	172	158	158
q3	2014	1953	1928	1928
q4	1373	1247	1193	1193
q5	3942	3919	4045	3919
q6	250	132	131	131
q7	1412	878	883	878
q8	2788	2800	2782	2782
q9	50392	13103	9426	9426
q10	13153	3561	3525	3525
q11	397	257	254	254
q12	1326	302	289	289
q13	21871	3817	3793	3793
q14	319	281	294	281
q15	590	534	522	522
q16	670	586	574	574
q17	1147	970	961	961
q18	8987	7342	7365	7342
q19	3945	1643	1651	1643
q20	574	290	283	283
q21	7590	4050	3979	3979
q22	483	378	369	369
Total cold run time: 128503 ms
Total hot run time: 48885 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4644	4598	4598	4598
q2	350	235	261	235
q3	4018	3994	3999	3994
q4	2703	2692	2688	2688
q5	9602	9534	9615	9534
q6	243	124	124	124
q7	3009	2457	2481	2457
q8	4436	4435	4420	4420
q9	12938	12707	12846	12707
q10	4110	4170	4167	4167
q11	799	640	654	640
q12	958	803	811	803
q13	4397	3582	3544	3544
q14	387	343	340	340
q15	578	519	540	519
q16	732	664	657	657
q17	3812	3881	3911	3881
q18	9624	9041	9012	9012
q19	1819	1778	1772	1772
q20	2383	2093	2061	2061
q21	8847	8492	8716	8492
q22	941	796	815	796
Total cold run time: 81330 ms
Total hot run time: 77441 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.67 seconds
stream load tsv: 568 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 32.2 seconds inserted 10000000 Rows, about 310K ops/s
storage size: 17099429674 Bytes

luozenglin
luozenglin previously approved these changes Nov 24, 2023
Copy link
Contributor

@luozenglin luozenglin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 24, 2023
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@luozenglin
Copy link
Contributor

Maybe documentation should be provided on how to convert a plaintext password to initial_root_password.

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Nov 24, 2023
@WinkerDu
Copy link
Contributor Author

run buildall

@WinkerDu
Copy link
Contributor Author

Maybe documentation should be provided on how to convert a plaintext password to initial_root_password.

@luozenglin new commit is pushed, PTAL

luozenglin
luozenglin previously approved these changes Nov 24, 2023
Copy link
Contributor

@luozenglin luozenglin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 24, 2023
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

chenlinzhong
chenlinzhong previously approved these changes Nov 24, 2023
Copy link
Contributor

@chenlinzhong chenlinzhong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.68 seconds
stream load tsv: 582 seconds loaded 74807831229 Bytes, about 122 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17098478239 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit 8602c6d9c863d10627ec0aebd78092a88527f994, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4941	4651	4677	4651
q2	364	152	162	152
q3	2036	1848	1920	1848
q4	1396	1250	1240	1240
q5	3946	3939	4017	3939
q6	252	131	131	131
q7	1440	878	872	872
q8	2803	2796	2775	2775
q9	9789	9485	9699	9485
q10	3446	3529	3531	3529
q11	400	249	247	247
q12	432	291	294	291
q13	4563	3798	3819	3798
q14	312	279	287	279
q15	578	538	512	512
q16	666	582	590	582
q17	1135	959	943	943
q18	7888	7618	7568	7568
q19	1678	1676	1693	1676
q20	574	295	279	279
q21	4410	4016	4030	4016
q22	478	369	383	369
Total cold run time: 53527 ms
Total hot run time: 49182 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4623	4568	4557	4557
q2	333	223	239	223
q3	4039	4019	4020	4019
q4	2733	2725	2719	2719
q5	9606	9612	9562	9562
q6	248	122	125	122
q7	3040	2488	2515	2488
q8	4434	4449	4477	4449
q9	12902	12821	12765	12765
q10	4025	4102	4174	4102
q11	747	653	659	653
q12	980	818	814	814
q13	4282	3538	3509	3509
q14	401	351	354	351
q15	565	518	507	507
q16	744	686	684	684
q17	3892	3935	3908	3908
q18	9708	9194	9201	9194
q19	1835	1811	1772	1772
q20	2433	2053	2042	2042
q21	8861	8853	8630	8630
q22	869	821	819	819
Total cold run time: 81300 ms
Total hot run time: 77889 ms

Default: empty string
s it possible to dynamically configure: false
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
s it possible to dynamically configure: false
Is it possible to dynamically configure: false

public static double diagnose_balance_max_tablet_num_ratio = 1.1;

@ConfField(masterOnly = true, description = {
"设置 root 用户初始化2阶段 SHA-1 加密密码,默认为'',即不设置 root 密码。"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

可以多加一句,说明下如何获取 SHA-1 加密后的密码值

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已加

@WinkerDu WinkerDu dismissed stale reviews from chenlinzhong and luozenglin via 9c1bddf November 26, 2023 15:54
@WinkerDu
Copy link
Contributor Author

run buildall

@WinkerDu
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit a17145d3c54f87ea284ff0ce069e9555f494b4c1, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4931	4638	4621	4621
q2	358	158	158	158
q3	2046	1876	1889	1876
q4	1398	1262	1244	1244
q5	3954	3928	4007	3928
q6	253	129	130	129
q7	1424	877	864	864
q8	2790	2785	2793	2785
q9	9693	9465	9428	9428
q10	3455	3504	3494	3494
q11	382	250	255	250
q12	436	285	291	285
q13	4565	3811	3830	3811
q14	314	283	280	280
q15	583	519	534	519
q16	656	580	588	580
q17	1139	947	911	911
q18	7850	7577	7573	7573
q19	1661	1679	1678	1678
q20	571	308	276	276
q21	4436	3986	4023	3986
q22	480	368	378	368
Total cold run time: 53375 ms
Total hot run time: 49044 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4602	4564	4557	4557
q2	338	239	260	239
q3	4045	4021	4017	4017
q4	2728	2708	2708	2708
q5	9711	9668	9649	9649
q6	246	123	117	117
q7	3039	2491	2536	2491
q8	4409	4484	4483	4483
q9	12853	12745	12817	12745
q10	4053	4181	4152	4152
q11	783	706	657	657
q12	965	793	796	793
q13	4284	3533	3563	3533
q14	387	340	358	340
q15	568	526	514	514
q16	718	653	673	653
q17	3857	3980	3813	3813
q18	9584	9078	9200	9078
q19	1805	1804	1788	1788
q20	2390	2056	2098	2056
q21	8839	8410	8428	8410
q22	933	827	803	803
Total cold run time: 81137 ms
Total hot run time: 77596 ms

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.77 seconds
stream load tsv: 564 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 27 seconds loaded 2358488459 Bytes, about 83 MB/s
stream load orc: 72 seconds loaded 1101869774 Bytes, about 14 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.7 seconds inserted 10000000 Rows, about 348K ops/s
storage size: 17099336164 Bytes

@WinkerDu
Copy link
Contributor Author

run feut

Copy link
Contributor

@luozenglin luozenglin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@zxealous zxealous left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@luozenglin luozenglin merged commit 9aafcf2 into apache:master Nov 27, 2023
WinkerDu added a commit to WinkerDu/doris that referenced this pull request Nov 27, 2023
seawinde pushed a commit to seawinde/doris that referenced this pull request Nov 28, 2023
WinkerDu added a commit to WinkerDu/doris that referenced this pull request Nov 29, 2023
eldenmoon pushed a commit to eldenmoon/incubator-doris that referenced this pull request Dec 3, 2023
eldenmoon pushed a commit to eldenmoon/incubator-doris that referenced this pull request Dec 3, 2023
eldenmoon added a commit that referenced this pull request Dec 4, 2023
* [chore](case) Use correct insert stmt for cold heat separation case #27546 (#27585)

Co-authored-by: AlexYue <yj976240184@gmail.com>

* [enhance](S3) Print the error detail for every s3 operation (#27572) (#27615)

* [nereids] fix stats error when using dateTime type filter #27571 (#27577)

* [fix](planner)sort node should materialized required slots for itself #27605 (#27620)

* [fix](Nereids) non-deterministic expression should not be constant (#27606) (#27631)

* [enhancement](stats) Add process for aggstate type #27640 (#27642)

* [Fix](statistics)Fix bug and improve auto analyze. (#27626) (#27657)

1. Implement needReAnalyzeTable for ExternalTable. For now, external table will not be reanalyzed in 10 days.
2. For HiveMetastoreCache.loadPartitions, handle the empty iterator case to avoid Index out of boundary exception.
3. Wrap handle show analyze loop with try catch, so that when one table failed (for example, catalog dropped so the table couldn't be found anymore), we can still show the other tables.
4. For now, only OlapTable and Hive HMSExternalTable support sample analyze, throw exception for other types of table.
5. In StatisticsCollector, call constructJob after createTableLevelTaskForExternalTable to avoid NPE.

* [profile](bugfix) should not cache profile content because the profile may not be a full profile (#27635)

---------

Co-authored-by: yiguolei <yiguolei@gmail.com>

* [Enhance](fe) Support setting initial root password when FE firstly launch (#27438) (#27603)

* [opt](plan) only lock olap table when query plan #27639 (#27656)

bp #27639

* select coordinator node from user's tag when exec streaming load (#27106) (#27677)

* [fix](statistics)Need to recalculate health value when table row count become 0  #27673 (#27674)

backport #27673

* [fix](statistics)Fix sample min max npe bug  #27702 (#27707)

backport #27702

* [Bug](join) try fix wrong _has_null_in_build_side setted (#27684) (#27710)

* [Fix](show-load)Show load npe(userinfo is null) (#27698) (#27719)

* [pick](nereids)temporary partition is always pruned #27636 (#27722)

* [enhancement](stats) limit bq cap size for analyze task #27685 (#27687)

* [improvement](statistics) Add config for the threshold of column count for auto analyze #27713 (#27723)

* [doc](fix) k8s operator docs fix to 2.0 (#27476)

* [Improvement](planner)support select tablets with nereids optimize #23164 #23365 (#27740)

#23164
#23365

* [FIX](complextype)fix complex type hash equals (#27743)

* [fix](statistics) Fix show auto analyze missing jobs bug (#27761)

* [bugfix](topn) fix coredump in copy_column_data_to_block when nullable mismatch

return RuntimeError if copy_column_data_to_block nullable mismatch to avoid coredump in input_col_ptr->filter_by_selector(sel_rowid_idx, select_size, raw_res_ptr) .

The problem is reported by a doris user but I can not reproduce it, so there is no testcase added currently.

* [opt](stats) Use escape rather than base64 for min/max value #27746 (#27748)

* [refactor](http) disable snapshot and get_log_file api (#27724) (#27770)

* [branch-2.0](pick 27738) Warning log to trace send fragment #27738 (#27760)

* [branch-2.0](pick #27771) Add more detail msg for waitRPC exception (#27773)

* [Bug](pipeline) prevent PipelineFragmentContext destruct early (#27790)

* [deps](compression) Opt gzip decompress by libdeflate on X86 and X86_64 platforms: 1. Add libdeflate lib.  (#27542) (#27711)

Backport from #27542.

* [FIX](case)fix case truncate table first #27792

* [doc](stats) add auto_analyze_table_width_threshold description. (#27818) (#27832)

* [fix](bdbje) Fix bdbje logging level not work (#27597) (#27788)

* `EnvironmentConfig.FILE_LOGGING_LEVEL` only set FileHandlerLevel, we should
   set logger level firstly, otherwise it will not take effect.

* [Opt](compression) Opt gzip decompress by libdeflate on X86 and X86_64 platforms: 2. Opt gzip decompression by libdeflate lib. (#27669) (#27801)

Backport from #27669.

* [branch-2.0](fix) Fix broken exception message #27836

* [Bug](func) coredump in equal for null in function (#27843)

* [minor](stats) Update olap table row count after analyze (#27858)

pick from master #27814

* [fix](stats)min and max return NaN when table is empty (#27863)

fix analyze empty table and min/max null value bug:
1. Skip empty analyze task for sample analyze task. (Full analyze task already skipped).
2. Check sample rows is not 0 before calculate the scale factor.
3. Remove ' in sql template after remove base64 encoding for min/max value.

backport #27862

* [minor](stats) Throw error when sync analyze failed (#27846)

pick from master #27845

* [fix](stats) Don't save colToPartitions anymore to save mem (#27880)

pick from master #27879

* [fix](nereids) set operation's result type is wrong if decimal overflows (#27872)

pick from master #27870

* [Config] Modify the default value of tablet_schema_cache_recycle_interval (#27877)

* [fix](like_func) incorrect result of like with 'NO_BACKSLASH_ESCAPES' mode(#27842) (#27851)

* [fix](fe) Fix show frontends npt in some situations (#27295) (#27789)

```
java.lang.NullPointerException: null
    at com.sleepycat.je.rep.util.ReplicationGroupAdmin.getMasterSocket(ReplicationGroupAdmin.java:191)
    at com.sleepycat.je.rep.util.ReplicationGroupAdmin.doMessageExchange(ReplicationGroupAdmin.java:607)
    at com.sleepycat.je.rep.util.ReplicationGroupAdmin.getGroup(ReplicationGroupAdmin.java:406)
    at org.apache.doris.ha.BDBHA.getElectableNodes(BDBHA.java:132)
    at org.apache.doris.common.proc.FrontendsProcNode.getFrontendsInfo(FrontendsProcNode.java:84)
    at org.apache.doris.qe.ShowExecutor.handleShowFrontends(ShowExecutor.java:1923)
    at org.apache.doris.qe.ShowExecutor.execute(ShowExecutor.java:355)
    at org.apache.doris.qe.StmtExecutor.handleShow(StmtExecutor.java:2113)
    ...
```

* [branch-2.0](fix) Fix extremely high CPU usage caused by rf merge #27894 (#27895)

* [fix](stacktrace) ignore stacktrace for error code INVALID_ARGUMENT INVERTED_INDEX_NOT_IMPLEMENTED (#27898)

* ignore stacktrace for error INVALID_ARGUMENT INVERTED_INDEX_NOT_IMPLEMENTED

* AndBlockColumnPredicate::evaluate

* [opt](nereids) Branch-2.0: remove partition & histogram from col stats to reduce memory usage #27885 (#27896)

* [pick](Nereids) temporary partition is selected only if user manually specified: Branch-2.0 #27893 (#27905)

* [fix](multi-catalog)support the max compute partition prune (#27154) (#27902)

backport #27154

* [fix](Nereids) should not push down project to the nullable side of outer join #27912 (#27913)

* fix compile

---------

Co-authored-by: Dongyang Li <hello_stephen@qq.com>
Co-authored-by: AlexYue <yj976240184@gmail.com>
Co-authored-by: xzj7019 <131111794+xzj7019@users.noreply.github.com>
Co-authored-by: starocean999 <40539150+starocean999@users.noreply.github.com>
Co-authored-by: morrySnow <101034200+morrySnow@users.noreply.github.com>
Co-authored-by: AKIRA <33112463+Kikyou1997@users.noreply.github.com>
Co-authored-by: Jibing-Li <64681310+Jibing-Li@users.noreply.github.com>
Co-authored-by: yiguolei <676222867@qq.com>
Co-authored-by: yiguolei <yiguolei@gmail.com>
Co-authored-by: DuRipeng <453243496@qq.com>
Co-authored-by: Mingyu Chen <morningman@163.com>
Co-authored-by: wangbo <wangbo@apache.org>
Co-authored-by: Pxl <pxl290@qq.com>
Co-authored-by: Calvin Kirs <acm_master@163.com>
Co-authored-by: minghong <englefly@gmail.com>
Co-authored-by: catpineapple <42031973+catpineapple@users.noreply.github.com>
Co-authored-by: amory <wangqiannan@selectdb.com>
Co-authored-by: Kang <kxiao.tiger@gmail.com>
Co-authored-by: zhiqiang <seuhezhiqiang@163.com>
Co-authored-by: Qi Chen <kaka11.chen@gmail.com>
Co-authored-by: Lei Zhang <27994433+SWJTU-ZhangLei@users.noreply.github.com>
Co-authored-by: HappenLee <happenlee@hotmail.com>
Co-authored-by: Lightman <31928846+Lchangliang@users.noreply.github.com>
Co-authored-by: Jerry Hu <mrhhsg@gmail.com>
Co-authored-by: slothever <18522955+wsjz@users.noreply.github.com>
gnehil pushed a commit to gnehil/doris that referenced this pull request Dec 4, 2023
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants