[enhance](auth)Optimize the authentication logic of Ranger Doris #41207

zddr · 2024-09-24T07:24:31Z

Set the authentication type for row policy and datamask as select to avoid traversing all permission items within the ranger
The permission items on the ranger page are consistent with the grant syntax permission items on Doris
Add global permissions on the ranger side, and the ranger page needs to be specified in the input box*
No longer using cache to cache datamask and row policy, as changing the log level and specifying permission items has already made the speed fast enough
When using a ranger, there is no need to create a role with the same name within Doris
If you have sub level permissions, you can see the current level when showing. For example, if you have query permissions for table1 under db1, db1 will be displayed when showing databases

ranger ui: morningman/ranger#1

performance testing:

Each time the ranger authentication method is called, it takes less than 1ms,

If the user has global/db/table permissions, querying a large wide table takes approximately 1ms.

If the user only has partial column query permission, it will take approximately 80ms to query 50 columns

doris-robot · 2024-09-24T07:24:36Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

zddr · 2024-09-24T07:36:39Z

run buildall

zddr · 2024-09-24T07:52:48Z

run buildall

doris-robot · 2024-09-24T08:17:56Z

TPC-H: Total hot run time: 41256 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9d8feaf783ab51149e3b66d6bf1052dd964c4ffd, data reload: false

------ Round 1 ----------------------------------
q1	18147	7510	7428	7428
q2	2423	170	153	153
q3	11115	1235	1191	1191
q4	10612	760	803	760
q5	8816	2925	2916	2916
q6	249	156	156	156
q7	982	635	601	601
q8	9420	1939	1982	1939
q9	6637	6417	6400	6400
q10	6984	2265	2312	2265
q11	442	242	259	242
q12	409	223	224	223
q13	17779	3029	2958	2958
q14	251	218	212	212
q15	575	528	523	523
q16	666	619	623	619
q17	985	613	539	539
q18	7144	6684	6738	6684
q19	1400	1037	937	937
q20	581	308	291	291
q21	3970	3315	3208	3208
q22	1113	1011	1025	1011
Total cold run time: 110700 ms
Total hot run time: 41256 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7258	7293	7239	7239
q2	336	236	236	236
q3	2981	2779	2789	2779
q4	1964	1729	1727	1727
q5	5444	5465	5539	5465
q6	229	142	140	140
q7	2125	1703	1762	1703
q8	3233	3410	3449	3410
q9	8619	8522	8518	8518
q10	3511	3459	3447	3447
q11	587	484	476	476
q12	796	569	578	569
q13	6073	3030	3000	3000
q14	292	262	259	259
q15	565	520	510	510
q16	729	658	657	657
q17	1797	1567	1560	1560
q18	7871	7457	7422	7422
q19	1668	1581	1355	1355
q20	2035	1833	1854	1833
q21	5355	5219	5135	5135
q22	1094	1008	1016	1008
Total cold run time: 64562 ms
Total hot run time: 58448 ms

doris-robot · 2024-09-24T08:30:06Z

TPC-DS: Total hot run time: 191880 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9d8feaf783ab51149e3b66d6bf1052dd964c4ffd, data reload: false

query1	985	373	389	373
query2	6522	2228	2043	2043
query3	6697	212	227	212
query4	34736	23572	23497	23497
query5	4379	490	459	459
query6	275	168	159	159
query7	4617	310	305	305
query8	289	236	231	231
query9	9910	2725	2717	2717
query10	469	309	303	303
query11	18183	15122	15435	15122
query12	172	101	97	97
query13	1625	426	407	407
query14	10863	7532	7676	7532
query15	325	177	187	177
query16	8028	490	468	468
query17	1780	574	565	565
query18	2148	316	319	316
query19	372	153	151	151
query20	119	107	110	107
query21	212	105	108	105
query22	4624	4371	4160	4160
query23	34974	34001	33978	33978
query24	11183	2872	2880	2872
query25	673	414	425	414
query26	1522	165	166	165
query27	2786	298	306	298
query28	8248	2458	2448	2448
query29	960	444	440	440
query30	328	163	154	154
query31	1076	824	829	824
query32	102	59	59	59
query33	791	317	306	306
query34	925	493	511	493
query35	891	746	745	745
query36	1103	936	944	936
query37	163	88	86	86
query38	3976	3920	3991	3920
query39	1479	1418	1417	1417
query40	295	102	100	100
query41	50	55	50	50
query42	118	99	99	99
query43	549	491	515	491
query44	1251	837	815	815
query45	203	171	174	171
query46	1150	720	721	720
query47	1926	1790	1835	1790
query48	477	374	407	374
query49	1240	419	431	419
query50	831	425	418	418
query51	7095	6941	6987	6941
query52	99	91	89	89
query53	265	190	188	188
query54	1376	489	475	475
query55	77	76	82	76
query56	304	270	271	270
query57	1249	1099	1097	1097
query58	264	261	252	252
query59	3351	3191	3073	3073
query60	320	293	291	291
query61	137	103	104	103
query62	863	681	687	681
query63	230	193	188	188
query64	5086	648	634	634
query65	3297	3224	3198	3198
query66	1438	314	307	307
query67	15964	15400	15844	15400
query68	5029	577	585	577
query69	507	314	306	306
query70	1206	1080	1106	1080
query71	349	286	290	286
query72	7055	4067	3937	3937
query73	777	341	352	341
query74	9611	9023	8926	8926
query75	3649	2654	2627	2627
query76	2970	915	945	915
query77	531	314	300	300
query78	10172	9263	9194	9194
query79	1892	606	622	606
query80	1160	451	452	451
query81	585	248	250	248
query82	900	144	141	141
query83	248	146	139	139
query84	256	82	79	79
query85	1542	292	276	276
query86	465	305	304	304
query87	4460	4329	4404	4329
query88	3329	2443	2370	2370
query89	414	301	287	287
query90	2163	193	193	193
query91	180	144	144	144
query92	71	50	54	50
query93	1158	564	548	548
query94	1217	301	309	301
query95	365	256	267	256
query96	608	288	282	282
query97	3297	3176	3125	3125
query98	228	193	218	193
query99	1752	1317	1298	1298
Total cold run time: 306161 ms
Total hot run time: 191880 ms

doris-robot · 2024-09-24T08:35:36Z

ClickBench: Total hot run time: 32.92 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9d8feaf783ab51149e3b66d6bf1052dd964c4ffd, data reload: false

query1	0.05	0.05	0.04
query2	0.06	0.03	0.03
query3	0.23	0.07	0.07
query4	1.64	0.10	0.10
query5	0.50	0.49	0.51
query6	1.12	0.73	0.72
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.55	0.50	0.48
query10	0.56	0.58	0.55
query11	0.14	0.11	0.11
query12	0.14	0.10	0.11
query13	0.61	0.62	0.60
query14	3.10	2.98	3.06
query15	0.91	0.84	0.84
query16	0.38	0.38	0.39
query17	1.07	1.05	1.07
query18	0.24	0.21	0.23
query19	1.91	1.84	1.99
query20	0.01	0.01	0.01
query21	15.38	0.62	0.59
query22	2.73	2.37	1.65
query23	17.22	0.80	1.03
query24	2.85	1.20	1.04
query25	0.24	0.11	0.07
query26	0.47	0.14	0.13
query27	0.05	0.05	0.04
query28	10.56	1.09	1.07
query29	12.56	3.22	3.24
query30	0.25	0.07	0.06
query31	2.89	0.39	0.38
query32	3.27	0.47	0.46
query33	2.97	3.02	3.03
query34	16.98	4.48	4.53
query35	4.49	4.50	4.52
query36	0.67	0.48	0.48
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.15	0.12	0.12
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.02	0.02
Total cold run time: 107.31 s
Total hot run time: 32.92 s

morningman · 2024-10-08T08:07:34Z

.../main/java/org/apache/doris/catalog/authorizer/ranger/doris/RangerDorisAccessController.java


 public class RangerDorisAccessController extends RangerAccessController {
    private static final Logger LOG = LogManager.getLogger(RangerDorisAccessController.class);
+    // ranger must set name, we agreed that this name must be used


What does mean "we agreed"?

ranger ui上必须填一个name，但是全局的权限其实不需要name，因此需要约定一个name，然后校验全局权限的时候拿这个name校验

.../java/org/apache/doris/catalog/authorizer/ranger/hive/RangerHiveAccessControllerFactory.java

fe/fe-core/src/main/java/org/apache/doris/catalog/authorizer/ranger/RangerAccessController.java

zddr · 2024-10-08T09:46:25Z

run buildall

doris-robot · 2024-10-08T10:25:26Z

TPC-H: Total hot run time: 41207 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ea7e63ae2e8ee494eb0b43357c1cb7f0a7813b6d, data reload: false

------ Round 1 ----------------------------------
q1	17611	7350	7300	7300
q2	2016	294	271	271
q3	12128	1058	1147	1058
q4	10587	763	793	763
q5	7734	2874	2862	2862
q6	234	147	146	146
q7	1010	639	603	603
q8	9359	1973	1951	1951
q9	6583	6416	6405	6405
q10	6933	2286	2296	2286
q11	441	245	251	245
q12	405	221	228	221
q13	17797	2994	2996	2994
q14	246	211	211	211
q15	562	518	525	518
q16	647	579	575	575
q17	963	553	585	553
q18	7104	6753	6808	6753
q19	1326	942	988	942
q20	486	203	203	203
q21	4011	3348	3351	3348
q22	1091	1000	999	999
Total cold run time: 109274 ms
Total hot run time: 41207 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7284	7264	7242	7242
q2	329	227	236	227
q3	3079	2928	2930	2928
q4	2098	1863	1798	1798
q5	5762	5775	5733	5733
q6	239	148	144	144
q7	2266	1924	1845	1845
q8	3397	3609	3448	3448
q9	9150	9103	9038	9038
q10	3687	3596	3562	3562
q11	620	504	498	498
q12	832	598	612	598
q13	11846	3230	3235	3230
q14	300	276	273	273
q15	584	532	542	532
q16	685	680	658	658
q17	1853	1626	1613	1613
q18	8184	7892	7416	7416
q19	1724	1634	1545	1545
q20	2100	1859	1859	1859
q21	5678	5469	5361	5361
q22	1137	1066	1087	1066
Total cold run time: 72834 ms
Total hot run time: 60614 ms

doris-robot · 2024-10-08T10:37:27Z

TPC-DS: Total hot run time: 192433 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ea7e63ae2e8ee494eb0b43357c1cb7f0a7813b6d, data reload: false

query1	901	394	396	394
query2	6274	2084	2053	2053
query3	8680	199	205	199
query4	34175	23422	23502	23422
query5	3474	479	468	468
query6	277	178	172	172
query7	4194	304	316	304
query8	286	226	224	224
query9	9667	2722	2686	2686
query10	434	279	289	279
query11	17963	15366	15194	15194
query12	155	98	99	98
query13	1592	479	459	459
query14	9656	7441	7487	7441
query15	283	188	178	178
query16	8026	506	458	458
query17	1652	605	597	597
query18	2143	329	328	328
query19	363	153	150	150
query20	123	118	110	110
query21	221	111	107	107
query22	4764	4675	4698	4675
query23	34728	34194	34199	34194
query24	10756	2916	2822	2822
query25	584	411	419	411
query26	718	157	166	157
query27	2020	310	299	299
query28	6589	2438	2405	2405
query29	751	440	425	425
query30	275	158	161	158
query31	1048	790	830	790
query32	100	55	63	55
query33	699	309	289	289
query34	913	513	497	497
query35	875	727	724	724
query36	1119	943	968	943
query37	143	85	103	85
query38	4098	3854	3885	3854
query39	1502	1444	1419	1419
query40	205	97	99	97
query41	47	44	45	44
query42	123	96	98	96
query43	535	505	499	499
query44	1183	806	806	806
query45	201	167	165	165
query46	1134	714	724	714
query47	1928	1830	1848	1830
query48	448	356	344	344
query49	908	421	404	404
query50	816	451	420	420
query51	7072	7045	6924	6924
query52	103	88	90	88
query53	258	180	178	178
query54	1275	489	483	483
query55	78	74	78	74
query56	284	266	250	250
query57	1261	1147	1158	1147
query58	235	239	237	237
query59	3212	3102	2945	2945
query60	289	269	262	262
query61	101	101	103	101
query62	848	657	669	657
query63	214	187	186	186
query64	4090	652	651	651
query65	3299	3166	3246	3166
query66	872	304	303	303
query67	15758	15543	15508	15508
query68	4677	551	565	551
query69	536	296	293	293
query70	1162	1150	1078	1078
query71	360	268	277	268
query72	7285	3942	3910	3910
query73	770	359	355	355
query74	10274	8921	9081	8921
query75	3418	2657	2700	2657
query76	3019	889	928	889
query77	619	299	299	299
query78	10429	9504	9518	9504
query79	1103	600	576	576
query80	2176	450	464	450
query81	581	242	245	242
query82	750	134	139	134
query83	285	141	135	135
query84	282	81	88	81
query85	1373	298	279	279
query86	389	300	301	300
query87	4566	4287	4371	4287
query88	3137	2467	2535	2467
query89	399	286	290	286
query90	2212	190	189	189
query91	152	106	106	106
query92	61	48	50	48
query93	1041	542	546	542
query94	1122	307	308	307
query95	365	259	258	258
query96	619	294	289	289
query97	3294	3165	3211	3165
query98	217	205	198	198
query99	1734	1331	1267	1267
Total cold run time: 297405 ms
Total hot run time: 192433 ms

doris-robot · 2024-10-08T10:42:55Z

ClickBench: Total hot run time: 32.04 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ea7e63ae2e8ee494eb0b43357c1cb7f0a7813b6d, data reload: false

query1	0.04	0.05	0.04
query2	0.06	0.02	0.03
query3	0.23	0.06	0.06
query4	1.64	0.10	0.10
query5	0.53	0.51	0.52
query6	1.12	0.72	0.72
query7	0.02	0.01	0.02
query8	0.04	0.04	0.03
query9	0.55	0.49	0.50
query10	0.54	0.57	0.54
query11	0.14	0.11	0.10
query12	0.14	0.11	0.10
query13	0.63	0.60	0.60
query14	2.85	2.83	2.74
query15	0.90	0.82	0.82
query16	0.39	0.39	0.37
query17	1.04	1.07	1.04
query18	0.20	0.19	0.20
query19	1.99	1.78	2.04
query20	0.01	0.01	0.01
query21	15.36	0.56	0.58
query22	2.59	2.72	1.87
query23	16.99	0.89	0.84
query24	2.99	0.38	0.98
query25	0.27	0.10	0.05
query26	0.37	0.14	0.14
query27	0.05	0.04	0.04
query28	11.48	1.07	1.07
query29	12.55	3.21	3.24
query30	0.25	0.06	0.06
query31	2.87	0.37	0.38
query32	3.27	0.48	0.46
query33	3.01	3.02	2.98
query34	16.94	4.46	4.49
query35	4.46	4.48	4.48
query36	0.68	0.52	0.50
query37	0.09	0.05	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.03
query40	0.15	0.13	0.13
query41	0.07	0.02	0.02
query42	0.03	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.64 s
Total hot run time: 32.04 s

morningman

LGTM

github-actions · 2024-10-11T03:49:19Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-10-11T03:49:20Z

PR approved by anyone and no changes requested.

…1207) (#41840) pick from master #41207

doris pr: apache/doris#41207

…) (#46179) pick: #41207 only pick Performance optimization related

zddr added 21 commits August 26, 2024 17:08

1

13f1619

1

e90a641

1

dec7166

1

63f14c7

1

8b7ea3a

1

4d75b01

1

b0a465c

1

67ed015

1

d205ff7

1

d5ce13e

1

607b8a7

1

7d63197

Merge branch 'master' into ranger_enhance1

57259b5

1

1794ddd

1

8dd362a

1

5ae7a6a

1

dc62e0b

1

5566d34

Merge branch 'master' into ranger_show

b00b24e

1

ad89eb5

1

3703dd7

1

0c667fb

1

9d8feaf

morningman reviewed Oct 8, 2024

View reviewed changes

morningman added the kind/behavior-changed label Oct 8, 2024

comment

ea7e63a

morrySnow added the dev/3.0.x label Oct 10, 2024

morningman approved these changes Oct 11, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 11, 2024

github-actions bot added the reviewed label Oct 11, 2024

morrySnow approved these changes Oct 11, 2024

View reviewed changes

morrySnow merged commit 6ed0bc8 into apache:master Oct 11, 2024

zddr mentioned this pull request Oct 15, 2024

[enhance](auth)Optimize the authentication logic of Ranger Doris #41840

Merged

morrySnow pushed a commit that referenced this pull request Nov 1, 2024

[enhance](auth) Optimize the authentication logic of Ranger Doris (#4…

d27c1bb

…1207) (#41840) pick from master #41207

morrySnow added dev/3.0.3-merged and removed dev/3.0.x labels Nov 1, 2024

zddr mentioned this pull request Dec 30, 2024

[enhance](auth)Increase global permissions morningman/ranger#1

Merged

morningman pushed a commit to morningman/ranger that referenced this pull request Dec 30, 2024

[enhance](auth)Increase global permissions (#1)

519cfe7

doris pr: apache/doris#41207

zddr mentioned this pull request Dec 30, 2024

[enhance](auth)Optimize the authentication logic of Ranger Doris(#41207) #46179

Merged

yiguolei pushed a commit that referenced this pull request Dec 31, 2024

[enhance](auth)Optimize the authentication logic of Ranger Doris(#41207…

fa20529

…) (#46179) pick: #41207 only pick Performance optimization related

yiguolei added the dev/2.1.8-merged label Dec 31, 2024

yiguolei mentioned this pull request Jan 19, 2025

Release Note 2.1.8 #47198

Closed

[enhance](auth)Optimize the authentication logic of Ranger Doris #41207

[enhance](auth)Optimize the authentication logic of Ranger Doris #41207

Uh oh!

Conversation

zddr commented Sep 24, 2024 • edited by morrySnow Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

doris-robot commented Sep 24, 2024

Uh oh!

zddr commented Sep 24, 2024

Uh oh!

zddr commented Sep 24, 2024

Uh oh!

doris-robot commented Sep 24, 2024

Uh oh!

doris-robot commented Sep 24, 2024

Uh oh!

doris-robot commented Sep 24, 2024

Uh oh!

morningman Oct 8, 2024

Choose a reason for hiding this comment

Uh oh!

zddr Oct 8, 2024

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

zddr commented Oct 8, 2024

Uh oh!

doris-robot commented Oct 8, 2024

Uh oh!

doris-robot commented Oct 8, 2024

Uh oh!

doris-robot commented Oct 8, 2024

Uh oh!

morningman left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Oct 11, 2024

Uh oh!

github-actions bot commented Oct 11, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

zddr commented Sep 24, 2024 •

edited by morrySnow

Loading