Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](kerberos)fix and refactor ugi login for kerberos and simple authentication #37301

Merged
merged 3 commits into from
Jul 18, 2024
Merged

[fix](kerberos)fix and refactor ugi login for kerberos and simple authentication #37301

merged 3 commits into from
Jul 18, 2024

Conversation

wsjz
Copy link
Contributor

@wsjz wsjz commented Jul 4, 2024 •
edited
Loading

Proposed changes

optimize kerberos ugi login:

  1. support authentication framework for external table
  2. cache ugi to avoid conflicts when set configuration for ugi login
  3. do ugi login just on creating catalog, which reduces the ugi create times
  4. only simple authentication will use getloginUser, which avoids conflicts between simple and kerberos authentication

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@wsjz wsjz force-pushed the ugi_opt branch 2 times, most recently from 5606297 to 8db5881 Compare July 11, 2024 19:27
@wsjz wsjz marked this pull request as ready for review July 11, 2024 19:28
@wsjz wsjz changed the title [fix](kerberos)fix ugi login [fix](kerberos)fix and refactor ugi login for kerberos and simple authentication Jul 11, 2024
@wsjz
Copy link
Contributor Author

wsjz commented Jul 11, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40069 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8db58812f172de7f61940bab29b9e802c565a2ad, data reload: false

------ Round 1 ----------------------------------
q1	17628	4362	4351	4351
q2	2023	204	202	202
q3	10426	1224	1098	1098
q4	10174	850	882	850
q5	7546	2699	2666	2666
q6	231	140	134	134
q7	966	597	602	597
q8	9220	2098	2101	2098
q9	8734	6569	6573	6569
q10	8825	3784	3743	3743
q11	449	253	237	237
q12	472	237	237	237
q13	17760	2975	2988	2975
q14	279	221	231	221
q15	538	493	497	493
q16	512	378	389	378
q17	987	648	701	648
q18	8047	7503	7295	7295
q19	7750	1442	1471	1442
q20	702	335	318	318
q21	4969	3168	3222	3168
q22	401	350	349	349
Total cold run time: 118639 ms
Total hot run time: 40069 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4394	4309	4279	4279
q2	372	282	272	272
q3	3024	2962	2975	2962
q4	1998	1707	1734	1707
q5	5551	5547	5415	5415
q6	234	139	139	139
q7	2246	1851	1844	1844
q8	3298	3403	3408	3403
q9	8699	8858	8745	8745
q10	4186	3784	3888	3784
q11	599	497	492	492
q12	822	664	627	627
q13	16085	3199	3153	3153
q14	323	293	299	293
q15	534	497	475	475
q16	513	455	438	438
q17	1831	1550	1508	1508
q18	8112	7961	7807	7807
q19	1779	1667	1628	1628
q20	2198	1912	1898	1898
q21	10083	4848	4923	4848
q22	642	563	567	563
Total cold run time: 77523 ms
Total hot run time: 56280 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174031 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8db58812f172de7f61940bab29b9e802c565a2ad, data reload: false

query1	909	376	370	370
query2	6441	2437	2446	2437
query3	6641	216	215	215
query4	28148	17397	17356	17356
query5	3675	490	469	469
query6	261	161	161	161
query7	4577	292	290	290
query8	345	316	296	296
query9	8576	2453	2452	2452
query10	426	282	270	270
query11	10629	9918	9901	9901
query12	118	81	86	81
query13	1628	369	356	356
query14	10135	7695	7002	7002
query15	246	182	185	182
query16	7329	325	325	325
query17	1350	544	534	534
query18	1896	274	268	268
query19	196	150	145	145
query20	89	84	82	82
query21	201	133	127	127
query22	4269	4028	3946	3946
query23	34301	33747	33586	33586
query24	11002	2948	2858	2858
query25	593	400	392	392
query26	703	152	155	152
query27	2234	272	280	272
query28	6040	2149	2137	2137
query29	898	620	637	620
query30	244	148	151	148
query31	1002	782	761	761
query32	102	55	58	55
query33	652	300	319	300
query34	889	497	496	496
query35	705	594	583	583
query36	1160	953	977	953
query37	152	82	90	82
query38	2921	2930	2844	2844
query39	884	850	825	825
query40	213	123	120	120
query41	55	53	51	51
query42	114	96	106	96
query43	572	537	573	537
query44	1081	718	731	718
query45	201	165	163	163
query46	1098	732	735	732
query47	1823	1743	1783	1743
query48	364	289	293	289
query49	814	403	421	403
query50	779	387	392	387
query51	6810	6839	6814	6814
query52	106	97	97	97
query53	361	324	295	295
query54	858	447	463	447
query55	76	73	72	72
query56	276	259	262	259
query57	1123	1032	1022	1022
query58	241	245	242	242
query59	3539	3284	3316	3284
query60	315	268	273	268
query61	96	93	93	93
query62	768	631	657	631
query63	318	294	290	290
query64	9124	2265	1651	1651
query65	3187	3132	3088	3088
query66	774	338	326	326
query67	15449	14977	14761	14761
query68	4514	531	529	529
query69	526	406	368	368
query70	1133	1105	1146	1105
query71	388	294	279	279
query72	7093	5785	6018	5785
query73	747	331	330	330
query74	5917	5542	5434	5434
query75	3385	2698	2665	2665
query76	2124	925	852	852
query77	429	309	305	305
query78	9621	9142	8966	8966
query79	2899	526	519	519
query80	2187	483	483	483
query81	587	223	222	222
query82	729	139	137	137
query83	277	170	164	164
query84	275	98	92	92
query85	1835	305	297	297
query86	481	303	306	303
query87	3313	3090	3069	3069
query88	4190	2463	2452	2452
query89	495	368	366	366
query90	1795	194	185	185
query91	131	103	104	103
query92	56	52	49	49
query93	4924	509	496	496
query94	1196	212	207	207
query95	405	326	321	321
query96	608	282	277	277
query97	3217	3014	3020	3014
query98	223	200	195	195
query99	1581	1254	1268	1254
Total cold run time: 280383 ms
Total hot run time: 174031 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.73 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8db58812f172de7f61940bab29b9e802c565a2ad, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.07
query5	0.49	0.49	0.50
query6	1.14	0.74	0.72
query7	0.02	0.01	0.01
query8	0.06	0.05	0.04
query9	0.57	0.49	0.51
query10	0.55	0.54	0.55
query11	0.15	0.12	0.11
query12	0.15	0.11	0.12
query13	0.62	0.58	0.60
query14	0.76	0.77	0.78
query15	0.86	0.80	0.81
query16	0.35	0.35	0.36
query17	0.98	0.96	1.03
query18	0.24	0.24	0.23
query19	1.93	1.74	1.83
query20	0.01	0.00	0.00
query21	15.40	0.78	0.65
query22	4.46	6.55	1.98
query23	18.27	1.32	1.32
query24	2.17	0.24	0.22
query25	0.15	0.09	0.08
query26	0.30	0.22	0.21
query27	0.46	0.23	0.23
query28	13.22	1.02	1.01
query29	12.66	3.27	3.29
query30	0.26	0.06	0.06
query31	2.87	0.40	0.40
query32	3.26	0.47	0.47
query33	2.99	2.87	2.87
query34	16.98	4.31	4.30
query35	4.43	4.43	4.41
query36	0.65	0.52	0.52
query37	0.18	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.04	0.04
query40	0.16	0.11	0.13
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.17 s
Total hot run time: 30.73 s

@wsjz wsjz force-pushed the ugi_opt branch 2 times, most recently from a0845e4 to 36e7cde Compare July 15, 2024 09:32
@wsjz
Copy link
Contributor Author

wsjz commented Jul 15, 2024

run buildall

@wsjz wsjz force-pushed the ugi_opt branch 2 times, most recently from e3bfb6c to 93b506f Compare July 15, 2024 10:35
@wsjz
Copy link
Contributor Author

wsjz commented Jul 15, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@wsjz
Copy link
Contributor Author

wsjz commented Jul 15, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39958 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e1c710d8db7261a31376d348433d9a56482a6a46, data reload: false

------ Round 1 ----------------------------------
q1	17621	4312	4245	4245
q2	2012	192	204	192
q3	10445	1172	1177	1172
q4	10201	754	922	754
q5	7586	2672	2653	2653
q6	224	136	136	136
q7	961	621	591	591
q8	9210	2067	2066	2066
q9	8611	6553	6535	6535
q10	8715	3785	3821	3785
q11	470	233	230	230
q12	419	222	223	222
q13	17786	2975	2989	2975
q14	274	240	232	232
q15	532	480	489	480
q16	491	374	374	374
q17	954	716	685	685
q18	7951	7463	7476	7463
q19	5791	1449	1362	1362
q20	666	305	319	305
q21	4852	3213	3306	3213
q22	347	290	288	288
Total cold run time: 116119 ms
Total hot run time: 39958 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4362	4255	4320	4255
q2	373	271	270	270
q3	3063	2884	2924	2884
q4	2038	1766	1699	1699
q5	5729	5569	5519	5519
q6	236	136	132	132
q7	2191	1813	1907	1813
q8	3248	3417	3401	3401
q9	8808	8708	8925	8708
q10	4111	4014	3792	3792
q11	587	487	495	487
q12	914	649	645	645
q13	16601	3144	3180	3144
q14	319	301	307	301
q15	517	493	486	486
q16	493	451	444	444
q17	1823	1504	1523	1504
q18	8085	7822	7923	7822
q19	1797	1678	1584	1584
q20	3033	1882	1900	1882
q21	10296	4795	4742	4742
q22	629	519	509	509
Total cold run time: 79253 ms
Total hot run time: 56023 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173112 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e1c710d8db7261a31376d348433d9a56482a6a46, data reload: false

query1	921	370	373	370
query2	6470	1944	1835	1835
query3	6638	203	215	203
query4	25961	17376	17211	17211
query5	3658	505	463	463
query6	267	164	164	164
query7	4583	283	278	278
query8	235	186	185	185
query9	8483	2383	2345	2345
query10	444	292	272	272
query11	11836	9957	10070	9957
query12	121	84	83	83
query13	1633	379	388	379
query14	10073	7736	7488	7488
query15	222	175	168	168
query16	7740	323	315	315
query17	1553	584	534	534
query18	1918	288	282	282
query19	204	152	153	152
query20	87	79	85	79
query21	216	127	126	126
query22	4293	3961	3973	3961
query23	33764	33938	33511	33511
query24	11005	2892	2969	2892
query25	621	412	453	412
query26	708	145	147	145
query27	2332	275	279	275
query28	6165	2026	2022	2022
query29	895	624	671	624
query30	260	149	150	149
query31	962	767	769	767
query32	99	53	59	53
query33	657	293	289	289
query34	899	487	497	487
query35	713	586	597	586
query36	1135	1014	986	986
query37	140	84	95	84
query38	2947	2874	2801	2801
query39	923	860	810	810
query40	208	119	115	115
query41	49	42	43	42
query42	118	93	102	93
query43	497	479	457	457
query44	1109	726	732	726
query45	196	170	163	163
query46	1086	738	742	738
query47	1849	1753	1787	1753
query48	369	290	295	290
query49	842	420	423	420
query50	779	384	384	384
query51	6989	6824	6785	6785
query52	113	95	92	92
query53	352	278	280	278
query54	873	444	442	442
query55	73	69	72	69
query56	287	269	270	269
query57	1143	1031	1069	1031
query58	248	246	262	246
query59	3000	2834	2641	2641
query60	287	276	277	276
query61	94	96	123	96
query62	796	651	647	647
query63	318	286	283	283
query64	9393	2250	1653	1653
query65	3154	3092	3141	3092
query66	760	327	329	327
query67	15501	15114	15142	15114
query68	4518	534	532	532
query69	637	465	351	351
query70	1173	1111	1119	1111
query71	415	283	287	283
query72	8485	5590	5500	5500
query73	748	323	319	319
query74	6065	5748	5707	5707
query75	3980	2694	2717	2694
query76	2849	935	948	935
query77	665	312	324	312
query78	9885	9139	8880	8880
query79	2259	524	529	524
query80	2878	483	477	477
query81	589	215	216	215
query82	806	140	136	136
query83	310	165	173	165
query84	265	93	87	87
query85	1922	322	319	319
query86	460	311	313	311
query87	3285	3074	3077	3074
query88	4016	2448	2451	2448
query89	483	385	394	385
query90	1875	197	195	195
query91	131	105	101	101
query92	63	49	50	49
query93	2456	495	489	489
query94	1133	215	210	210
query95	404	317	318	317
query96	609	291	276	276
query97	3249	3023	3060	3023
query98	228	208	247	208
query99	1583	1270	1214	1214
Total cold run time: 280688 ms
Total hot run time: 173112 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.05 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e1c710d8db7261a31376d348433d9a56482a6a46, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.03	0.04
query3	0.23	0.05	0.04
query4	1.68	0.07	0.06
query5	0.50	0.50	0.48
query6	1.15	0.73	0.72
query7	0.02	0.02	0.01
query8	0.06	0.04	0.05
query9	0.56	0.48	0.49
query10	0.55	0.55	0.56
query11	0.15	0.12	0.11
query12	0.15	0.12	0.13
query13	0.59	0.58	0.58
query14	0.76	0.77	0.78
query15	0.85	0.82	0.81
query16	0.37	0.37	0.37
query17	0.97	1.01	0.94
query18	0.23	0.22	0.21
query19	1.84	1.82	1.71
query20	0.01	0.00	0.01
query21	15.42	0.75	0.67
query22	4.23	7.84	1.45
query23	18.23	1.49	1.26
query24	2.06	0.24	0.22
query25	0.15	0.09	0.08
query26	0.31	0.20	0.20
query27	0.45	0.24	0.23
query28	13.32	1.00	0.98
query29	12.58	3.33	3.31
query30	0.25	0.06	0.05
query31	2.89	0.40	0.39
query32	3.25	0.47	0.48
query33	2.87	2.97	2.93
query34	16.98	4.34	4.38
query35	4.41	4.43	4.38
query36	0.65	0.46	0.52
query37	0.19	0.15	0.16
query38	0.16	0.14	0.15
query39	0.04	0.03	0.03
query40	0.16	0.12	0.13
query41	0.10	0.04	0.05
query42	0.05	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.59 s
Total hot run time: 30.05 s

@wsjz
Copy link
Contributor Author

wsjz commented Jul 15, 2024

run external

@wsjz
Copy link
Contributor Author

wsjz commented Jul 16, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40043 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e9cb6707142e90dd9240ef54042a36517853b5ae, data reload: false

------ Round 1 ----------------------------------
q1	17978	4497	4362	4362
q2	2604	203	195	195
q3	12107	1183	1062	1062
q4	11497	786	829	786
q5	7768	2692	2684	2684
q6	221	139	133	133
q7	971	599	587	587
q8	9316	2062	2077	2062
q9	8758	6570	6565	6565
q10	8753	3807	3795	3795
q11	450	237	248	237
q12	393	221	218	218
q13	18685	3000	2959	2959
q14	277	244	244	244
q15	533	501	506	501
q16	492	383	373	373
q17	971	733	684	684
q18	8112	7427	7441	7427
q19	8535	1436	1416	1416
q20	676	314	314	314
q21	4952	3302	3148	3148
q22	354	291	292	291
Total cold run time: 124403 ms
Total hot run time: 40043 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4341	4311	4280	4280
q2	380	276	263	263
q3	3024	2713	2721	2713
q4	1893	1597	1581	1581
q5	5298	5311	5313	5311
q6	218	128	130	128
q7	2133	1708	1737	1708
q8	3211	3313	3305	3305
q9	8430	8372	8349	8349
q10	3887	3722	3665	3665
q11	567	495	492	492
q12	783	623	599	599
q13	17313	2979	2944	2944
q14	296	264	288	264
q15	518	472	470	470
q16	465	408	411	408
q17	1763	1478	1463	1463
q18	7606	7680	7351	7351
q19	2628	1520	1569	1520
q20	2012	1761	1766	1761
q21	4878	4614	4669	4614
q22	595	507	510	507
Total cold run time: 72239 ms
Total hot run time: 53696 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173879 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e9cb6707142e90dd9240ef54042a36517853b5ae, data reload: false

query1	911	369	382	369
query2	6443	1928	1768	1768
query3	6672	210	215	210
query4	28712	17673	17295	17295
query5	4244	482	488	482
query6	275	194	158	158
query7	4587	301	284	284
query8	230	185	195	185
query9	8428	2388	2371	2371
query10	430	280	280	280
query11	10658	10227	10208	10208
query12	128	87	89	87
query13	1674	370	370	370
query14	9666	7729	7840	7729
query15	213	178	167	167
query16	7818	325	315	315
query17	1811	554	553	553
query18	1925	288	292	288
query19	201	151	155	151
query20	91	84	86	84
query21	207	136	125	125
query22	4429	4292	4065	4065
query23	33808	33169	33272	33169
query24	12162	2908	2906	2906
query25	671	393	400	393
query26	1773	152	149	149
query27	2978	275	271	271
query28	7413	2005	1995	1995
query29	1134	651	629	629
query30	283	155	152	152
query31	966	749	741	741
query32	92	56	55	55
query33	798	318	315	315
query34	938	486	485	485
query35	706	578	627	578
query36	1073	943	958	943
query37	297	84	79	79
query38	2911	2785	2740	2740
query39	858	801	808	801
query40	289	124	121	121
query41	49	53	48	48
query42	113	99	102	99
query43	512	472	482	472
query44	1209	747	725	725
query45	191	161	160	160
query46	1094	752	741	741
query47	1919	1804	1808	1804
query48	371	299	296	296
query49	1212	427	430	427
query50	790	394	399	394
query51	6882	6812	6802	6802
query52	110	95	98	95
query53	359	289	291	289
query54	930	477	465	465
query55	80	75	75	75
query56	307	280	297	280
query57	1145	1079	1066	1066
query58	265	254	262	254
query59	2878	2730	2720	2720
query60	434	269	309	269
query61	95	92	98	92
query62	844	655	656	655
query63	338	284	280	280
query64	10469	2199	1675	1675
query65	3172	3080	3148	3080
query66	1384	338	339	338
query67	15784	15005	15164	15005
query68	6555	540	540	540
query69	783	436	336	336
query70	1239	1155	1147	1147
query71	520	275	274	274
query72	8652	6333	5728	5728
query73	824	323	324	323
query74	6408	5776	5718	5718
query75	5097	2681	2736	2681
query76	4717	915	936	915
query77	786	295	311	295
query78	9793	9098	9036	9036
query79	8490	527	527	527
query80	1016	485	469	469
query81	587	269	220	220
query82	799	139	138	138
query83	333	166	162	162
query84	276	89	92	89
query85	1410	306	304	304
query86	400	337	311	311
query87	3325	3165	3128	3128
query88	4671	2386	2370	2370
query89	551	402	384	384
query90	2044	189	188	188
query91	126	103	100	100
query92	65	48	51	48
query93	7005	508	499	499
query94	1307	259	207	207
query95	406	312	313	312
query96	600	273	271	271
query97	3180	3061	3026	3026
query98	218	198	200	198
query99	1549	1305	1263	1263
Total cold run time: 305571 ms
Total hot run time: 173879 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.7 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e9cb6707142e90dd9240ef54042a36517853b5ae, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.04	0.05
query4	1.68	0.07	0.07
query5	0.50	0.47	0.48
query6	1.13	0.73	0.72
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.56	0.47	0.48
query10	0.55	0.54	0.53
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.59	0.59	0.59
query14	0.78	0.78	0.80
query15	0.85	0.82	0.83
query16	0.36	0.37	0.36
query17	0.96	0.97	0.96
query18	0.22	0.22	0.23
query19	1.77	1.74	1.68
query20	0.02	0.01	0.01
query21	15.40	0.78	0.67
query22	4.44	7.18	1.95
query23	18.67	1.39	1.30
query24	2.11	0.22	0.24
query25	0.14	0.09	0.09
query26	0.30	0.21	0.21
query27	0.46	0.23	0.24
query28	13.24	1.02	1.01
query29	12.59	3.30	3.32
query30	0.25	0.06	0.05
query31	2.88	0.39	0.40
query32	3.31	0.49	0.48
query33	2.92	2.91	2.90
query34	16.89	4.43	4.37
query35	4.54	4.47	4.41
query36	0.65	0.46	0.47
query37	0.19	0.17	0.17
query38	0.16	0.15	0.14
query39	0.05	0.04	0.03
query40	0.15	0.12	0.12
query41	0.09	0.04	0.04
query42	0.06	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 110.22 s
Total hot run time: 30.7 s

morningman
morningman reviewed Jul 16, 2024
View reviewed changes
...c/main/java/org/apache/doris/common/security/authentication/HadoopKerberosAuthenticator.java Outdated Show resolved Hide resolved
...src/main/java/org/apache/doris/common/security/authentication/HadoopSimpleAuthenticator.java Outdated Show resolved Hide resolved
...src/main/java/org/apache/doris/common/security/authentication/HadoopSimpleAuthenticator.java Outdated Show resolved Hide resolved
...n/java/org/apache/doris/common/security/authentication/ImpersonatingHadoopAuthenticator.java Outdated Show resolved Hide resolved
...n/java/org/apache/doris/common/security/authentication/ImpersonatingHadoopAuthenticator.java Show resolved Hide resolved
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreCache.java Outdated Show resolved Hide resolved
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveMetaStoreCache.java Outdated Show resolved Hide resolved
@wsjz
Copy link
Contributor Author

wsjz commented Jul 16, 2024

run buildall

@wsjz
Copy link
Contributor Author

wsjz commented Jul 18, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40119 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c184c1906b4de7d915bdd8624a5cdc53694d08e1, data reload: false

------ Round 1 ----------------------------------
q1	17633	4366	4321	4321
q2	2023	193	188	188
q3	10460	1206	1124	1124
q4	10189	900	861	861
q5	7531	2732	2776	2732
q6	217	134	136	134
q7	951	591	618	591
q8	9219	2076	2072	2072
q9	8734	6568	6585	6568
q10	8818	3772	3741	3741
q11	468	237	246	237
q12	409	231	230	230
q13	18655	2975	3032	2975
q14	275	242	232	232
q15	559	488	493	488
q16	498	388	380	380
q17	974	629	650	629
q18	8095	7538	7473	7473
q19	7535	1459	1404	1404
q20	650	311	322	311
q21	4901	3146	3353	3146
q22	358	287	282	282
Total cold run time: 119152 ms
Total hot run time: 40119 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4448	4262	4264	4262
q2	368	275	262	262
q3	3083	2933	2959	2933
q4	1939	1724	1749	1724
q5	5560	5539	5478	5478
q6	228	135	136	135
q7	2244	1857	1866	1857
q8	3320	3460	3397	3397
q9	8781	8854	8818	8818
q10	4147	3829	3842	3829
q11	579	492	523	492
q12	809	623	622	622
q13	15987	3209	3202	3202
q14	300	310	288	288
q15	543	510	505	505
q16	478	445	435	435
q17	1838	1553	1511	1511
q18	8062	8007	7741	7741
q19	1756	1438	1478	1438
q20	2224	1882	1871	1871
q21	12432	4870	4708	4708
q22	607	520	534	520
Total cold run time: 79733 ms
Total hot run time: 56028 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173411 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c184c1906b4de7d915bdd8624a5cdc53694d08e1, data reload: false

query1	911	382	365	365
query2	6431	1867	1822	1822
query3	6662	209	220	209
query4	28226	17431	17092	17092
query5	3691	496	475	475
query6	285	175	169	169
query7	4586	285	289	285
query8	241	185	184	184
query9	8617	2443	2439	2439
query10	434	282	290	282
query11	11280	9938	10139	9938
query12	127	90	85	85
query13	1655	379	368	368
query14	9539	7955	8043	7955
query15	219	170	172	170
query16	7639	315	323	315
query17	1352	560	538	538
query18	1632	282	285	282
query19	206	155	154	154
query20	88	84	82	82
query21	211	128	122	122
query22	4505	4062	4048	4048
query23	34053	33548	33714	33548
query24	10465	2934	2850	2850
query25	593	407	404	404
query26	699	147	147	147
query27	2360	284	282	282
query28	6067	2091	2081	2081
query29	880	624	649	624
query30	246	150	158	150
query31	944	767	751	751
query32	97	56	55	55
query33	671	296	296	296
query34	878	512	522	512
query35	665	601	569	569
query36	1140	1015	985	985
query37	142	85	91	85
query38	2993	2849	2797	2797
query39	872	794	855	794
query40	209	118	136	118
query41	44	45	45	45
query42	115	104	102	102
query43	517	477	436	436
query44	1095	730	730	730
query45	189	157	161	157
query46	1076	724	731	724
query47	1841	1798	1773	1773
query48	358	301	287	287
query49	828	425	424	424
query50	779	397	389	389
query51	6925	6824	6736	6736
query52	108	97	96	96
query53	359	289	292	289
query54	878	447	444	444
query55	76	76	73	73
query56	293	271	271	271
query57	1133	1085	1055	1055
query58	252	262	270	262
query59	2911	2821	2609	2609
query60	306	277	317	277
query61	97	97	95	95
query62	780	645	663	645
query63	322	296	287	287
query64	9097	2224	1679	1679
query65	3196	3097	3124	3097
query66	703	325	331	325
query67	15539	15038	14905	14905
query68	5702	536	541	536
query69	752	465	373	373
query70	1219	1133	1050	1050
query71	444	282	290	282
query72	9039	5542	5629	5542
query73	778	331	324	324
query74	6016	5705	5647	5647
query75	4213	2689	2720	2689
query76	3602	1000	955	955
query77	711	316	338	316
query78	9771	8950	10054	8950
query79	2439	527	528	527
query80	2225	480	477	477
query81	595	219	217	217
query82	948	142	135	135
query83	290	168	167	167
query84	261	93	86	86
query85	1443	326	303	303
query86	472	288	316	288
query87	3281	3087	3140	3087
query88	3289	2368	2374	2368
query89	476	377	393	377
query90	1898	197	193	193
query91	129	102	103	102
query92	61	49	50	49
query93	2145	508	496	496
query94	1267	212	210	210
query95	409	377	318	318
query96	594	274	273	273
query97	3194	3013	3021	3013
query98	243	203	197	197
query99	1465	1262	1225	1225
Total cold run time: 281699 ms
Total hot run time: 173411 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.43 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c184c1906b4de7d915bdd8624a5cdc53694d08e1, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.03	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.07
query5	0.49	0.49	0.49
query6	1.14	0.73	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.05
query9	0.55	0.48	0.49
query10	0.55	0.56	0.55
query11	0.15	0.11	0.12
query12	0.15	0.13	0.13
query13	0.60	0.59	0.60
query14	0.75	0.77	0.82
query15	0.86	0.81	0.82
query16	0.37	0.35	0.38
query17	0.99	0.96	0.98
query18	0.24	0.23	0.24
query19	1.81	1.68	1.68
query20	0.02	0.01	0.01
query21	15.39	0.76	0.66
query22	4.92	6.58	1.81
query23	18.69	1.43	1.31
query24	2.11	0.22	0.22
query25	0.14	0.09	0.09
query26	0.30	0.21	0.21
query27	0.45	0.23	0.23
query28	13.20	1.02	0.99
query29	12.63	3.32	3.29
query30	0.25	0.06	0.05
query31	2.87	0.39	0.40
query32	3.29	0.48	0.48
query33	2.93	2.83	2.92
query34	16.76	4.37	4.32
query35	4.38	4.42	4.37
query36	0.66	0.49	0.48
query37	0.19	0.16	0.16
query38	0.16	0.14	0.14
query39	0.05	0.03	0.03
query40	0.16	0.12	0.12
query41	0.09	0.05	0.05
query42	0.06	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 110.49 s
Total hot run time: 30.43 s

morningman
morningman approved these changes Jul 18, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 18, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@hello-stephen hello-stephen merged commit a5fd824 into apache:master Jul 18, 2024
29 of 31 checks passed
@wsjz wsjz deleted the ugi_opt branch July 18, 2024 08:05
@wsjz wsjz mentioned this pull request Jul 18, 2024
Closed
dataroaring pushed a commit that referenced this pull request Jul 19, 2024
@wsjz @dataroaring
[fix](kerberos)fix and refactor ugi login for kerberos and simple aut…
ee4dbc0 
…hentication (#37301)

## Proposed changes
optimize kerberos ugi login:
1. support authentication framework for external table
2. cache ugi to avoid conflicts when set configuration for ugi login 
3. do ugi login just on creating catalog, which reduces the ugi create
times
4. only simple authentication will use getloginUser, which avoids
conflicts between simple and kerberos authentication

<!--Describe your changes.-->
@wsjz wsjz mentioned this pull request Jul 31, 2024
Merged
morningman pushed a commit that referenced this pull request Aug 1, 2024
@wsjz
[fix](kerberos)fix and refactor ugi login for kerberos and simple aut…
b094306 
…hentication (#38607)

pick from  (#37301)
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
@yiguolei yiguolei mentioned this pull request Sep 5, 2024
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hello-stephen hello-stephen hello-stephen approved these changes

@wuwenchi wuwenchi wuwenchi approved these changes

@morningman morningman morningman approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.1-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@wsjz @doris-robot @morningman @wuwenchi @hello-stephen @dataroaring