Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](auth)Fix after fe restarting, external permissions are lost #43275

Merged
merged 2 commits into from
Nov 6, 2024

Conversation

zddr
Copy link
Contributor

@zddr zddr commented Nov 5, 2024

cause by: #39597

This PR checks whether the authorized tableName exists. After restarting, when replaying the log, the metadata of the external cannot be obtained, so the replay will report an error, but it does not affect the startup, only printing the log.

Solution: No longer verify during playback

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zddr
Copy link
Contributor Author

zddr commented Nov 5, 2024

run buildall

@wm1581066 wm1581066 added dev/2.1.x dev/3.0.x usercase Important user case type label p0_b labels Nov 5, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 41419 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ce560f799ee0cb470f9eff51517f7422eddb10ad, data reload: false

------ Round 1 ----------------------------------
q1	17609	7464	7303	7303
q2	2057	163	158	158
q3	10568	1142	1160	1142
q4	10228	836	866	836
q5	7728	3082	3085	3082
q6	236	149	146	146
q7	1012	604	605	604
q8	9345	1955	1996	1955
q9	6578	6398	6424	6398
q10	7066	2396	2449	2396
q11	458	263	260	260
q12	408	216	218	216
q13	17764	3013	2997	2997
q14	243	208	212	208
q15	570	530	531	530
q16	626	583	594	583
q17	964	625	538	538
q18	7474	6760	6708	6708
q19	1350	1040	934	934
q20	472	183	180	180
q21	4083	3261	3225	3225
q22	1110	1031	1020	1020
Total cold run time: 107949 ms
Total hot run time: 41419 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7298	7298	7211	7211
q2	329	233	229	229
q3	3002	2826	2834	2826
q4	1973	1737	1772	1737
q5	5469	5513	5531	5513
q6	228	139	138	138
q7	2139	1729	1757	1729
q8	3264	3393	3421	3393
q9	8581	8567	8614	8567
q10	3529	3478	3454	3454
q11	589	505	504	504
q12	759	591	621	591
q13	8506	3016	2985	2985
q14	286	260	269	260
q15	551	525	504	504
q16	681	633	625	625
q17	1824	1604	1544	1544
q18	7802	7502	7497	7497
q19	1687	1445	1445	1445
q20	2043	1807	1832	1807
q21	5496	5390	5350	5350
q22	1134	1023	1031	1023
Total cold run time: 67170 ms
Total hot run time: 58932 ms

Copy link
Contributor

github-actions bot commented Nov 5, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Nov 5, 2024
Copy link
Contributor

github-actions bot commented Nov 5, 2024

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-DS: Total hot run time: 191785 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ce560f799ee0cb470f9eff51517f7422eddb10ad, data reload: false

query1	980	370	365	365
query2	6513	2112	2033	2033
query3	6787	223	219	219
query4	33965	23770	23678	23678
query5	4344	460	432	432
query6	264	170	164	164
query7	4602	290	284	284
query8	281	226	217	217
query9	9785	2643	2631	2631
query10	470	251	247	247
query11	18102	15393	15324	15324
query12	148	106	102	102
query13	1666	410	410	410
query14	10628	6747	7445	6747
query15	258	182	181	181
query16	8086	472	474	472
query17	1662	582	560	560
query18	2143	300	294	294
query19	356	158	146	146
query20	117	110	109	109
query21	209	102	101	101
query22	4486	4376	4450	4376
query23	35066	34172	34138	34138
query24	11200	2814	2821	2814
query25	678	411	405	405
query26	1610	165	160	160
query27	2842	282	283	282
query28	8246	2436	2412	2412
query29	991	430	419	419
query30	317	160	167	160
query31	1056	833	819	819
query32	98	56	62	56
query33	772	278	272	272
query34	990	508	530	508
query35	862	742	726	726
query36	1103	959	943	943
query37	134	86	73	73
query38	4538	4269	4227	4227
query39	1470	1433	1424	1424
query40	287	104	103	103
query41	50	46	47	46
query42	106	99	97	97
query43	530	491	492	491
query44	1327	829	806	806
query45	188	172	166	166
query46	1124	730	694	694
query47	1921	1851	1846	1846
query48	417	322	329	322
query49	1263	395	397	395
query50	819	392	409	392
query51	7356	7127	7112	7112
query52	101	89	92	89
query53	265	178	183	178
query54	1204	404	408	404
query55	78	76	77	76
query56	265	266	242	242
query57	1293	1171	1140	1140
query58	234	211	207	207
query59	3434	2922	3038	2922
query60	268	248	244	244
query61	107	106	108	106
query62	875	673	680	673
query63	222	191	188	188
query64	5237	637	625	625
query65	3278	3252	3231	3231
query66	1423	302	311	302
query67	16133	15875	15752	15752
query68	4962	567	547	547
query69	433	268	254	254
query70	1191	1151	1144	1144
query71	418	256	254	254
query72	6761	4106	3955	3955
query73	798	351	378	351
query74	10551	9061	9043	9043
query75	3452	2663	2654	2654
query76	2914	1070	1130	1070
query77	380	261	266	261
query78	10263	9502	9367	9367
query79	1712	590	599	590
query80	1080	432	415	415
query81	544	238	236	236
query82	942	123	115	115
query83	210	133	135	133
query84	230	70	67	67
query85	1232	296	322	296
query86	362	318	313	313
query87	4796	4801	4660	4660
query88	3349	2180	2143	2143
query89	395	287	297	287
query90	1938	190	191	190
query91	141	103	102	102
query92	58	50	48	48
query93	1124	521	525	521
query94	886	299	290	290
query95	350	250	251	250
query96	614	276	275	275
query97	2890	2691	2740	2691
query98	209	198	195	195
query99	1554	1289	1304	1289
Total cold run time: 304369 ms
Total hot run time: 191785 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.03 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ce560f799ee0cb470f9eff51517f7422eddb10ad, data reload: false

query1	0.04	0.03	0.04
query2	0.07	0.03	0.02
query3	0.23	0.06	0.07
query4	1.64	0.10	0.10
query5	0.40	0.40	0.42
query6	1.14	0.64	0.66
query7	0.02	0.01	0.01
query8	0.04	0.03	0.04
query9	0.56	0.50	0.50
query10	0.55	0.56	0.54
query11	0.14	0.11	0.10
query12	0.13	0.11	0.11
query13	0.60	0.59	0.58
query14	2.72	2.70	2.71
query15	0.90	0.84	0.82
query16	0.39	0.37	0.38
query17	1.04	0.99	1.10
query18	0.24	0.22	0.21
query19	1.97	1.93	1.89
query20	0.01	0.01	0.01
query21	15.40	0.60	0.56
query22	3.21	2.06	1.07
query23	16.97	0.85	0.84
query24	2.58	1.29	1.49
query25	0.26	0.24	0.06
query26	0.49	0.13	0.14
query27	0.04	0.05	0.04
query28	10.43	1.10	1.06
query29	12.53	3.22	3.20
query30	0.25	0.06	0.06
query31	2.87	0.40	0.39
query32	3.25	0.46	0.45
query33	3.03	3.02	3.04
query34	16.95	4.50	4.47
query35	4.56	4.50	4.53
query36	0.66	0.49	0.49
query37	0.09	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.01
query43	0.04	0.03	0.03
Total cold run time: 106.78 s
Total hot run time: 32.03 s

@zddr
Copy link
Contributor Author

zddr commented Nov 5, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41495 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ce560f799ee0cb470f9eff51517f7422eddb10ad, data reload: false

------ Round 1 ----------------------------------
q1	17578	7390	7283	7283
q2	2033	188	156	156
q3	10611	1088	1205	1088
q4	10565	873	787	787
q5	7747	3065	3057	3057
q6	245	152	151	151
q7	1036	628	623	623
q8	9379	1926	2068	1926
q9	6524	6414	6421	6414
q10	7094	2463	2459	2459
q11	473	260	268	260
q12	409	220	220	220
q13	17768	3043	3056	3043
q14	235	207	212	207
q15	576	526	513	513
q16	671	589	573	573
q17	970	567	550	550
q18	7353	6784	6743	6743
q19	1339	970	974	970
q20	482	186	189	186
q21	4086	3300	3349	3300
q22	1084	986	1018	986
Total cold run time: 108258 ms
Total hot run time: 41495 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7288	7262	7304	7262
q2	316	242	226	226
q3	3031	3005	2982	2982
q4	2135	1871	1804	1804
q5	5770	5776	5812	5776
q6	225	138	144	138
q7	2264	1822	1837	1822
q8	3395	3462	3536	3462
q9	8995	8951	8786	8786
q10	3599	3566	3570	3566
q11	605	521	513	513
q12	852	630	638	630
q13	9762	3204	3195	3195
q14	309	285	281	281
q15	580	551	523	523
q16	665	632	664	632
q17	1839	1678	1594	1594
q18	8242	7739	7653	7653
q19	1699	1563	1709	1563
q20	2101	1909	1879	1879
q21	5429	5523	5588	5523
q22	1152	1100	1078	1078
Total cold run time: 70253 ms
Total hot run time: 60888 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195343 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ce560f799ee0cb470f9eff51517f7422eddb10ad, data reload: false

query1	1200	896	906	896
query2	6219	2070	2035	2035
query3	10764	4007	3869	3869
query4	67606	29710	23748	23748
query5	4959	441	434	434
query6	392	172	163	163
query7	5625	284	290	284
query8	326	223	218	218
query9	9355	2635	2638	2635
query10	460	260	244	244
query11	17555	15325	15719	15325
query12	161	99	102	99
query13	1562	433	429	429
query14	9771	6846	7259	6846
query15	216	190	205	190
query16	7099	458	415	415
query17	1021	564	559	559
query18	1806	297	293	293
query19	195	151	144	144
query20	114	109	111	109
query21	205	99	103	99
query22	4647	4412	4115	4115
query23	34773	34341	34073	34073
query24	6380	2776	2787	2776
query25	512	397	403	397
query26	648	160	163	160
query27	1905	284	277	277
query28	4177	2406	2392	2392
query29	692	433	419	419
query30	228	158	148	148
query31	993	805	832	805
query32	71	54	59	54
query33	426	266	264	264
query34	917	500	527	500
query35	852	732	722	722
query36	1056	909	963	909
query37	119	75	75	75
query38	4322	4298	4371	4298
query39	1490	1440	1403	1403
query40	202	97	99	97
query41	48	48	47	47
query42	115	99	95	95
query43	523	508	479	479
query44	1168	823	815	815
query45	180	164	172	164
query46	1149	702	703	702
query47	1952	1844	1855	1844
query48	420	340	317	317
query49	740	406	401	401
query50	803	393	401	393
query51	7156	7042	7184	7042
query52	96	89	88	88
query53	252	180	184	180
query54	521	401	402	401
query55	78	97	72	72
query56	238	228	232	228
query57	1258	1167	1132	1132
query58	207	199	205	199
query59	3254	3188	3046	3046
query60	269	246	257	246
query61	99	108	97	97
query62	789	669	688	669
query63	207	192	189	189
query64	1352	638	622	622
query65	3305	3252	3215	3215
query66	698	291	328	291
query67	15989	15720	15688	15688
query68	3244	574	574	574
query69	416	257	256	256
query70	1168	1101	1127	1101
query71	355	251	256	251
query72	6223	3970	4007	3970
query73	760	355	356	355
query74	10073	8946	9125	8946
query75	3343	2653	2686	2653
query76	1797	1068	1145	1068
query77	475	278	257	257
query78	10526	9321	9401	9321
query79	1495	605	594	594
query80	868	426	458	426
query81	512	245	240	240
query82	1288	114	117	114
query83	152	135	137	135
query84	280	68	71	68
query85	861	298	286	286
query86	339	294	307	294
query87	4881	4842	4612	4612
query88	3492	2181	2147	2147
query89	405	292	284	284
query90	1994	197	181	181
query91	138	110	111	110
query92	61	48	49	48
query93	1873	517	537	517
query94	781	288	296	288
query95	338	246	251	246
query96	606	287	291	287
query97	2957	2680	2636	2636
query98	219	195	194	194
query99	1569	1311	1304	1304
Total cold run time: 317376 ms
Total hot run time: 195343 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.33 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ce560f799ee0cb470f9eff51517f7422eddb10ad, data reload: false

query1	0.03	0.03	0.02
query2	0.06	0.03	0.03
query3	0.23	0.07	0.07
query4	1.62	0.10	0.10
query5	0.40	0.39	0.41
query6	1.16	0.66	0.65
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.56	0.49	0.50
query10	0.55	0.55	0.54
query11	0.14	0.10	0.10
query12	0.14	0.11	0.10
query13	0.61	0.60	0.58
query14	2.72	2.71	2.88
query15	0.89	0.82	0.82
query16	0.40	0.38	0.38
query17	1.03	1.05	1.04
query18	0.20	0.19	0.20
query19	1.90	1.79	1.92
query20	0.01	0.00	0.01
query21	15.36	0.59	0.58
query22	2.62	2.60	2.38
query23	17.05	0.97	0.77
query24	2.84	0.46	1.15
query25	0.23	0.17	0.04
query26	0.40	0.14	0.13
query27	0.03	0.04	0.04
query28	11.05	1.09	1.07
query29	12.54	3.23	3.20
query30	0.25	0.08	0.07
query31	2.85	0.39	0.38
query32	3.28	0.46	0.46
query33	3.03	3.04	3.05
query34	16.94	4.45	4.46
query35	4.51	4.44	4.47
query36	0.67	0.49	0.48
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.15	0.12	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 106.83 s
Total hot run time: 32.33 s

@zddr
Copy link
Contributor Author

zddr commented Nov 6, 2024

run p0

@morrySnow morrySnow merged commit c33af31 into apache:master Nov 6, 2024
27 of 29 checks passed
github-actions bot pushed a commit that referenced this pull request Nov 6, 2024
cause by: #39597

This PR checks whether the authorized tableName exists. After
restarting, when replaying the log, the metadata of the external cannot
be obtained, so the replay will report an error, but it does not affect
the startup, only printing the log.

Solution: No longer verify during playback
github-actions bot pushed a commit that referenced this pull request Nov 6, 2024
cause by: #39597

This PR checks whether the authorized tableName exists. After
restarting, when replaying the log, the metadata of the external cannot
be obtained, so the replay will report an error, but it does not affect
the startup, only printing the log.

Solution: No longer verify during playback
morningman pushed a commit that referenced this pull request Nov 6, 2024
zddr added a commit to zddr/incubator-doris that referenced this pull request Nov 6, 2024
…e#43275)

cause by: apache#39597

This PR checks whether the authorized tableName exists. After
restarting, when replaying the log, the metadata of the external cannot
be obtained, so the replay will report an error, but it does not affect
the startup, only printing the log.

Solution: No longer verify during playback
dataroaring pushed a commit that referenced this pull request Nov 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.x dev/2.1.7-merged dev/3.0.3-merged p0_b reviewed usercase Important user case type label
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants