Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](Nereids) remove db readlock before get table from db (#38660) #38730

Merged
merged 1 commit into from
Aug 1, 2024

Conversation

morrySnow
Copy link
Contributor

pick from master #38660

insert will hold readlock of target table before planning. if nereids need db readlock after it, will lead to dead lock. because other statement need to hold db lock before get table lock

for example:

insert: target table read lock -> database read lock
drop table: database write lock -> target table write lock

)

pick from master apache#38660

insert will hold readlock of target table before planning. if nereids
need db readlock after it, will lead to dead lock. because other
statement need to hold db lock before get table lock

for example:

insert: target table read lock -> database read lock
drop table: database write lock -> target table write lock
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@morrySnow
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49771 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 92f2201b104856526a2265afae3a063036d0ea6e, data reload: false

------ Round 1 ----------------------------------
q1	17714	4345	4331	4331
q2	2066	159	151	151
q3	10262	1888	1883	1883
q4	10380	1271	1335	1271
q5	8501	3857	3937	3857
q6	237	127	125	125
q7	2063	1620	1603	1603
q8	9300	2719	2701	2701
q9	10715	10272	10193	10193
q10	8600	3475	3497	3475
q11	421	248	247	247
q12	463	303	305	303
q13	18360	3989	4045	3989
q14	349	329	324	324
q15	508	476	473	473
q16	693	572	562	562
q17	1122	954	976	954
q18	7406	6848	6932	6848
q19	1765	1674	1656	1656
q20	533	319	301	301
q21	4431	4097	4079	4079
q22	520	445	449	445
Total cold run time: 116409 ms
Total hot run time: 49771 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4308	4280	4267	4267
q2	324	228	224	224
q3	4150	4127	4113	4113
q4	2747	2778	2743	2743
q5	7139	7195	7149	7149
q6	235	121	119	119
q7	3184	2865	2836	2836
q8	4334	4457	4490	4457
q9	16776	16745	16764	16745
q10	4240	4237	4272	4237
q11	757	693	678	678
q12	1038	862	840	840
q13	6788	3718	3787	3718
q14	455	421	436	421
q15	504	457	455	455
q16	743	680	695	680
q17	3920	3931	3800	3800
q18	8802	8766	8876	8766
q19	1697	1731	1712	1712
q20	2388	2145	2151	2145
q21	8621	8485	8468	8468
q22	1041	970	989	970
Total cold run time: 84191 ms
Total hot run time: 79543 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 204076 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 92f2201b104856526a2265afae3a063036d0ea6e, data reload: false

query1	932	392	417	392
query2	6522	2914	2457	2457
query3	6917	205	203	203
query4	19929	18185	18092	18092
query5	19737	6503	6522	6503
query6	277	221	235	221
query7	4173	297	305	297
query8	433	458	388	388
query9	3139	2713	2648	2648
query10	419	325	294	294
query11	11329	10807	10852	10807
query12	127	82	77	77
query13	5623	707	706	706
query14	17881	13090	13469	13090
query15	361	248	256	248
query16	6464	287	267	267
query17	1712	1455	870	870
query18	2333	419	413	413
query19	198	156	160	156
query20	82	78	80	78
query21	187	104	97	97
query22	5279	5047	5019	5019
query23	32752	31809	31888	31809
query24	7001	6532	6497	6497
query25	516	442	424	424
query26	532	164	160	160
query27	1884	297	306	297
query28	6115	2409	2377	2377
query29	2909	2788	2794	2788
query30	246	170	168	168
query31	915	795	774	774
query32	65	66	57	57
query33	406	263	268	263
query34	846	477	510	477
query35	1113	939	892	892
query36	1355	1339	1222	1222
query37	90	64	62	62
query38	3085	2910	2902	2902
query39	1386	1306	1345	1306
query40	207	99	89	89
query41	46	47	44	44
query42	84	85	83	83
query43	781	671	743	671
query44	1125	725	720	720
query45	242	242	234	234
query46	1241	954	979	954
query47	1887	1773	1623	1623
query48	1031	711	718	711
query49	632	375	380	375
query50	875	652	625	625
query51	4716	4659	4641	4641
query52	99	86	84	84
query53	458	334	327	327
query54	2651	2428	2482	2428
query55	91	78	83	78
query56	230	223	217	217
query57	1147	1206	1121	1121
query58	222	210	219	210
query59	4189	4156	3874	3874
query60	227	193	208	193
query61	101	93	101	93
query62	810	464	448	448
query63	481	346	356	346
query64	2603	1532	1496	1496
query65	3630	3554	3545	3545
query66	776	384	374	374
query67	18173	15794	17093	15794
query68	8616	660	651	651
query69	572	378	377	377
query70	1763	1611	1368	1368
query71	411	321	322	321
query72	6550	3491	3499	3491
query73	731	319	321	319
query74	6294	5864	5923	5864
query75	5327	3689	3726	3689
query76	5112	1162	1174	1162
query77	892	260	258	258
query78	12724	11633	11708	11633
query79	7988	636	639	636
query80	1179	405	419	405
query81	495	239	236	236
query82	1672	101	100	100
query83	171	134	135	134
query84	260	72	72	72
query85	878	333	335	333
query86	333	289	293	289
query87	3285	2998	3026	2998
query88	4993	2329	2300	2300
query89	416	287	302	287
query90	1960	206	215	206
query91	180	159	143	143
query92	61	57	55	55
query93	5461	631	542	542
query94	723	214	212	212
query95	1115	1076	1072	1072
query96	638	336	327	327
query97	6443	6446	6366	6366
query98	194	182	173	173
query99	3011	892	861	861
Total cold run time: 314473 ms
Total hot run time: 204076 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.89 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 92f2201b104856526a2265afae3a063036d0ea6e, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.25	0.05	0.05
query4	1.81	0.06	0.07
query5	0.53	0.54	0.52
query6	1.23	0.62	0.61
query7	0.01	0.01	0.01
query8	0.03	0.03	0.02
query9	0.53	0.48	0.48
query10	0.55	0.54	0.53
query11	0.12	0.10	0.08
query12	0.12	0.10	0.09
query13	0.62	0.62	0.61
query14	0.79	0.79	0.79
query15	0.78	0.76	0.76
query16	0.36	0.38	0.36
query17	1.03	1.03	0.99
query18	0.24	0.24	0.26
query19	1.91	1.78	1.86
query20	0.01	0.01	0.01
query21	15.47	0.55	0.59
query22	2.02	2.64	1.85
query23	17.51	1.00	0.83
query24	5.19	1.12	1.56
query25	0.40	0.12	0.04
query26	0.60	0.16	0.14
query27	0.05	0.04	0.04
query28	7.34	0.79	0.72
query29	12.60	2.36	2.25
query30	0.55	0.53	0.54
query31	2.81	0.40	0.37
query32	3.38	0.49	0.50
query33	3.12	3.09	3.07
query34	15.26	4.78	4.76
query35	4.86	4.85	4.85
query36	1.06	1.02	1.01
query37	0.06	0.04	0.05
query38	0.03	0.02	0.03
query39	0.02	0.01	0.02
query40	0.16	0.14	0.14
query41	0.06	0.02	0.01
query42	0.02	0.01	0.02
query43	0.02	0.02	0.01
Total cold run time: 103.6 s
Total hot run time: 30.89 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 92f2201b104856526a2265afae3a063036d0ea6e with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.5 seconds inserted 10000000 Rows, about 465K ops/s

@xiaokang xiaokang merged commit 6177a4b into apache:branch-2.0 Aug 1, 2024
23 of 25 checks passed
@morrySnow morrySnow deleted the 2.0_38660 branch September 18, 2024 03:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants