Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](Nereids) remove db readlock before get table from db #38660

Merged
merged 1 commit into from
Aug 1, 2024

Conversation

morrySnow
Copy link
Contributor

insert will hold readlock of target table before planning. if nereids need db readlock after it, will lead to dead lock. because other statement need to hold db lock before get table lock

for example:

insert: target table read lock -> database read lock
drop table: database write lock -> target table write lock

insert will hold readlock of target table before planning.
if nereids need db readlock after it, will lead to dead lock.
because other statement need to hold db lock before get table lock

for example:

insert: target table read lock -> database read lock
drop table: database write lock -> target table write lock
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@morrySnow
Copy link
Contributor Author

run buildall

@github-actions github-actions bot added the doing label Aug 1, 2024
@morrySnow morrySnow added usercase Important user case type label p0_b dev/2.0.x dev/2.1.x labels Aug 1, 2024
Copy link
Contributor

github-actions bot commented Aug 1, 2024

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 42007 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0758aaec110e8f73ac79e92b68fbb0eb7ff9c12d, data reload: false

------ Round 1 ----------------------------------
q1	18262	4138	4120	4120
q2	2021	198	201	198
q3	10472	1296	1377	1296
q4	10439	851	948	851
q5	8347	2999	2960	2960
q6	220	135	135	135
q7	1032	622	613	613
q8	9437	1781	1935	1781
q9	8531	6660	6708	6660
q10	9004	4077	4053	4053
q11	1338	284	269	269
q12	1010	253	258	253
q13	19319	3000	2963	2963
q14	268	252	241	241
q15	521	504	500	500
q16	538	404	392	392
q17	988	951	909	909
q18	9097	7357	7303	7303
q19	1679	1220	1214	1214
q20	524	328	347	328
q21	5220	4831	4693	4693
q22	352	284	275	275
Total cold run time: 118619 ms
Total hot run time: 42007 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4077	4006	4035	4006
q2	331	233	225	225
q3	3137	3218	3114	3114
q4	1979	1993	1934	1934
q5	5479	5437	5464	5437
q6	221	133	133	133
q7	2162	1780	1814	1780
q8	3309	3356	3357	3356
q9	8614	8729	8587	8587
q10	3958	3912	4007	3912
q11	546	465	476	465
q12	784	595	596	595
q13	13878	3139	3151	3139
q14	304	283	269	269
q15	546	490	486	486
q16	479	441	436	436
q17	1737	1707	1729	1707
q18	8269	7684	7708	7684
q19	2422	1713	1692	1692
q20	2047	1832	1829	1829
q21	5859	5128	5328	5128
q22	525	475	471	471
Total cold run time: 70663 ms
Total hot run time: 56385 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 170559 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0758aaec110e8f73ac79e92b68fbb0eb7ff9c12d, data reload: false

query1	910	370	367	367
query2	6437	1733	1692	1692
query3	6663	211	232	211
query4	19803	17423	17509	17423
query5	3639	503	512	503
query6	282	170	157	157
query7	4609	307	296	296
query8	247	188	186	186
query9	8512	2442	2414	2414
query10	430	284	283	283
query11	10598	10198	10095	10095
query12	121	89	87	87
query13	1642	381	385	381
query14	8810	7106	7824	7106
query15	208	166	170	166
query16	6982	481	431	431
query17	966	578	565	565
query18	1944	293	292	292
query19	199	153	147	147
query20	92	85	89	85
query21	206	103	101	101
query22	4146	4207	3955	3955
query23	33513	33601	33593	33593
query24	9260	3093	3098	3093
query25	668	410	437	410
query26	1670	156	153	153
query27	2950	284	288	284
query28	7556	2062	2039	2039
query29	1196	449	439	439
query30	238	156	153	153
query31	947	802	773	773
query32	104	54	55	54
query33	711	324	316	316
query34	921	483	505	483
query35	901	754	760	754
query36	1037	886	880	880
query37	190	82	81	81
query38	3010	2841	2928	2841
query39	859	816	812	812
query40	252	127	112	112
query41	44	45	43	43
query42	124	102	98	98
query43	486	419	428	419
query44	1174	735	728	728
query45	209	178	176	176
query46	1089	815	790	790
query47	1809	1741	1726	1726
query48	366	300	296	296
query49	930	418	409	409
query50	923	441	431	431
query51	6722	6632	6717	6632
query52	102	86	93	86
query53	256	184	182	182
query54	615	449	468	449
query55	75	77	78	77
query56	276	271	261	261
query57	1130	1067	1055	1055
query58	276	267	285	267
query59	2717	2512	2304	2304
query60	295	275	299	275
query61	101	96	92	92
query62	879	663	665	663
query63	213	182	185	182
query64	5645	1915	1912	1912
query65	3185	3117	3079	3079
query66	1015	334	337	334
query67	15385	14915	14850	14850
query68	4335	576	590	576
query69	637	390	300	300
query70	1130	1024	1082	1024
query71	447	276	283	276
query72	7156	2710	2489	2489
query73	784	337	333	333
query74	5982	5725	5654	5654
query75	3392	2727	2743	2727
query76	2762	1335	1412	1335
query77	575	306	306	306
query78	9396	8929	8905	8905
query79	2741	550	543	543
query80	1099	532	556	532
query81	560	228	227	227
query82	840	126	132	126
query83	253	167	168	167
query84	284	81	86	81
query85	1493	323	306	306
query86	473	329	306	306
query87	3292	3104	3087	3087
query88	3938	2505	2516	2505
query89	404	294	287	287
query90	1801	187	187	187
query91	131	100	104	100
query92	61	51	48	48
query93	2124	615	635	615
query94	842	303	289	289
query95	380	273	273	273
query96	620	287	293	287
query97	3272	3142	3145	3142
query98	235	211	203	203
query99	1657	1310	1294	1294
Total cold run time: 262566 ms
Total hot run time: 170559 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.9 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0758aaec110e8f73ac79e92b68fbb0eb7ff9c12d, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.04
query3	0.22	0.05	0.05
query4	1.68	0.06	0.06
query5	0.50	0.50	0.48
query6	1.14	0.72	0.71
query7	0.02	0.02	0.01
query8	0.06	0.05	0.04
query9	0.57	0.51	0.54
query10	0.58	0.55	0.57
query11	0.15	0.12	0.12
query12	0.15	0.12	0.12
query13	0.61	0.61	0.61
query14	0.78	0.80	0.79
query15	0.91	0.86	0.86
query16	0.35	0.36	0.36
query17	0.99	0.97	1.02
query18	0.22	0.21	0.20
query19	1.80	1.70	1.73
query20	0.01	0.00	0.01
query21	15.41	0.78	0.66
query22	4.11	8.65	1.26
query23	17.98	1.26	1.37
query24	2.25	0.22	0.22
query25	0.19	0.09	0.09
query26	0.31	0.22	0.22
query27	0.46	0.24	0.23
query28	13.16	0.99	0.97
query29	12.57	3.26	3.27
query30	0.26	0.05	0.06
query31	2.89	0.41	0.41
query32	3.22	0.49	0.49
query33	2.91	2.98	3.01
query34	15.44	4.26	4.24
query35	4.28	4.27	4.26
query36	0.68	0.48	0.48
query37	0.20	0.16	0.16
query38	0.18	0.15	0.15
query39	0.04	0.04	0.03
query40	0.16	0.14	0.14
query41	0.09	0.05	0.05
query42	0.05	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 107.73 s
Total hot run time: 29.9 s

@morrySnow
Copy link
Contributor Author

run p1

@morrySnow morrySnow removed the doing label Aug 1, 2024
@morrySnow
Copy link
Contributor Author

run cloud_p0

Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

github-actions bot commented Aug 1, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 1, 2024
@morrySnow morrySnow merged commit e66e01f into apache:master Aug 1, 2024
30 of 31 checks passed
@morrySnow morrySnow deleted the fix_dead_lock branch August 1, 2024 12:08
morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request Aug 1, 2024
)

pick from master apache#38660

insert will hold readlock of target table before planning. if nereids
need db readlock after it, will lead to dead lock. because other
statement need to hold db lock before get table lock

for example:

insert: target table read lock -> database read lock
drop table: database write lock -> target table write lock
morrySnow added a commit to morrySnow/incubator-doris that referenced this pull request Aug 1, 2024
)

pick from master apache#38660

insert will hold readlock of target table before planning. if nereids
need db readlock after it, will lead to dead lock. because other
statement need to hold db lock before get table lock

for example:

insert: target table read lock -> database read lock
drop table: database write lock -> target table write lock
yiguolei pushed a commit that referenced this pull request Aug 2, 2024
…38729)

pick from master #38660

insert will hold readlock of target table before planning. if nereids
need db readlock after it, will lead to dead lock. because other
statement need to hold db lock before get table lock

for example:

insert: target table read lock -> database read lock
drop table: database write lock -> target table write lock
dataroaring pushed a commit that referenced this pull request Aug 2, 2024
insert will hold readlock of target table before planning. if nereids
need db readlock after it, will lead to dead lock. because other
statement need to hold db lock before get table lock

for example:

insert: target table read lock -> database read lock
drop table: database write lock -> target table write lock
feiniaofeiafei pushed a commit to feiniaofeiafei/doris that referenced this pull request Aug 9, 2024
)

insert will hold readlock of target table before planning. if nereids
need db readlock after it, will lead to dead lock. because other
statement need to hold db lock before get table lock

for example:

insert: target table read lock -> database read lock
drop table: database write lock -> target table write lock
morningman pushed a commit to morningman/doris that referenced this pull request Aug 18, 2024
) (apache#38729)

pick from master apache#38660

insert will hold readlock of target table before planning. if nereids
need db readlock after it, will lead to dead lock. because other
statement need to hold db lock before get table lock

for example:

insert: target table read lock -> database read lock
drop table: database write lock -> target table write lock
@yiguolei yiguolei mentioned this pull request Sep 5, 2024
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.14-merged dev/2.1.6-merged dev/3.0.1-merged p0_b reviewed usercase Important user case type label
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants