Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](fe) fix several blocking bugs #37756

Merged
merged 2 commits into from
Jul 15, 2024
Merged

[fix](fe) fix several blocking bugs #37756

merged 2 commits into from
Jul 15, 2024

Conversation

morningman
Copy link
Contributor

@morningman morningman commented Jul 14, 2024 •
edited
Loading

  1. Redundant hashCode of FunctionCallExpr.java
    This line is redundant, and will cause the amount of hashcode calculation to grow exponentially.

  2. A potential deadlock of external catalog
    The following case may causing this deadlock:

    • high frequency load.
    • querying external table join inner table on non-master FE.
    • refresh external catalog frequently.

  3. A workaround to avoid FE restart failure because db does not found
    introduced from [opt](catalog) support using loading cache for db/table list in external catalog #33610 and fixed in [fix](catalog) fix wrong check when using "use_meta_cache=true" #36530.
    This PR is to make previous metadata restart successfully.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@morningman morningman mentioned this pull request Jul 14, 2024
Merged
@morningman
Copy link
Contributor Author

run buildall

@morningman morningman mentioned this pull request Jul 14, 2024
Merged
@doris-robot
Copy link

TPC-H: Total hot run time: 39765 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 55200fa645dff84214b6cf38684dcb90394216ac, data reload: false

------ Round 1 ----------------------------------
q1	17970	4525	4346	4346
q2	2504	233	186	186
q3	10953	1167	1109	1109
q4	10242	734	702	702
q5	7576	2721	2716	2716
q6	220	144	142	142
q7	959	609	609	609
q8	9252	2103	2094	2094
q9	9097	6564	6540	6540
q10	8710	3768	3743	3743
q11	455	239	233	233
q12	398	232	220	220
q13	18747	2964	2954	2954
q14	284	237	234	234
q15	522	487	492	487
q16	497	377	369	369
q17	957	597	629	597
q18	8092	7598	7389	7389
q19	9122	1315	1397	1315
q20	689	333	315	315
q21	5205	3182	3831	3182
q22	345	296	283	283
Total cold run time: 122796 ms
Total hot run time: 39765 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4431	4219	4228	4219
q2	363	284	262	262
q3	3002	2724	2742	2724
q4	1907	1542	1617	1542
q5	5304	5260	5303	5260
q6	225	129	130	129
q7	2085	1737	1726	1726
q8	3169	3323	3307	3307
q9	8400	8331	8469	8331
q10	3937	3651	3723	3651
q11	569	493	499	493
q12	799	629	602	602
q13	16367	2950	2981	2950
q14	299	277	267	267
q15	505	469	479	469
q16	473	403	407	403
q17	1746	1482	1488	1482
q18	7609	7533	7251	7251
q19	1630	1593	1598	1593
q20	1978	1784	1822	1784
q21	4839	4691	4762	4691
q22	599	506	486	486
Total cold run time: 70236 ms
Total hot run time: 53622 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 171724 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 55200fa645dff84214b6cf38684dcb90394216ac, data reload: false

query1	913	381	362	362
query2	6458	1958	1849	1849
query3	6652	203	225	203
query4	28450	17495	17388	17388
query5	4173	492	477	477
query6	290	176	169	169
query7	4612	286	278	278
query8	244	190	194	190
query9	8413	2395	2387	2387
query10	431	275	275	275
query11	10550	9918	10072	9918
query12	128	88	80	80
query13	1651	380	356	356
query14	10025	7137	7776	7137
query15	206	163	163	163
query16	7648	324	312	312
query17	1566	564	521	521
query18	1919	277	267	267
query19	188	147	149	147
query20	88	81	81	81
query21	203	129	128	128
query22	4347	4036	3945	3945
query23	33635	33333	32910	32910
query24	12069	2927	2868	2868
query25	676	389	403	389
query26	1774	142	143	142
query27	2909	266	264	264
query28	7520	1967	1959	1959
query29	1058	620	603	603
query30	282	148	145	145
query31	958	739	757	739
query32	95	53	57	53
query33	765	299	292	292
query34	956	491	492	491
query35	684	570	593	570
query36	1091	927	974	927
query37	177	77	83	77
query38	2865	2755	2722	2722
query39	874	787	819	787
query40	282	119	118	118
query41	45	46	45	45
query42	109	97	99	97
query43	525	450	480	450
query44	1240	742	733	733
query45	197	161	160	160
query46	1084	715	703	703
query47	1827	1778	1785	1778
query48	371	287	298	287
query49	1180	417	408	408
query50	762	407	392	392
query51	6825	6802	6751	6751
query52	101	88	96	88
query53	359	286	299	286
query54	1032	453	449	449
query55	74	75	75	75
query56	289	297	270	270
query57	1156	1059	1029	1029
query58	263	239	247	239
query59	2857	2656	2668	2656
query60	310	270	275	270
query61	100	98	93	93
query62	857	630	627	627
query63	324	299	285	285
query64	10514	2303	1654	1654
query65	3279	3112	3100	3100
query66	1384	333	336	333
query67	15395	15005	15007	15005
query68	4576	542	536	536
query69	470	323	313	313
query70	1192	1089	1155	1089
query71	399	272	269	269
query72	7397	5648	5789	5648
query73	749	319	324	319
query74	5939	5489	5476	5476
query75	3433	2712	2677	2677
query76	2653	998	923	923
query77	459	312	298	298
query78	9534	8968	8844	8844
query79	2554	519	525	519
query80	1895	502	487	487
query81	577	218	221	218
query82	719	137	130	130
query83	280	175	173	173
query84	284	93	98	93
query85	2068	375	354	354
query86	502	316	312	312
query87	3286	3119	3134	3119
query88	4225	2497	2462	2462
query89	471	375	393	375
query90	1853	197	200	197
query91	139	112	113	112
query92	72	50	49	49
query93	2449	493	491	491
query94	1273	234	223	223
query95	422	328	329	328
query96	592	279	275	275
query97	3178	3000	3074	3000
query98	225	205	200	200
query99	1614	1303	1263	1263
Total cold run time: 285763 ms
Total hot run time: 171724 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.55 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 55200fa645dff84214b6cf38684dcb90394216ac, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.03
query3	0.22	0.05	0.04
query4	1.71	0.06	0.06
query5	0.48	0.46	0.50
query6	1.13	0.73	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.48	0.48
query10	0.56	0.55	0.54
query11	0.14	0.12	0.11
query12	0.14	0.12	0.12
query13	0.59	0.58	0.58
query14	0.76	0.77	0.78
query15	0.85	0.82	0.82
query16	0.34	0.34	0.37
query17	1.03	0.96	0.98
query18	0.23	0.22	0.22
query19	1.76	1.64	1.67
query20	0.01	0.00	0.01
query21	15.39	0.73	0.67
query22	4.31	6.68	2.03
query23	18.25	1.36	1.30
query24	2.10	0.23	0.22
query25	0.14	0.09	0.08
query26	0.30	0.21	0.22
query27	0.45	0.22	0.22
query28	13.25	1.01	1.00
query29	12.62	3.26	3.26
query30	0.26	0.06	0.06
query31	2.88	0.39	0.38
query32	3.30	0.48	0.47
query33	2.85	2.93	2.95
query34	16.98	4.40	4.30
query35	4.43	4.42	4.44
query36	0.65	0.45	0.48
query37	0.18	0.15	0.15
query38	0.15	0.15	0.14
query39	0.05	0.04	0.04
query40	0.16	0.12	0.12
query41	0.10	0.04	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.58 s
Total hot run time: 30.55 s

morningman added a commit that referenced this pull request Jul 14, 2024
@morningman
[fix](fe) fix several blocking bugs #37756 (#37758)
0bcc177 
bp #37756
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

kaka11chen
kaka11chen approved these changes Jul 15, 2024
View reviewed changes
Copy link
Contributor

@kaka11chen kaka11chen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 15, 2024
@morningman morningman merged commit d5afe54 into apache:master Jul 15, 2024
26 of 29 checks passed
morningman added a commit that referenced this pull request Jul 15, 2024
@morningman
[fix](fe) fix several blocking bugs #37756 (#37757)
ff7a040 
bp #37756
wsjz pushed a commit to wsjz/incubator-doris that referenced this pull request Jul 15, 2024
@morningman @wsjz
[fix](fe) fix several blocking bugs (apache#37756)
d0b1910 
1. Redundant hashCode of FunctionCallExpr.java
This line is redundant, and will cause the amount of hashcode
calculation to grow exponentially.

2. A potential deadlock of external catalog
    The following case may causing this deadlock:
    - high frequency load.
    - querying external table join inner table on non-master FE. 
    - refresh external catalog frequently.

3. A workaround to avoid FE restart failure because db does not found
    introduced from apache#33610 and fixed in apache#36530.
    This PR is to make previous metadata restart successfully.
seawinde pushed a commit to seawinde/doris that referenced this pull request Jul 17, 2024
@morningman @seawinde
[fix](fe) fix several blocking bugs (apache#37756)
ab70620 
1. Redundant hashCode of FunctionCallExpr.java
This line is redundant, and will cause the amount of hashcode
calculation to grow exponentially.

2. A potential deadlock of external catalog
    The following case may causing this deadlock:
    - high frequency load.
    - querying external table join inner table on non-master FE. 
    - refresh external catalog frequently.

3. A workaround to avoid FE restart failure because db does not found
    introduced from apache#33610 and fixed in apache#36530.
    This PR is to make previous metadata restart successfully.
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@morningman @dataroaring
[fix](fe) fix several blocking bugs (#37756)
1cf8ba4 
1. Redundant hashCode of FunctionCallExpr.java
This line is redundant, and will cause the amount of hashcode
calculation to grow exponentially.

2. A potential deadlock of external catalog
    The following case may causing this deadlock:
    - high frequency load.
    - querying external table join inner table on non-master FE. 
    - refresh external catalog frequently.

3. A workaround to avoid FE restart failure because db does not found
    introduced from #33610 and fixed in #36530.
    This PR is to make previous metadata restart successfully.
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@morningman
[fix](fe) fix several blocking bugs apache#37756 (apache#37758)
02c2836 
bp apache#37756
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wuwenchi wuwenchi wuwenchi approved these changes

@kaka11chen kaka11chen kaka11chen approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.13-merged dev/2.1.5-merged dev/3.0.1-merged p0_b reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@morningman @doris-robot @kaka11chen @wuwenchi @dataroaring