Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fix](Group commit) Fix group commit forward fault #38228

Merged
merged 1 commit into from
Jul 23, 2024
Merged

[Fix](Group commit) Fix group commit forward fault #38228

merged 1 commit into from
Jul 23, 2024

Conversation

Yukang-Lian
Copy link
Collaborator

Proposed changes

When client connect to observer and do group commit, observer will forward to master. The forward parameter GroupCommitInfo maybe null, which leads to an exception.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Yukang-Lian
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40039 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c4b0c583b12a24e8ff50a73445bc70a2f4b8dc72, data reload: false

------ Round 1 ----------------------------------
q1	17630	4327	4357	4327
q2	2018	195	187	187
q3	10451	1227	1120	1120
q4	10191	747	847	747
q5	7520	2692	2755	2692
q6	217	135	138	135
q7	945	600	608	600
q8	9211	2053	2070	2053
q9	8728	6556	6565	6556
q10	8808	3754	3786	3754
q11	467	232	237	232
q12	453	229	228	228
q13	17765	2952	2984	2952
q14	289	242	237	237
q15	524	481	500	481
q16	504	376	371	371
q17	957	744	717	717
q18	8075	7470	7397	7397
q19	2517	1440	1373	1373
q20	683	315	346	315
q21	4911	3272	3340	3272
q22	352	293	299	293
Total cold run time: 113216 ms
Total hot run time: 40039 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4394	4215	4268	4215
q2	381	276	267	267
q3	2966	2819	2794	2794
q4	1981	1696	1693	1693
q5	5515	5717	5471	5471
q6	241	134	134	134
q7	2192	1846	1823	1823
q8	3350	3408	3397	3397
q9	8826	9217	9262	9217
q10	4115	3887	3906	3887
q11	565	502	502	502
q12	789	597	633	597
q13	15844	3184	3171	3171
q14	319	274	286	274
q15	535	479	504	479
q16	496	457	443	443
q17	1825	1545	1525	1525
q18	7983	7840	7839	7839
q19	1798	1514	1556	1514
q20	2076	1878	1930	1878
q21	5067	4779	5030	4779
q22	570	500	490	490
Total cold run time: 71828 ms
Total hot run time: 56389 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173884 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c4b0c583b12a24e8ff50a73445bc70a2f4b8dc72, data reload: false

query1	907	390	365	365
query2	6436	1913	1857	1857
query3	6637	204	216	204
query4	24220	17365	17382	17365
query5	3629	474	502	474
query6	269	191	169	169
query7	4576	285	282	282
query8	244	197	203	197
query9	8515	2408	2405	2405
query10	447	287	278	278
query11	10552	9967	10161	9967
query12	117	85	88	85
query13	1644	382	362	362
query14	10355	8316	7629	7629
query15	229	166	165	165
query16	7722	437	438	437
query17	1545	560	513	513
query18	1377	276	263	263
query19	193	148	146	146
query20	92	82	80	80
query21	207	132	124	124
query22	4244	4137	3911	3911
query23	34227	33721	34254	33721
query24	10915	3032	2927	2927
query25	611	383	388	383
query26	701	147	145	145
query27	2238	276	281	276
query28	6391	2035	2036	2035
query29	891	638	632	632
query30	247	151	153	151
query31	981	777	758	758
query32	100	53	54	53
query33	754	334	331	331
query34	926	493	485	485
query35	856	793	729	729
query36	1158	970	985	970
query37	139	81	85	81
query38	2898	2995	2835	2835
query39	926	887	834	834
query40	203	120	121	120
query41	48	49	44	44
query42	117	96	102	96
query43	498	476	463	463
query44	1169	765	719	719
query45	195	163	160	160
query46	1090	727	748	727
query47	1830	1795	1790	1790
query48	362	288	283	283
query49	822	404	406	404
query50	768	399	386	386
query51	6852	6733	6689	6689
query52	107	93	96	93
query53	352	291	285	285
query54	886	439	448	439
query55	74	74	72	72
query56	282	277	269	269
query57	1142	1078	1056	1056
query58	251	251	256	251
query59	2824	2628	2673	2628
query60	294	280	307	280
query61	96	95	124	95
query62	781	645	649	645
query63	311	292	291	291
query64	9137	2222	1652	1652
query65	3154	3078	3080	3078
query66	680	330	329	329
query67	15345	15294	14868	14868
query68	5108	529	528	528
query69	655	447	347	347
query70	1135	1131	1134	1131
query71	447	278	275	275
query72	7083	5827	5680	5680
query73	773	314	333	314
query74	6345	5714	5674	5674
query75	3404	2728	2688	2688
query76	3717	953	957	953
query77	656	307	317	307
query78	11102	9029	9007	9007
query79	2611	510	507	507
query80	2032	472	484	472
query81	587	220	220	220
query82	731	138	134	134
query83	270	168	169	168
query84	272	86	84	84
query85	988	313	301	301
query86	404	308	320	308
query87	3286	3121	3116	3116
query88	3766	2344	2332	2332
query89	483	391	393	391
query90	1811	191	190	190
query91	137	97	100	97
query92	55	49	48	48
query93	3202	484	486	484
query94	825	276	295	276
query95	406	321	309	309
query96	603	275	269	269
query97	3213	3030	3005	3005
query98	295	196	186	186
query99	1542	1285	1247	1247
Total cold run time: 276730 ms
Total hot run time: 173884 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.73 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c4b0c583b12a24e8ff50a73445bc70a2f4b8dc72, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.05
query4	1.66	0.10	0.09
query5	0.50	0.50	0.49
query6	1.13	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.54	0.48	0.48
query10	0.54	0.55	0.53
query11	0.16	0.11	0.11
query12	0.14	0.12	0.13
query13	0.60	0.58	0.58
query14	0.76	0.77	0.77
query15	0.86	0.80	0.82
query16	0.37	0.37	0.37
query17	0.96	0.98	1.01
query18	0.23	0.22	0.23
query19	1.83	1.65	1.65
query20	0.01	0.02	0.02
query21	15.38	0.75	0.66
query22	4.29	6.85	1.98
query23	18.23	1.37	1.28
query24	2.13	0.23	0.21
query25	0.16	0.09	0.08
query26	0.30	0.21	0.20
query27	0.46	0.23	0.22
query28	13.37	1.04	1.00
query29	12.66	3.35	3.31
query30	0.26	0.05	0.06
query31	2.86	0.39	0.39
query32	3.26	0.48	0.49
query33	2.86	2.95	2.96
query34	16.89	4.37	4.36
query35	4.41	4.42	4.40
query36	0.65	0.48	0.47
query37	0.18	0.16	0.16
query38	0.15	0.15	0.15
query39	0.04	0.03	0.04
query40	0.15	0.12	0.13
query41	0.10	0.04	0.04
query42	0.06	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.6 s
Total hot run time: 30.73 s

dataroaring
dataroaring approved these changes Jul 23, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 23, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

zhannngchen
zhannngchen approved these changes Jul 23, 2024
View reviewed changes
Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 0a66a51 into apache:master Jul 23, 2024
30 of 32 checks passed
dataroaring pushed a commit that referenced this pull request Jul 23, 2024
@Yukang-Lian @dataroaring
[Fix](Group commit) Fix group commit forward fault (#38228)
6ee0bfa 
## Proposed changes

<!--Describe your changes.-->

When client connect to observer and do group commit, observer will
forward to master. The forward parameter `GroupCommitInfo` maybe null,
which leads to an exception.
Yukang-Lian added a commit to Yukang-Lian/doris that referenced this pull request Jul 25, 2024
@Yukang-Lian
[Fix](Group commit) Fix group commit forward fault (apache#38228)
11202e3 
## Proposed changes

<!--Describe your changes.-->

When client connect to observer and do group commit, observer will
forward to master. The forward parameter `GroupCommitInfo` maybe null,
which leads to an exception.
@Yukang-Lian Yukang-Lian mentioned this pull request Jul 25, 2024
Merged
@mymeiyi mymeiyi mentioned this pull request Jul 25, 2024
Merged
yiguolei pushed a commit that referenced this pull request Jul 25, 2024
@Yukang-Lian
[cherry-pick](branch-2.1) Fix some group commit forward to master pro…
3253011 
…blems (#38345)

## Proposed changes

<!--Describe your changes.-->

Pick #38228 #38265
dataroaring pushed a commit that referenced this pull request Jul 25, 2024
@mymeiyi
[fix](group commit) Pick some group commit pr (#38320)
91f0301 
Pick #38292,
#34021,
#38228, some modify of
#37260, some modify of
#37595
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring dataroaring approved these changes

@bobhan1 bobhan1 bobhan1 approved these changes

@zhannngchen zhannngchen zhannngchen approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.1-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@Yukang-Lian @doris-robot @zhannngchen @bobhan1 @dataroaring @yiguolei