Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](move-memtable) check segment num when closing each tablet #36753

Merged
merged 6 commits into from
Jul 8, 2024
Merged

[fix](move-memtable) check segment num when closing each tablet #36753

merged 6 commits into from
Jul 8, 2024

Conversation

kaijchen
Copy link
Contributor

Proposed changes

Previously, there is chance that sender failed to send some data while the receiver being unaware of.
This will cause lost data if some segments are skipped.

This PR fixes the problem by including checks in both sender and receiver.
When sender failed to send rpc, LoadStreamStub will mark the involved tablets failed.
Each sender will send segment num for each tablet in CLOSE_LOAD,
and receivers (LoadStream) will sum up and check total segment nums.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@kaijchen
Copy link
Contributor Author

run buildall

@kaijchen
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40062 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 20a47dc5952d69b1201eb9b24edb6fc0bd343a91, data reload: false

------ Round 1 ----------------------------------
q1	19057	6770	4363	4363
q2	2019	193	192	192
q3	10559	1086	1207	1086
q4	10269	811	814	811
q5	7470	2669	2604	2604
q6	223	132	134	132
q7	966	604	625	604
q8	9230	2085	2093	2085
q9	9033	6536	6501	6501
q10	9083	3712	3700	3700
q11	466	250	245	245
q12	464	223	225	223
q13	18850	3009	2981	2981
q14	270	219	208	208
q15	521	495	476	476
q16	541	373	373	373
q17	988	642	669	642
q18	8089	7535	7435	7435
q19	4930	1595	1439	1439
q20	657	317	330	317
q21	4846	3296	3892	3296
q22	406	357	349	349
Total cold run time: 118937 ms
Total hot run time: 40062 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4460	4294	4220	4220
q2	384	276	286	276
q3	3131	2960	2830	2830
q4	2082	1780	1703	1703
q5	5490	5500	5572	5500
q6	226	137	133	133
q7	2285	1846	1888	1846
q8	3285	3445	3432	3432
q9	8778	8903	8783	8783
q10	3997	3789	3816	3789
q11	609	506	498	498
q12	826	637	617	617
q13	17140	3167	3140	3140
q14	306	286	270	270
q15	544	487	485	485
q16	490	427	422	422
q17	1819	1509	1489	1489
q18	8266	7983	7829	7829
q19	1849	1617	1534	1534
q20	2730	1890	1846	1846
q21	5160	4817	5000	4817
q22	639	554	605	554
Total cold run time: 74496 ms
Total hot run time: 56013 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174365 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 20a47dc5952d69b1201eb9b24edb6fc0bd343a91, data reload: false

query1	924	381	402	381
query2	6399	2317	2259	2259
query3	6613	204	200	200
query4	19331	17194	17352	17194
query5	3582	462	489	462
query6	251	160	153	153
query7	4574	291	288	288
query8	322	309	311	309
query9	8564	2508	2439	2439
query10	550	315	274	274
query11	10574	10035	9855	9855
query12	111	82	83	82
query13	1629	372	376	372
query14	10093	7628	7654	7628
query15	240	188	193	188
query16	7816	278	269	269
query17	1895	554	532	532
query18	1970	282	280	280
query19	193	152	163	152
query20	88	84	88	84
query21	211	134	127	127
query22	4357	3953	4101	3953
query23	33581	33706	33574	33574
query24	10739	3008	2827	2827
query25	598	435	395	395
query26	707	157	169	157
query27	2253	326	330	326
query28	6082	2161	2166	2161
query29	896	666	644	644
query30	292	169	159	159
query31	955	754	751	751
query32	99	57	59	57
query33	674	309	311	309
query34	894	487	484	484
query35	761	670	647	647
query36	1092	956	978	956
query37	143	81	76	76
query38	2973	2799	2855	2799
query39	906	805	842	805
query40	295	134	134	134
query41	58	54	51	51
query42	111	100	103	100
query43	588	522	555	522
query44	1103	721	721	721
query45	187	162	163	162
query46	1073	712	715	712
query47	1860	1773	1779	1773
query48	383	295	302	295
query49	884	401	409	401
query50	772	392	382	382
query51	6772	6760	6740	6740
query52	103	92	95	92
query53	376	285	277	277
query54	864	445	439	439
query55	73	74	74	74
query56	283	264	266	264
query57	1130	1073	1072	1072
query58	251	240	250	240
query59	3428	3189	3060	3060
query60	302	263	285	263
query61	113	91	95	91
query62	598	444	438	438
query63	319	285	293	285
query64	8550	2257	1777	1777
query65	3201	3130	3110	3110
query66	751	321	333	321
query67	15501	15035	14855	14855
query68	5910	550	547	547
query69	695	477	392	392
query70	1170	1172	1142	1142
query71	449	269	263	263
query72	8892	5534	5655	5534
query73	775	325	328	325
query74	5860	5538	5472	5472
query75	4655	2644	2626	2626
query76	3412	985	911	911
query77	742	306	300	300
query78	10486	9831	9890	9831
query79	4344	517	514	514
query80	2614	486	477	477
query81	558	217	225	217
query82	1465	110	103	103
query83	326	173	168	168
query84	275	88	89	88
query85	1401	285	271	271
query86	456	321	327	321
query87	3300	3101	3115	3101
query88	4599	2465	2432	2432
query89	492	382	369	369
query90	1831	194	194	194
query91	134	101	100	100
query92	60	53	52	52
query93	5340	528	525	525
query94	1199	193	196	193
query95	401	323	320	320
query96	608	274	270	270
query97	3185	3039	3011	3011
query98	231	200	195	195
query99	1125	863	842	842
Total cold run time: 280851 ms
Total hot run time: 174365 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.24 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 20a47dc5952d69b1201eb9b24edb6fc0bd343a91, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.05
query3	0.22	0.05	0.05
query4	1.67	0.09	0.07
query5	0.50	0.48	0.47
query6	1.14	0.72	0.72
query7	0.02	0.02	0.01
query8	0.04	0.03	0.05
query9	0.56	0.51	0.52
query10	0.56	0.56	0.57
query11	0.16	0.13	0.12
query12	0.16	0.12	0.12
query13	0.59	0.58	0.58
query14	0.75	0.78	0.78
query15	0.82	0.82	0.80
query16	0.35	0.36	0.37
query17	1.04	1.05	1.02
query18	0.22	0.24	0.25
query19	1.83	1.83	1.76
query20	0.01	0.00	0.00
query21	15.45	0.66	0.64
query22	4.65	5.62	2.67
query23	18.24	1.38	1.16
query24	2.12	0.22	0.23
query25	0.17	0.08	0.08
query26	0.26	0.17	0.18
query27	0.08	0.09	0.08
query28	13.22	1.02	0.99
query29	12.61	3.27	3.27
query30	0.26	0.07	0.05
query31	2.86	0.38	0.38
query32	3.29	0.47	0.46
query33	2.88	2.86	2.94
query34	17.20	4.40	4.45
query35	4.55	4.48	4.50
query36	0.64	0.47	0.46
query37	0.19	0.16	0.15
query38	0.15	0.14	0.14
query39	0.05	0.03	0.03
query40	0.17	0.14	0.14
query41	0.09	0.06	0.05
query42	0.05	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.99 s
Total hot run time: 31.24 s

@kaijchen
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40390 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4a12ca8f205883dbbd9126b63eddf9957ef80d27, data reload: false

------ Round 1 ----------------------------------
q1	18361	4860	4473	4473
q2	2634	202	206	202
q3	11155	1079	1261	1079
q4	10645	760	754	754
q5	8509	2722	2702	2702
q6	225	140	136	136
q7	974	617	647	617
q8	9705	2111	2085	2085
q9	9079	6529	6466	6466
q10	9072	3729	3723	3723
q11	436	249	250	249
q12	450	237	225	225
q13	18062	3017	2997	2997
q14	274	220	226	220
q15	528	491	478	478
q16	538	376	390	376
q17	1001	645	687	645
q18	7948	7543	7424	7424
q19	7223	1539	1511	1511
q20	659	320	325	320
q21	4961	3370	4435	3370
q22	404	339	338	338
Total cold run time: 122843 ms
Total hot run time: 40390 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4387	4258	4265	4258
q2	357	270	267	267
q3	2993	2745	2701	2701
q4	1919	1637	1661	1637
q5	5227	5329	5286	5286
q6	227	129	128	128
q7	2154	1801	1782	1782
q8	3193	3382	3337	3337
q9	8367	8366	8363	8363
q10	3923	3652	3721	3652
q11	581	498	500	498
q12	808	607	602	602
q13	17484	3016	3032	3016
q14	294	284	272	272
q15	523	485	491	485
q16	483	418	424	418
q17	1808	1507	1473	1473
q18	7732	7712	7317	7317
q19	1688	1563	1626	1563
q20	1962	1796	1778	1778
q21	4895	4770	4898	4770
q22	641	542	554	542
Total cold run time: 71646 ms
Total hot run time: 54145 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174086 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4a12ca8f205883dbbd9126b63eddf9957ef80d27, data reload: false

query1	927	379	373	373
query2	6430	2493	2315	2315
query3	6641	195	200	195
query4	20639	17286	17326	17286
query5	4143	465	480	465
query6	253	158	154	154
query7	4592	297	295	295
query8	333	287	283	283
query9	8530	2412	2411	2411
query10	610	296	287	287
query11	10607	10097	10099	10097
query12	139	83	81	81
query13	1632	390	384	384
query14	9562	7812	7600	7600
query15	240	200	189	189
query16	8119	292	275	275
query17	1879	560	532	532
query18	2099	284	282	282
query19	213	161	162	161
query20	89	83	83	83
query21	212	131	128	128
query22	4352	4077	3980	3980
query23	33763	32972	33087	32972
query24	11133	2926	2872	2872
query25	631	386	388	386
query26	1491	165	164	164
query27	2903	324	327	324
query28	7274	2107	2107	2107
query29	955	640	660	640
query30	298	155	159	155
query31	969	744	752	744
query32	95	64	58	58
query33	802	302	306	302
query34	908	475	476	475
query35	774	649	632	632
query36	1109	958	914	914
query37	158	74	77	74
query38	2892	2749	2721	2721
query39	849	796	817	796
query40	251	130	136	130
query41	57	60	57	57
query42	121	103	103	103
query43	596	571	546	546
query44	1176	745	751	745
query45	201	164	169	164
query46	1086	741	737	737
query47	1847	1763	1756	1756
query48	395	325	302	302
query49	1171	426	428	426
query50	778	393	401	393
query51	6885	6777	6761	6761
query52	106	100	96	96
query53	372	294	298	294
query54	930	463	449	449
query55	78	76	76	76
query56	304	277	282	277
query57	1161	1076	1052	1052
query58	272	259	278	259
query59	3454	3291	3135	3135
query60	334	290	291	290
query61	115	108	111	108
query62	667	553	437	437
query63	326	290	289	289
query64	9938	2242	1750	1750
query65	3194	3101	3088	3088
query66	1373	353	339	339
query67	15901	14821	14980	14821
query68	8357	581	566	566
query69	753	463	423	423
query70	1190	1106	1144	1106
query71	521	277	269	269
query72	9082	5576	5847	5576
query73	1109	328	336	328
query74	5907	5595	5479	5479
query75	5324	2684	2662	2662
query76	4928	953	957	953
query77	754	311	311	311
query78	10476	10033	9642	9642
query79	7859	515	523	515
query80	1132	486	466	466
query81	560	217	217	217
query82	795	102	105	102
query83	333	166	166	166
query84	268	82	87	82
query85	1391	293	306	293
query86	463	313	317	313
query87	3266	3091	3090	3090
query88	4666	2363	2362	2362
query89	505	380	370	370
query90	1900	186	191	186
query91	131	99	99	99
query92	70	47	47	47
query93	5938	541	526	526
query94	1166	187	189	187
query95	402	313	309	309
query96	599	277	259	259
query97	3198	3069	3091	3069
query98	218	197	202	197
query99	1185	847	844	844
Total cold run time: 296780 ms
Total hot run time: 174086 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.49 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4a12ca8f205883dbbd9126b63eddf9957ef80d27, data reload: false

query1	0.04	0.04	0.03
query2	0.09	0.04	0.04
query3	0.23	0.04	0.05
query4	1.70	0.09	0.10
query5	0.49	0.49	0.48
query6	1.12	0.73	0.72
query7	0.02	0.02	0.01
query8	0.06	0.04	0.04
query9	0.54	0.50	0.48
query10	0.54	0.55	0.55
query11	0.17	0.11	0.11
query12	0.14	0.11	0.12
query13	0.60	0.59	0.60
query14	0.77	0.77	0.81
query15	0.82	0.80	0.81
query16	0.36	0.38	0.36
query17	1.06	1.03	1.03
query18	0.20	0.26	0.25
query19	1.76	1.75	1.72
query20	0.01	0.01	0.01
query21	15.44	0.64	0.64
query22	4.14	7.53	1.87
query23	18.22	1.43	1.26
query24	2.15	0.22	0.23
query25	0.15	0.08	0.08
query26	0.26	0.18	0.17
query27	0.08	0.08	0.07
query28	13.23	1.00	0.99
query29	12.62	3.28	3.27
query30	0.26	0.08	0.07
query31	2.82	0.39	0.38
query32	3.27	0.47	0.46
query33	2.92	2.87	2.98
query34	17.04	4.44	4.39
query35	4.47	4.44	4.47
query36	0.64	0.47	0.48
query37	0.18	0.15	0.16
query38	0.15	0.15	0.15
query39	0.04	0.03	0.04
query40	0.18	0.14	0.15
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.19 s
Total hot run time: 30.49 s

@kaijchen
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@kaijchen
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39941 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 088393884b14df6e3a33b318ed07ba79b76a3448, data reload: false

------ Round 1 ----------------------------------
q1	18559	4693	4392	4392
q2	2025	196	197	196
q3	10533	1120	1198	1120
q4	10262	790	778	778
q5	7498	2659	2763	2659
q6	216	132	131	131
q7	943	596	603	596
q8	9231	2107	2054	2054
q9	8945	6482	6486	6482
q10	8930	3725	3767	3725
q11	458	233	229	229
q12	455	221	224	221
q13	17855	2934	2968	2934
q14	250	223	224	223
q15	511	480	473	473
q16	522	370	374	370
q17	968	689	651	651
q18	8072	7451	7404	7404
q19	4180	1574	1524	1524
q20	658	316	324	316
q21	4911	3136	3174	3136
q22	391	327	327	327
Total cold run time: 116373 ms
Total hot run time: 39941 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4363	4224	4193	4193
q2	376	267	263	263
q3	2997	2798	2819	2798
q4	1980	1749	1719	1719
q5	5658	5504	5502	5502
q6	214	127	124	124
q7	2191	1826	1847	1826
q8	3284	3412	3424	3412
q9	8727	8667	8872	8667
q10	4157	3910	3747	3747
q11	593	509	499	499
q12	826	603	625	603
q13	16007	3166	3170	3166
q14	307	295	256	256
q15	534	481	482	481
q16	476	418	436	418
q17	1845	1535	1491	1491
q18	8058	7796	7740	7740
q19	1838	1666	1648	1648
q20	3050	1912	1875	1875
q21	5273	5009	4925	4925
q22	645	553	555	553
Total cold run time: 73399 ms
Total hot run time: 55906 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 171429 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 088393884b14df6e3a33b318ed07ba79b76a3448, data reload: false

query1	942	389	378	378
query2	6465	2370	2274	2274
query3	6638	210	217	210
query4	19385	17378	17231	17231
query5	3705	484	486	484
query6	264	173	183	173
query7	4603	302	295	295
query8	299	288	283	283
query9	8599	2462	2453	2453
query10	552	295	281	281
query11	10506	10076	9964	9964
query12	116	85	82	82
query13	1639	367	361	361
query14	9726	7138	7611	7138
query15	223	189	187	187
query16	7779	264	258	258
query17	1906	527	540	527
query18	1949	265	269	265
query19	195	148	150	148
query20	90	78	83	78
query21	210	126	122	122
query22	4310	4058	4084	4058
query23	33685	33529	33576	33529
query24	10682	2954	2865	2865
query25	609	413	393	393
query26	717	163	155	155
query27	2310	329	329	329
query28	6067	2234	2204	2204
query29	901	678	632	632
query30	258	166	168	166
query31	957	771	757	757
query32	106	59	62	59
query33	679	311	296	296
query34	879	481	498	481
query35	762	649	661	649
query36	1154	978	987	978
query37	144	79	77	77
query38	2919	2844	2797	2797
query39	950	858	820	820
query40	211	127	138	127
query41	53	51	59	51
query42	101	105	102	102
query43	601	555	537	537
query44	1087	757	742	742
query45	193	157	161	157
query46	1063	723	743	723
query47	1822	1786	1748	1748
query48	367	294	300	294
query49	848	402	437	402
query50	764	389	389	389
query51	6827	6828	6740	6740
query52	115	90	99	90
query53	353	288	295	288
query54	866	467	445	445
query55	75	74	72	72
query56	282	259	266	259
query57	1143	1016	1045	1016
query58	250	246	236	236
query59	3419	2982	3154	2982
query60	294	277	280	277
query61	92	113	95	95
query62	608	444	434	434
query63	316	293	290	290
query64	8564	2230	1742	1742
query65	3182	3090	3112	3090
query66	751	327	329	327
query67	15457	15087	14962	14962
query68	4843	573	543	543
query69	579	514	437	437
query70	1136	1155	1134	1134
query71	413	286	296	286
query72	8432	5304	2754	2754
query73	747	324	321	321
query74	5872	5418	5434	5418
query75	3651	2653	2732	2653
query76	3165	941	973	941
query77	703	307	310	307
query78	10337	9981	9917	9917
query79	2397	521	514	514
query80	2410	469	471	469
query81	612	221	218	218
query82	824	109	113	109
query83	332	167	169	167
query84	266	91	86	86
query85	1575	298	276	276
query86	489	332	307	307
query87	3246	3123	3076	3076
query88	4302	2393	2376	2376
query89	490	403	390	390
query90	1696	188	193	188
query91	135	102	105	102
query92	62	50	50	50
query93	2256	522	519	519
query94	1131	192	195	192
query95	401	316	322	316
query96	600	266	271	266
query97	3212	3045	3012	3012
query98	231	198	194	194
query99	1144	861	837	837
Total cold run time: 271273 ms
Total hot run time: 171429 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.8 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 088393884b14df6e3a33b318ed07ba79b76a3448, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.06
query4	1.67	0.08	0.11
query5	0.50	0.48	0.48
query6	1.14	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.55	0.50	0.49
query10	0.55	0.54	0.54
query11	0.16	0.12	0.12
query12	0.14	0.12	0.13
query13	0.59	0.60	0.59
query14	0.75	0.78	0.78
query15	0.85	0.80	0.81
query16	0.37	0.36	0.39
query17	1.04	1.03	0.97
query18	0.23	0.25	0.25
query19	1.91	1.73	1.77
query20	0.01	0.01	0.02
query21	15.42	0.75	0.67
query22	4.10	7.24	2.12
query23	18.24	1.39	1.22
query24	2.06	0.23	0.21
query25	0.15	0.08	0.07
query26	0.27	0.18	0.17
query27	0.08	0.08	0.08
query28	13.27	1.02	0.98
query29	12.58	3.30	3.32
query30	0.26	0.06	0.05
query31	2.88	0.38	0.38
query32	3.27	0.46	0.46
query33	2.90	2.91	2.91
query34	16.88	4.44	4.48
query35	4.46	4.47	4.53
query36	0.65	0.46	0.47
query37	0.18	0.15	0.16
query38	0.16	0.15	0.15
query39	0.04	0.04	0.03
query40	0.18	0.13	0.15
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.09 s
Total hot run time: 30.8 s

@liaoxin01
Copy link
Contributor

Please add an injection regression case.

@kaijchen
Copy link
Contributor Author

kaijchen commented Jul 3, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 3, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40192 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 123ee5d29e8a145b35fadabe80e9373d5d9ec184, data reload: false

------ Round 1 ----------------------------------
q1	18112	5086	4304	4304
q2	2015	191	186	186
q3	10542	1178	1174	1174
q4	10230	913	728	728
q5	7472	2690	2733	2690
q6	219	136	139	136
q7	958	598	612	598
q8	9228	2112	2079	2079
q9	8996	6538	6504	6504
q10	8844	3715	3721	3715
q11	485	242	241	241
q12	461	241	250	241
q13	18245	3002	3016	3002
q14	266	230	220	220
q15	513	493	490	490
q16	509	396	375	375
q17	991	694	809	694
q18	8081	7546	7533	7533
q19	5317	1515	1458	1458
q20	673	336	321	321
q21	4967	3938	3165	3165
q22	402	338	344	338
Total cold run time: 117526 ms
Total hot run time: 40192 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4395	4209	4228	4209
q2	363	281	267	267
q3	3028	2915	3001	2915
q4	1976	1746	1791	1746
q5	5571	5515	5415	5415
q6	225	137	140	137
q7	2292	1895	1900	1895
q8	3253	3458	3427	3427
q9	8708	8799	8793	8793
q10	4142	3775	3846	3775
q11	581	485	506	485
q12	841	645	630	630
q13	17310	3148	3155	3148
q14	295	279	284	279
q15	539	485	483	483
q16	526	446	448	446
q17	1846	1558	1500	1500
q18	8140	7967	7802	7802
q19	1802	1705	1554	1554
q20	3004	1894	1857	1857
q21	5200	4756	4809	4756
q22	626	541	564	541
Total cold run time: 74663 ms
Total hot run time: 56060 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172934 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 123ee5d29e8a145b35fadabe80e9373d5d9ec184, data reload: false

query1	941	381	371	371
query2	6475	2445	2342	2342
query3	6632	204	208	204
query4	19361	17378	17249	17249
query5	3792	487	495	487
query6	260	168	176	168
query7	4608	298	302	298
query8	307	291	303	291
query9	8557	2465	2449	2449
query10	581	309	280	280
query11	10455	9861	9960	9861
query12	123	84	84	84
query13	1662	388	370	370
query14	9628	7115	6963	6963
query15	239	194	190	190
query16	7581	311	312	311
query17	1377	570	538	538
query18	1955	286	274	274
query19	211	158	161	158
query20	90	88	83	83
query21	218	126	128	126
query22	4210	4098	4024	4024
query23	33815	33729	33883	33729
query24	11254	2894	2941	2894
query25	649	396	414	396
query26	808	155	161	155
query27	2310	332	329	329
query28	6254	2193	2184	2184
query29	916	670	657	657
query30	304	180	164	164
query31	991	788	751	751
query32	98	59	56	56
query33	805	315	316	315
query34	985	492	510	492
query35	764	698	645	645
query36	1169	1026	966	966
query37	142	84	89	84
query38	2904	2792	2808	2792
query39	895	840	836	836
query40	213	130	138	130
query41	58	56	54	54
query42	121	104	104	104
query43	612	563	550	550
query44	1223	761	743	743
query45	186	170	169	169
query46	1083	726	747	726
query47	1854	1768	1768	1768
query48	381	305	304	304
query49	996	418	425	418
query50	764	394	383	383
query51	6887	6870	6745	6745
query52	111	90	103	90
query53	369	297	295	295
query54	885	460	455	455
query55	76	73	74	73
query56	279	262	277	262
query57	1115	1081	1053	1053
query58	248	246	252	246
query59	3363	3459	3026	3026
query60	302	292	311	292
query61	97	95	92	92
query62	586	466	434	434
query63	320	301	287	287
query64	8924	2313	1770	1770
query65	3150	3096	3107	3096
query66	732	324	323	323
query67	15693	15004	14963	14963
query68	8606	555	568	555
query69	735	408	335	335
query70	1448	1090	1039	1039
query71	519	284	286	284
query72	9044	5235	5528	5235
query73	2223	328	322	322
query74	5886	5473	5437	5437
query75	5183	2621	2704	2621
query76	4963	1015	919	919
query77	801	302	306	302
query78	10289	8916	8850	8850
query79	8292	510	520	510
query80	1115	482	484	482
query81	562	221	222	221
query82	288	108	103	103
query83	340	172	166	166
query84	272	87	88	87
query85	928	317	267	267
query86	357	295	309	295
query87	3337	3087	3048	3048
query88	4498	2402	2384	2384
query89	517	391	388	388
query90	2138	183	187	183
query91	132	100	100	100
query92	62	50	48	48
query93	5937	530	514	514
query94	1547	211	209	209
query95	411	320	318	318
query96	615	267	263	263
query97	3191	3012	3012	3012
query98	206	197	195	195
query99	1155	825	850	825
Total cold run time: 290551 ms
Total hot run time: 172934 ms

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 6, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39965 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 51a1d3ba821bfcc9430f0fe2b678bd636a2bebf4, data reload: false

------ Round 1 ----------------------------------
q1	18475	4601	4308	4308
q2	2026	193	193	193
q3	10537	1218	1210	1210
q4	10249	794	770	770
q5	7587	2663	2608	2608
q6	221	139	137	137
q7	970	592	603	592
q8	9239	2101	2084	2084
q9	8997	6538	6521	6521
q10	8974	3711	3749	3711
q11	467	231	234	231
q12	481	225	225	225
q13	18694	2960	2997	2960
q14	276	229	227	227
q15	535	491	489	489
q16	522	374	373	373
q17	991	646	646	646
q18	8128	7563	7421	7421
q19	4881	1510	1469	1469
q20	663	314	331	314
q21	4984	3131	3863	3131
q22	397	345	347	345
Total cold run time: 118294 ms
Total hot run time: 39965 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4440	4293	4319	4293
q2	377	277	276	276
q3	3005	2907	2953	2907
q4	1986	1743	1739	1739
q5	5601	5577	5434	5434
q6	220	128	129	128
q7	2280	1879	1841	1841
q8	3271	3482	3455	3455
q9	9160	9355	9199	9199
q10	4053	3696	3801	3696
q11	584	515	497	497
q12	855	640	638	638
q13	15893	3156	3178	3156
q14	316	271	283	271
q15	520	498	477	477
q16	509	430	434	430
q17	1825	1546	1527	1527
q18	8126	7908	7690	7690
q19	3033	1655	1679	1655
q20	2094	1858	1878	1858
q21	5231	5004	4949	4949
q22	626	544	566	544
Total cold run time: 74005 ms
Total hot run time: 56660 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174301 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 51a1d3ba821bfcc9430f0fe2b678bd636a2bebf4, data reload: false

query1	920	371	363	363
query2	6448	2408	2340	2340
query3	6632	201	221	201
query4	28554	17448	17466	17448
query5	3593	497	482	482
query6	259	195	158	158
query7	4589	294	300	294
query8	335	300	306	300
query9	8463	2374	2359	2359
query10	570	303	295	295
query11	12888	10125	10202	10125
query12	117	101	81	81
query13	1652	387	379	379
query14	9700	7713	7964	7713
query15	280	194	192	192
query16	7795	326	330	326
query17	1783	573	526	526
query18	1622	281	294	281
query19	193	151	156	151
query20	91	84	84	84
query21	221	137	130	130
query22	4339	4026	3940	3940
query23	34339	33640	33710	33640
query24	11064	2937	2941	2937
query25	617	407	416	407
query26	1050	167	159	159
query27	2457	324	322	322
query28	7125	2128	2090	2090
query29	931	645	645	645
query30	255	159	160	159
query31	986	746	768	746
query32	96	56	61	56
query33	778	319	308	308
query34	925	500	495	495
query35	771	651	659	651
query36	1125	985	946	946
query37	151	85	80	80
query38	3005	2838	2851	2838
query39	866	787	811	787
query40	213	131	126	126
query41	55	54	55	54
query42	121	101	99	99
query43	585	535	559	535
query44	1216	728	750	728
query45	188	165	164	164
query46	1093	739	830	739
query47	1857	1785	1784	1784
query48	382	307	301	301
query49	840	415	407	407
query50	769	383	391	383
query51	6863	6822	6759	6759
query52	110	92	94	92
query53	356	280	299	280
query54	906	447	449	447
query55	75	70	71	70
query56	284	259	268	259
query57	1121	1071	1048	1048
query58	256	240	246	240
query59	3391	3173	3308	3173
query60	307	274	273	273
query61	94	92	97	92
query62	623	433	450	433
query63	322	286	285	285
query64	9422	2137	1607	1607
query65	3285	3094	3070	3070
query66	803	330	323	323
query67	15871	15053	15041	15041
query68	6861	533	518	518
query69	728	450	363	363
query70	1169	1114	1090	1090
query71	540	287	276	276
query72	8679	5966	5426	5426
query73	823	328	323	323
query74	6072	5494	5628	5494
query75	4954	2656	2657	2656
query76	4804	904	959	904
query77	786	295	296	295
query78	9508	9538	8899	8899
query79	8492	528	519	519
query80	1136	477	479	477
query81	584	219	228	219
query82	771	107	109	107
query83	341	163	167	163
query84	279	87	87	87
query85	1405	310	360	310
query86	434	335	321	321
query87	3271	3073	3120	3073
query88	4980	2460	2457	2457
query89	539	418	407	407
query90	2030	190	190	190
query91	132	102	104	102
query92	68	49	51	49
query93	6654	525	521	521
query94	1302	213	222	213
query95	413	326	321	321
query96	614	274	267	267
query97	3182	2982	3016	2982
query98	214	209	206	206
query99	1162	820	861	820
Total cold run time: 301920 ms
Total hot run time: 174301 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.63 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 51a1d3ba821bfcc9430f0fe2b678bd636a2bebf4, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.06	0.06
query4	1.65	0.09	0.10
query5	0.49	0.50	0.49
query6	1.14	0.73	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.50	0.48
query10	0.54	0.55	0.53
query11	0.16	0.11	0.12
query12	0.15	0.12	0.13
query13	0.60	0.59	0.58
query14	0.77	0.77	0.79
query15	0.87	0.82	0.82
query16	0.37	0.38	0.38
query17	1.02	0.98	1.05
query18	0.22	0.25	0.24
query19	1.84	1.83	1.69
query20	0.02	0.01	0.01
query21	15.39	0.72	0.64
query22	4.44	6.38	2.03
query23	18.26	1.36	1.20
query24	2.10	0.22	0.22
query25	0.15	0.08	0.08
query26	0.30	0.20	0.21
query27	0.45	0.24	0.23
query28	13.29	1.02	1.00
query29	12.61	3.31	3.25
query30	0.25	0.05	0.06
query31	2.86	0.40	0.39
query32	3.27	0.48	0.46
query33	2.84	2.93	2.95
query34	17.07	4.34	4.40
query35	4.50	4.45	4.38
query36	0.65	0.49	0.49
query37	0.18	0.15	0.15
query38	0.14	0.15	0.14
query39	0.05	0.03	0.04
query40	0.15	0.12	0.12
query41	0.10	0.04	0.05
query42	0.06	0.05	0.04
query43	0.05	0.04	0.03
Total cold run time: 109.95 s
Total hot run time: 30.63 s

@kaijchen
Copy link
Contributor Author

kaijchen commented Jul 7, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 7, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40547 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit efaac721a623e7197c93293c1f184f848f4f986f, data reload: false

------ Round 1 ----------------------------------
q1	18735	5526	4323	4323
q2	2038	193	187	187
q3	10561	1153	1135	1135
q4	10297	740	741	740
q5	7491	2617	2644	2617
q6	223	136	135	135
q7	944	603	602	602
q8	9227	2079	2052	2052
q9	8669	6509	6483	6483
q10	8900	3705	3716	3705
q11	470	237	233	233
q12	437	240	225	225
q13	17782	2996	2996	2996
q14	272	234	222	222
q15	522	487	494	487
q16	514	365	376	365
q17	972	689	667	667
q18	8107	7453	7517	7453
q19	5730	1513	1453	1453
q20	656	327	320	320
q21	5062	3815	3941	3815
q22	385	332	336	332
Total cold run time: 117994 ms
Total hot run time: 40547 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4418	4246	4248	4246
q2	396	273	274	273
q3	3112	2894	2937	2894
q4	1940	1780	1714	1714
q5	5517	5489	5408	5408
q6	226	141	133	133
q7	2270	1866	1862	1862
q8	3286	3418	3409	3409
q9	8679	8803	8701	8701
q10	4194	3743	3832	3743
q11	584	509	489	489
q12	816	668	648	648
q13	16308	3184	3173	3173
q14	302	276	258	258
q15	531	499	493	493
q16	501	437	437	437
q17	1794	1513	1494	1494
q18	8092	7819	7817	7817
q19	1862	1658	1526	1526
q20	2926	1888	1847	1847
q21	5193	4803	4705	4705
q22	657	557	555	555
Total cold run time: 73604 ms
Total hot run time: 55825 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173391 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit efaac721a623e7197c93293c1f184f848f4f986f, data reload: false

query1	914	390	373	373
query2	6437	2429	2433	2429
query3	6634	205	216	205
query4	26564	17588	17306	17306
query5	3701	504	501	501
query6	272	173	172	172
query7	4601	300	303	300
query8	323	310	279	279
query9	8562	2371	2354	2354
query10	567	309	283	283
query11	10585	9949	9948	9948
query12	118	85	85	85
query13	1636	379	383	379
query14	9337	7749	7691	7691
query15	240	178	180	178
query16	7738	308	309	308
query17	1675	539	528	528
query18	1907	280	275	275
query19	198	151	147	147
query20	93	80	83	80
query21	212	139	131	131
query22	4366	4115	3927	3927
query23	34291	33594	33647	33594
query24	10895	2843	2811	2811
query25	596	394	394	394
query26	767	161	157	157
query27	2261	325	337	325
query28	5966	2123	2116	2116
query29	892	646	626	626
query30	255	157	154	154
query31	1042	759	751	751
query32	92	56	54	54
query33	661	302	299	299
query34	886	507	519	507
query35	727	612	630	612
query36	1160	974	1010	974
query37	150	84	83	83
query38	2939	2890	2803	2803
query39	862	870	854	854
query40	206	131	132	131
query41	54	52	52	52
query42	108	100	103	100
query43	611	546	526	526
query44	1105	723	734	723
query45	188	159	155	155
query46	1074	716	700	700
query47	1839	1783	1762	1762
query48	371	298	304	298
query49	839	407	413	407
query50	774	384	406	384
query51	6841	6699	6809	6699
query52	106	98	98	98
query53	351	287	292	287
query54	890	461	446	446
query55	73	72	73	72
query56	282	263	271	263
query57	1171	1074	1044	1044
query58	255	238	242	238
query59	3390	3293	3293	3293
query60	326	283	274	274
query61	99	93	95	93
query62	601	444	457	444
query63	332	285	289	285
query64	9178	2198	1616	1616
query65	3174	3084	3077	3077
query66	748	333	347	333
query67	15293	15120	14821	14821
query68	4535	522	540	522
query69	489	326	338	326
query70	1183	1164	1050	1050
query71	363	283	274	274
query72	6873	5389	5646	5389
query73	752	322	324	322
query74	6014	5509	5507	5507
query75	3319	2646	2713	2646
query76	2198	949	924	924
query77	506	305	289	289
query78	9463	8746	8877	8746
query79	2916	507	518	507
query80	2135	469	473	469
query81	594	220	220	220
query82	924	105	105	105
query83	281	175	173	173
query84	275	88	91	88
query85	1982	336	306	306
query86	486	297	318	297
query87	3286	3099	3099	3099
query88	3749	2463	2448	2448
query89	478	391	386	386
query90	1750	193	189	189
query91	133	104	104	104
query92	58	48	46	46
query93	4691	518	518	518
query94	1143	276	209	209
query95	415	334	319	319
query96	601	277	268	268
query97	3218	3004	3040	3004
query98	215	201	196	196
query99	1326	831	858	831
Total cold run time: 277752 ms
Total hot run time: 173391 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.21 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit efaac721a623e7197c93293c1f184f848f4f986f, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.66	0.08	0.07
query5	0.48	0.48	0.48
query6	1.13	0.74	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.49	0.49
query10	0.55	0.54	0.54
query11	0.16	0.11	0.12
query12	0.15	0.12	0.12
query13	0.60	0.59	0.58
query14	0.76	0.77	0.77
query15	0.86	0.83	0.82
query16	0.37	0.35	0.37
query17	1.04	1.00	0.97
query18	0.24	0.23	0.25
query19	1.83	1.73	1.79
query20	0.02	0.00	0.01
query21	15.40	0.72	0.64
query22	4.28	6.90	1.77
query23	18.25	1.35	1.16
query24	2.12	0.22	0.22
query25	0.14	0.09	0.09
query26	0.28	0.21	0.20
query27	0.46	0.25	0.23
query28	13.29	1.01	0.99
query29	12.53	3.30	3.21
query30	0.25	0.06	0.05
query31	2.88	0.39	0.38
query32	3.27	0.48	0.47
query33	2.86	2.97	2.91
query34	17.13	4.32	4.43
query35	4.44	4.42	4.36
query36	0.64	0.49	0.47
query37	0.18	0.15	0.16
query38	0.16	0.15	0.14
query39	0.04	0.04	0.04
query40	0.15	0.11	0.12
query41	0.10	0.05	0.05
query42	0.06	0.06	0.05
query43	0.04	0.05	0.04
Total cold run time: 109.75 s
Total hot run time: 30.21 s

liaoxin01
liaoxin01 approved these changes Jul 8, 2024
View reviewed changes
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 8, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 8, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 8, 2024

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Jul 8, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 71a9207 into apache:master Jul 8, 2024
25 of 29 checks passed
kaijchen added a commit to kaijchen/doris that referenced this pull request Jul 9, 2024
@kaijchen
[fix](move-memtable) check segment num when closing each tablet (apac…
d6a0bda 
…he#36753)

Previously, there is chance that sender failed to send some data while
the receiver being unaware of.
This will cause lost data if some segments are skipped.

This PR fixes the problem by including checks in both sender and
receiver.
When sender failed to send rpc, LoadStreamStub will mark the involved
tablets failed.
Each sender will send segment num for each tablet in CLOSE_LOAD,
and receivers (LoadStream) will sum up and check total segment nums.
@kaijchen kaijchen mentioned this pull request Jul 9, 2024
Merged
kaijchen added a commit to kaijchen/doris that referenced this pull request Jul 10, 2024
@kaijchen
[fix](move-memtable) check segment num when closing each tablet (apac…
e9d9d8a 
…he#36753)

Previously, there is chance that sender failed to send some data while
the receiver being unaware of.
This will cause lost data if some segments are skipped.

This PR fixes the problem by including checks in both sender and
receiver.
When sender failed to send rpc, LoadStreamStub will mark the involved
tablets failed.
Each sender will send segment num for each tablet in CLOSE_LOAD,
and receivers (LoadStream) will sum up and check total segment nums.
@kaijchen kaijchen mentioned this pull request Jul 11, 2024
Merged
liaoxin01 pushed a commit that referenced this pull request Jul 11, 2024
@kaijchen
[fix](move-memtable) check segment num when closing each tablet (#36753…
fed632b 
…) (#37536)

cherry-pick #36753 and #37660
dataroaring pushed a commit that referenced this pull request Jul 12, 2024
@kaijchen
[test](be-ut) fix load stream test after segment num check (#37660)
d1bd06c 
## Proposed changes

fix load stream test after #36753
seawinde pushed a commit to seawinde/doris that referenced this pull request Jul 17, 2024
@kaijchen @seawinde
[test](be-ut) fix load stream test after segment num check (apache#37660
e89e168 
)

## Proposed changes

fix load stream test after apache#36753
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@kaijchen @dataroaring
[fix](move-memtable) check segment num when closing each tablet (#36753)
8648e47 
## Proposed changes

Previously, there is chance that sender failed to send some data while
the receiver being unaware of.
This will cause lost data if some segments are skipped.

This PR fixes the problem by including checks in both sender and
receiver.
When sender failed to send rpc, LoadStreamStub will mark the involved
tablets failed.
Each sender will send segment num for each tablet in CLOSE_LOAD,
and receivers (LoadStream) will sum up and check total segment nums.
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@kaijchen @dataroaring
[test](be-ut) fix load stream test after segment num check (#37660)
1133197 
## Proposed changes

fix load stream test after #36753
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring dataroaring approved these changes

@liaoxin01 liaoxin01 liaoxin01 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.5-merged dev/3.0.1-merged meta-change p0_w reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@kaijchen @doris-robot @liaoxin01 @dataroaring