[fix](scanner) Fix incorrect _max_thread_num in scanner context #40569

zhiqiang-hhhh · 2024-09-09T15:12:45Z

Minor refactor for scanner constructor, calculation of _max_thread_num is moved to init method
The expected value of _max_thread_num is changed. There is no need to submit too many scan task to scan scheduler, since thread num is limited.
Calculation of _max_bytes_in_queue is changed. _max_bytes_in_queue for each scan instance is limited to 100MB by default.

doris-robot · 2024-09-09T15:12:50Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

zhiqiang-hhhh · 2024-09-09T15:12:53Z

run buildall

zhiqiang-hhhh · 2024-09-09T15:21:11Z

run buildall

doris-robot · 2024-09-09T15:42:34Z

TeamCity be ut coverage result:
Function Coverage: 36.84% (9385/25478)
Line Coverage: 28.24% (77421/274173)
Region Coverage: 27.63% (39967/144646)
Branch Coverage: 24.27% (20338/83814)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d37aa09505d072baf025b40ad841583cb44cb50f_d37aa09505d072baf025b40ad841583cb44cb50f/report/index.html

doris-robot · 2024-09-09T15:52:03Z

TPC-H: Total hot run time: 37813 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d37aa09505d072baf025b40ad841583cb44cb50f, data reload: false

------ Round 1 ----------------------------------
q1	17600	4371	4265	4265
q2	2025	185	186	185
q3	11839	975	1094	975
q4	10521	805	718	718
q5	7739	2844	2832	2832
q6	228	141	137	137
q7	979	615	595	595
q8	9325	2077	2063	2063
q9	7221	6515	6574	6515
q10	7021	2202	2218	2202
q11	479	243	240	240
q12	394	220	222	220
q13	17989	3043	3094	3043
q14	284	229	241	229
q15	539	499	487	487
q16	517	448	417	417
q17	985	690	730	690
q18	7307	6718	6781	6718
q19	1385	1085	998	998
q20	705	327	329	327
q21	3954	2980	2951	2951
q22	1184	1057	1006	1006
Total cold run time: 110220 ms
Total hot run time: 37813 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4359	4298	4285	4285
q2	378	270	270	270
q3	2872	2645	2655	2645
q4	1945	1610	1652	1610
q5	5587	5730	5671	5671
q6	228	137	141	137
q7	2262	1833	1779	1779
q8	3292	3430	3459	3430
q9	8820	8799	8791	8791
q10	3556	3378	3362	3362
q11	612	504	520	504
q12	809	665	687	665
q13	15171	3376	3180	3180
q14	321	308	294	294
q15	555	493	490	490
q16	511	518	501	501
q17	1813	1567	1534	1534
q18	8169	7867	7740	7740
q19	1723	1563	1562	1562
q20	2194	1906	1922	1906
q21	5656	5486	5483	5483
q22	1137	1049	1052	1049
Total cold run time: 71970 ms
Total hot run time: 56888 ms

doris-robot · 2024-09-09T16:04:12Z

TPC-DS: Total hot run time: 192190 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d37aa09505d072baf025b40ad841583cb44cb50f, data reload: false

query1	1242	888	896	888
query2	6296	1910	1889	1889
query3	10741	4060	4059	4059
query4	60024	24467	23426	23426
query5	5181	490	517	490
query6	402	168	160	160
query7	5781	282	281	281
query8	305	217	240	217
query9	8639	2482	2456	2456
query10	446	268	262	262
query11	17977	14931	15151	14931
query12	163	101	107	101
query13	1530	386	387	386
query14	10732	7240	7363	7240
query15	224	171	196	171
query16	7405	507	468	468
query17	1095	562	569	562
query18	2031	286	291	286
query19	265	149	143	143
query20	123	116	111	111
query21	207	105	109	105
query22	4706	4614	4771	4614
query23	34459	33420	33534	33420
query24	5925	2860	2839	2839
query25	552	407	432	407
query26	652	161	148	148
query27	1730	274	270	270
query28	3786	2015	2004	2004
query29	678	394	393	393
query30	237	153	144	144
query31	912	736	769	736
query32	84	51	51	51
query33	412	277	277	277
query34	879	454	462	454
query35	813	714	718	714
query36	1067	949	928	928
query37	149	90	83	83
query38	3887	3751	3860	3751
query39	1454	1371	1542	1371
query40	196	111	111	111
query41	47	48	45	45
query42	118	92	94	92
query43	502	480	474	474
query44	1071	738	734	734
query45	192	159	162	159
query46	1079	727	761	727
query47	1903	1778	1791	1778
query48	360	287	288	287
query49	859	469	446	446
query50	823	406	419	406
query51	7054	6809	6898	6809
query52	95	84	84	84
query53	248	177	173	173
query54	552	439	440	439
query55	75	72	75	72
query56	279	252	240	240
query57	1180	1044	1068	1044
query58	220	223	216	216
query59	3054	2805	2726	2726
query60	288	264	260	260
query61	100	97	99	97
query62	765	628	648	628
query63	220	182	183	182
query64	2318	649	673	649
query65	3205	3102	3159	3102
query66	676	334	336	334
query67	15815	15408	15260	15260
query68	2927	561	567	561
query69	402	294	267	267
query70	1167	1052	1049	1049
query71	362	263	267	263
query72	6249	4092	3977	3977
query73	747	318	319	318
query74	9152	8759	8800	8759
query75	3431	2667	2686	2667
query76	1538	1018	964	964
query77	547	312	315	312
query78	10385	9059	9260	9059
query79	1067	525	511	511
query80	696	513	501	501
query81	492	234	228	228
query82	349	140	136	136
query83	171	147	150	147
query84	261	71	75	71
query85	730	289	270	270
query86	309	294	280	280
query87	4405	4271	4260	4260
query88	3039	2288	2282	2282
query89	398	291	287	287
query90	1776	185	185	185
query91	125	95	97	95
query92	58	50	51	50
query93	1100	535	539	535
query94	655	281	304	281
query95	343	245	247	245
query96	585	259	264	259
query97	3249	3085	3086	3085
query98	214	209	201	201
query99	1488	1278	1275	1275
Total cold run time: 304902 ms
Total hot run time: 192190 ms

doris-robot · 2024-09-09T16:09:48Z

ClickBench: Total hot run time: 31.31 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d37aa09505d072baf025b40ad841583cb44cb50f, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.66	0.09	0.09
query5	0.52	0.49	0.49
query6	1.13	0.74	0.73
query7	0.02	0.02	0.01
query8	0.05	0.05	0.05
query9	0.55	0.48	0.49
query10	0.54	0.55	0.55
query11	0.15	0.11	0.12
query12	0.15	0.13	0.12
query13	0.59	0.58	0.60
query14	1.39	1.42	1.41
query15	0.83	0.84	0.81
query16	0.38	0.36	0.38
query17	0.97	1.01	1.01
query18	0.21	0.20	0.20
query19	1.92	1.77	1.78
query20	0.01	0.02	0.01
query21	15.45	0.68	0.67
query22	4.94	8.05	1.88
query23	18.28	1.39	1.22
query24	2.14	0.22	0.22
query25	0.16	0.08	0.09
query26	0.25	0.18	0.17
query27	0.08	0.07	0.08
query28	13.27	1.03	0.99
query29	12.69	3.30	3.30
query30	0.24	0.05	0.05
query31	2.87	0.40	0.38
query32	3.26	0.47	0.48
query33	2.94	3.01	3.00
query34	17.21	4.43	4.41
query35	4.50	4.51	4.44
query36	0.66	0.47	0.48
query37	0.19	0.15	0.16
query38	0.15	0.15	0.14
query39	0.05	0.04	0.04
query40	0.16	0.13	0.13
query41	0.09	0.05	0.04
query42	0.06	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 111.1 s
Total hot run time: 31.31 s

zhiqiang-hhhh · 2024-09-10T01:43:17Z

run cloud_p0

zhiqiang-hhhh · 2024-09-10T08:29:27Z

run buildall

yiguolei · 2024-09-10T08:50:02Z

be/src/vec/exec/scan/scanner_context.cpp

+    // scan_queue_mem_limit on FE is 100MB by default, on backend we will make sure its actual value
+    // is larger than 10MB.
+    _max_bytes_in_queue =
+            std::max(_state->scan_queue_mem_limit(), (int64_t)1024 * 1024 * 1024 * 10);


是10MB，不是10gb

doris-robot · 2024-09-10T09:00:13Z

TeamCity be ut coverage result:
Function Coverage: 36.84% (9396/25508)
Line Coverage: 28.24% (77530/274524)
Region Coverage: 27.64% (40021/144798)
Branch Coverage: 24.26% (20354/83892)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e91aa7d91e847f6557823c3e1135a90ad1c575c2_e91aa7d91e847f6557823c3e1135a90ad1c575c2/report/index.html

yiguolei · 2024-09-10T10:06:01Z

be/src/pipeline/exec/scan_operator.cpp

@@ -117,7 +117,7 @@ Status ScanLocalState<Derived>::open(RuntimeState* state) {
    RETURN_IF_ERROR(status);
    if (_scanner_ctx) {
        DCHECK(!_eos && _num_scanners->value() > 0);
-        RETURN_IF_ERROR(_scanner_ctx->init());
+        RETURN_IF_ERROR(_scanner_ctx->init(p.ignore_data_distribution()));


可能把，这个参数，作为scanner context的构造函数的参数，传递给scanner context，这样你往2.1 pick的时候会比较方便。否则2.1 在多种pipeline 模型下，代码不好pick

…r-scanner

zhiqiang-hhhh · 2024-09-10T16:25:28Z

run buildall

zhiqiang-hhhh · 2024-09-10T16:46:36Z

run buildall

doris-robot · 2024-09-10T17:10:42Z

TeamCity be ut coverage result:
Function Coverage: 36.83% (9396/25510)
Line Coverage: 28.23% (77516/274539)
Region Coverage: 27.64% (40022/144806)
Branch Coverage: 24.26% (20353/83898)
Coverage Report: http://coverage.selectdb-in.cc/coverage/aa5f88250f276aa60acfb974bc9540d7a2baa365_aa5f88250f276aa60acfb974bc9540d7a2baa365/report/index.html

doris-robot · 2024-09-10T17:24:16Z

TPC-H: Total hot run time: 39300 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit aa5f88250f276aa60acfb974bc9540d7a2baa365, data reload: false

------ Round 1 ----------------------------------
q1	17590	4152	4117	4117
q2	2018	179	177	177
q3	10726	1199	1362	1199
q4	10163	1003	1057	1003
q5	7726	3155	3128	3128
q6	223	141	140	140
q7	1026	633	612	612
q8	9459	2033	1967	1967
q9	6718	6199	6231	6199
q10	6989	2486	2456	2456
q11	430	253	253	253
q12	411	231	235	231
q13	17755	3003	3031	3003
q14	293	242	256	242
q15	554	495	479	479
q16	516	425	436	425
q17	991	941	934	934
q18	7402	6695	6674	6674
q19	1385	1232	1215	1215
q20	600	339	341	339
q21	3851	3558	3526	3526
q22	1056	1040	981	981
Total cold run time: 107882 ms
Total hot run time: 39300 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4228	4168	4130	4130
q2	341	236	228	228
q3	2890	2876	2858	2858
q4	1988	1940	1927	1927
q5	5636	5625	5696	5625
q6	227	137	136	136
q7	2170	1793	1786	1786
q8	3310	3358	3363	3358
q9	8808	8841	8760	8760
q10	3460	3574	3583	3574
q11	583	489	481	481
q12	837	645	600	600
q13	10559	3144	3170	3144
q14	307	280	281	280
q15	537	498	498	498
q16	526	477	460	460
q17	1796	1751	1748	1748
q18	8404	7942	7799	7799
q19	1759	1729	1724	1724
q20	2101	1852	1860	1852
q21	6000	5690	5785	5690
q22	1121	992	997	992
Total cold run time: 67588 ms
Total hot run time: 57650 ms

doris-robot · 2024-09-10T17:36:25Z

TPC-DS: Total hot run time: 197893 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit aa5f88250f276aa60acfb974bc9540d7a2baa365, data reload: false

query1	1233	893	884	884
query2	6192	1801	1827	1801
query3	10617	3893	3775	3775
query4	55464	25209	23770	23770
query5	5039	528	518	518
query6	291	167	161	161
query7	5576	306	301	301
query8	282	221	222	221
query9	5956	2428	2433	2428
query10	394	274	266	266
query11	16086	15381	15379	15379
query12	149	94	96	94
query13	1401	381	363	363
query14	10215	7266	7218	7218
query15	195	173	183	173
query16	7080	502	462	462
query17	1153	638	592	592
query18	1820	308	316	308
query19	204	152	156	152
query20	125	112	112	112
query21	223	109	106	106
query22	4655	4437	4381	4381
query23	34386	33632	33488	33488
query24	6129	3072	3056	3056
query25	521	424	407	407
query26	614	152	156	152
query27	1573	286	287	286
query28	2813	2014	2000	2000
query29	726	427	432	427
query30	223	156	149	149
query31	977	761	786	761
query32	80	57	55	55
query33	444	281	302	281
query34	904	489	466	466
query35	853	719	718	718
query36	1046	878	912	878
query37	140	80	80	80
query38	3922	3908	3867	3867
query39	1454	1407	1407	1407
query40	204	115	115	115
query41	48	49	46	46
query42	120	96	99	96
query43	486	436	440	436
query44	1148	771	767	767
query45	196	168	170	168
query46	1092	837	819	819
query47	1886	1768	1769	1768
query48	358	301	321	301
query49	747	452	509	452
query50	911	440	436	436
query51	7054	6818	6962	6818
query52	97	87	86	86
query53	254	180	177	177
query54	574	472	475	472
query55	77	75	78	75
query56	294	267	292	267
query57	1186	1058	1070	1058
query58	235	248	233	233
query59	2804	2631	2432	2432
query60	292	276	281	276
query61	104	106	97	97
query62	790	683	649	649
query63	216	184	182	182
query64	1582	662	684	662
query65	3178	3118	3143	3118
query66	671	364	337	337
query67	15774	15284	15141	15141
query68	2073	561	578	561
query69	398	272	277	272
query70	1157	1110	1122	1110
query71	359	278	281	278
query72	6065	3986	3975	3975
query73	773	326	329	326
query74	9173	8914	8875	8875
query75	3299	2700	2735	2700
query76	1358	1289	1295	1289
query77	528	319	326	319
query78	9805	9637	9400	9400
query79	1536	869	854	854
query80	1057	851	822	822
query81	472	265	255	255
query82	1136	265	263	263
query83	194	187	189	187
query84	273	106	102	102
query85	778	405	392	392
query86	310	335	324	324
query87	4334	4242	4321	4242
query88	4236	4167	4156	4156
query89	381	363	360	360
query90	1899	317	309	309
query91	122	122	127	122
query92	83	76	79	76
query93	1261	1035	1028	1028
query94	796	365	362	362
query95	498	422	424	422
query96	475	477	475	475
query97	3121	3140	3105	3105
query98	230	225	230	225
query99	1680	1303	1280	1280
Total cold run time: 293393 ms
Total hot run time: 197893 ms

doris-robot · 2024-09-10T17:41:55Z

ClickBench: Total hot run time: 31.46 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit aa5f88250f276aa60acfb974bc9540d7a2baa365, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.06
query4	1.71	0.07	0.07
query5	0.51	0.50	0.49
query6	1.14	0.73	0.73
query7	0.02	0.01	0.01
query8	0.06	0.05	0.04
query9	0.55	0.51	0.51
query10	0.56	0.57	0.55
query11	0.15	0.11	0.11
query12	0.15	0.12	0.11
query13	0.62	0.60	0.59
query14	1.44	1.46	1.48
query15	0.91	0.86	0.87
query16	0.35	0.36	0.36
query17	1.02	1.00	1.00
query18	0.21	0.21	0.21
query19	1.91	1.82	1.82
query20	0.01	0.01	0.01
query21	15.38	0.68	0.66
query22	4.11	6.82	1.89
query23	17.97	1.32	1.30
query24	2.27	0.22	0.22
query25	0.18	0.08	0.08
query26	0.29	0.18	0.18
query27	0.07	0.08	0.08
query28	13.18	1.01	0.97
query29	12.58	3.44	3.38
query30	0.25	0.06	0.05
query31	2.87	0.40	0.40
query32	3.25	0.49	0.49
query33	2.99	3.04	3.01
query34	15.44	4.31	4.30
query35	4.30	4.31	4.29
query36	0.68	0.50	0.49
query37	0.18	0.16	0.16
query38	0.16	0.15	0.16
query39	0.05	0.04	0.04
query40	0.16	0.14	0.14
query41	0.10	0.04	0.04
query42	0.06	0.05	0.04
query43	0.04	0.04	0.05
Total cold run time: 108.23 s
Total hot run time: 31.46 s

github-actions · 2024-09-11T02:44:36Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-09-11T02:44:38Z

PR approved by anyone and no changes requested.

yiguolei · 2024-09-11T02:44:41Z

run performance

doris-robot · 2024-09-11T03:03:04Z

TPC-H: Total hot run time: 39569 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit aa5f88250f276aa60acfb974bc9540d7a2baa365, data reload: false

------ Round 1 ----------------------------------
q1	17913	4927	4245	4245
q2	2390	185	182	182
q3	10463	1321	1317	1317
q4	10159	980	1030	980
q5	7713	3128	3096	3096
q6	222	143	141	141
q7	1018	631	628	628
q8	9442	1980	1959	1959
q9	6744	6229	6238	6229
q10	7006	2464	2491	2464
q11	433	250	250	250
q12	398	233	227	227
q13	17767	3008	3010	3008
q14	293	264	258	258
q15	538	493	492	492
q16	522	437	419	419
q17	988	955	924	924
q18	7294	6678	6721	6678
q19	1381	1240	1215	1215
q20	617	326	333	326
q21	3885	3526	3517	3517
q22	1087	1014	1015	1014
Total cold run time: 108273 ms
Total hot run time: 39569 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4155	4235	4175	4175
q2	347	235	229	229
q3	2894	2862	2852	2852
q4	1956	1935	1949	1935
q5	5418	5388	5394	5388
q6	217	132	131	131
q7	2039	1683	1676	1676
q8	3223	3252	3257	3252
q9	8298	8311	8371	8311
q10	3356	3373	3369	3369
q11	571	462	480	462
q12	765	568	572	568
q13	6002	3048	2980	2980
q14	297	278	268	268
q15	532	493	496	493
q16	496	444	438	438
q17	1741	1706	1676	1676
q18	7897	7490	7571	7490
q19	1686	1674	1663	1663
q20	2006	1803	1780	1780
q21	5684	5295	5444	5295
q22	1068	1017	1054	1017
Total cold run time: 60648 ms
Total hot run time: 55448 ms

doris-robot · 2024-09-11T03:14:46Z

TPC-DS: Total hot run time: 193174 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit aa5f88250f276aa60acfb974bc9540d7a2baa365, data reload: false

query1	905	370	383	370
query2	6518	1770	1774	1770
query3	6651	220	224	220
query4	25741	23776	23612	23612
query5	5042	525	523	523
query6	261	178	160	160
query7	4585	299	300	299
query8	283	216	217	216
query9	8467	2439	2440	2439
query10	483	289	279	279
query11	16152	15706	15341	15341
query12	152	97	102	97
query13	1669	390	374	374
query14	9970	7062	6925	6925
query15	207	167	165	165
query16	7270	475	478	475
query17	1350	555	555	555
query18	1776	295	280	280
query19	187	139	141	139
query20	116	110	108	108
query21	206	103	100	100
query22	4523	4446	4176	4176
query23	34262	33572	33392	33392
query24	10191	3114	3007	3007
query25	664	378	384	378
query26	1282	153	159	153
query27	2864	278	285	278
query28	6913	2006	1984	1984
query29	944	411	400	400
query30	300	187	149	149
query31	1001	770	780	770
query32	97	53	57	53
query33	716	302	308	302
query34	908	462	501	462
query35	869	732	730	730
query36	1041	902	911	902
query37	164	84	78	78
query38	4008	3900	3893	3893
query39	1602	1431	1388	1388
query40	284	111	111	111
query41	47	45	42	42
query42	118	94	98	94
query43	490	444	445	444
query44	1194	767	740	740
query45	196	165	169	165
query46	1100	817	794	794
query47	1892	1766	1754	1754
query48	362	289	297	289
query49	1127	472	445	445
query50	915	434	437	434
query51	6937	6895	6915	6895
query52	100	88	87	87
query53	254	185	185	185
query54	786	482	478	478
query55	119	75	74	74
query56	283	255	261	255
query57	1196	1060	1059	1059
query58	239	229	256	229
query59	2684	2509	2438	2438
query60	292	267	271	267
query61	114	105	103	103
query62	922	665	662	662
query63	227	184	181	181
query64	5280	697	661	661
query65	3223	3128	3132	3128
query66	1377	335	355	335
query67	16033	15715	15282	15282
query68	3186	579	588	579
query69	403	279	280	279
query70	1083	1118	1102	1102
query71	347	276	268	268
query72	5918	4004	3977	3977
query73	749	339	330	330
query74	9343	8924	8981	8924
query75	3343	2701	2646	2646
query76	1883	1314	1293	1293
query77	430	319	369	319
query78	9876	9339	9502	9339
query79	913	876	868	868
query80	845	831	830	830
query81	445	263	272	263
query82	255	262	268	262
query83	198	194	198	194
query84	229	109	106	106
query85	701	415	415	415
query86	309	310	308	308
query87	4371	4322	4374	4322
query88	4249	4160	4177	4160
query89	376	364	371	364
query90	1579	322	320	320
query91	124	122	125	122
query92	78	76	85	76
query93	1036	1051	1041	1041
query94	755	387	376	376
query95	475	429	436	429
query96	475	477	477	477
query97	3137	3095	3099	3095
query98	225	234	226	226
query99	1566	1301	1291	1291
Total cold run time: 277033 ms
Total hot run time: 193174 ms

doris-robot · 2024-09-11T03:20:13Z

ClickBench: Total hot run time: 30.65 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit aa5f88250f276aa60acfb974bc9540d7a2baa365, data reload: false

query1	0.05	0.05	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.07	0.07
query5	0.50	0.51	0.50
query6	1.14	0.73	0.71
query7	0.02	0.01	0.01
query8	0.05	0.05	0.04
query9	0.56	0.52	0.52
query10	0.57	0.60	0.56
query11	0.16	0.12	0.12
query12	0.15	0.13	0.12
query13	0.61	0.60	0.60
query14	1.44	1.44	1.46
query15	0.91	0.88	0.87
query16	0.38	0.37	0.36
query17	1.05	0.99	1.02
query18	0.22	0.21	0.21
query19	1.89	1.72	1.76
query20	0.01	0.01	0.02
query21	15.41	0.68	0.66
query22	4.19	7.64	1.19
query23	17.94	1.37	1.34
query24	2.28	0.22	0.22
query25	0.18	0.09	0.08
query26	0.29	0.18	0.17
query27	0.08	0.08	0.08
query28	13.16	1.00	0.97
query29	12.55	3.38	3.37
query30	0.25	0.06	0.06
query31	2.87	0.42	0.40
query32	3.24	0.48	0.50
query33	3.00	3.06	2.99
query34	15.46	4.32	4.28
query35	4.33	4.29	4.29
query36	0.68	0.47	0.50
query37	0.19	0.18	0.16
query38	0.17	0.15	0.15
query39	0.04	0.03	0.04
query40	0.16	0.14	0.14
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 108.35 s
Total hot run time: 30.65 s

wangbo

LGTM

…he#40569) 1. Minor refactor for scanner constructor, calculation of _max_thread_num is moved to init method 2. The expected value of _max_thread_num is changed. There is no need to submit too many scan task to scan scheduler, since thread num is limited. 3. Calculation of _max_bytes_in_queue is changed. _max_bytes_in_queue for each scan instance is limited to 100MB by default.

…xt (apache#40569)" This reverts commit a912f77.

…xt" (#40804) Reverts #40569 We need more test to avoid performance issue

1. Minor refactor for scanner constructor, calculation of _max_thread_num is moved to init method 2. The expected value of _max_thread_num is changed. There is no need to submit too many scan task to scan scheduler, since thread num is limited. 3. Calculation of _max_bytes_in_queue is changed. _max_bytes_in_queue for each scan instance is limited to 100MB by default.

…xt" (#40804) Reverts #40569 We need more test to avoid performance issue

zhiqiang-hhhh added 2 commits September 9, 2024 23:08

X

6baa417

REF

23a2efd

X

d37aa09

X

e91aa7d

zhiqiang-hhhh changed the title ~~[refactor](scanner) Minor refactor to scanner constructor~~ [fix](scanner) Fix incorrect _max_thread_num in scanner context Sep 10, 2024

yiguolei reviewed Sep 10, 2024

View reviewed changes

FIX

76ae129

yiguolei reviewed Sep 10, 2024

View reviewed changes

yiguolei added dev/2.1.x dev/3.0.x labels Sep 10, 2024

zhiqiang-hhhh added 2 commits September 11, 2024 00:24

X

d6b88d8

Merge branch 'master' of https://github.com/apache/doris into refacto…

c1d693a

…r-scanner

zhiqiang-hhhh added 2 commits September 11, 2024 00:45

X

0e051e5

X

aa5f882

yiguolei approved these changes Sep 11, 2024

View reviewed changes

github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Sep 11, 2024

yiguolei approved these changes Sep 11, 2024

View reviewed changes

wangbo approved these changes Sep 12, 2024

View reviewed changes

yiguolei merged commit a912f77 into apache:master Sep 12, 2024
24 of 30 checks passed

zhiqiang-hhhh deleted the refactor-scanner branch September 12, 2024 11:27

zhiqiang-hhhh mentioned this pull request Sep 12, 2024

[fix](scanner) Fix incorrect _max_thread_num in scanner context #40569 #40760

Closed

zhiqiang-hhhh added a commit to zhiqiang-hhhh/doris that referenced this pull request Sep 13, 2024

Revert "[fix](scanner) Fix incorrect _max_thread_num in scanner conte…

397ee86

…xt (apache#40569)" This reverts commit a912f77.

zhiqiang-hhhh mentioned this pull request Sep 13, 2024

Revert "[fix](scanner) Fix incorrect _max_thread_num in scanner context" #40804

Merged

yiguolei pushed a commit to zhiqiang-hhhh/doris that referenced this pull request Sep 15, 2024

Revert "[fix](scanner) Fix incorrect _max_thread_num in scanner conte…

a551f8d

…xt (apache#40569)" This reverts commit a912f77.

yiguolei pushed a commit that referenced this pull request Sep 16, 2024

Revert "[fix](scanner) Fix incorrect _max_thread_num in scanner conte…

b92b63e

…xt" (#40804) Reverts #40569 We need more test to avoid performance issue

zhiqiang-hhhh mentioned this pull request Sep 25, 2024

[fix](scanner) Fix incorrect _max_thread_num in scanner context when many queries are running. #41273

Merged

dataroaring pushed a commit that referenced this pull request Oct 9, 2024

Revert "[fix](scanner) Fix incorrect _max_thread_num in scanner conte…

18588c5

…xt" (#40804) Reverts #40569 We need more test to avoid performance issue

dataroaring added dev/3.0.3-merged and removed dev/3.0.x labels Oct 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix](scanner) Fix incorrect _max_thread_num in scanner context #40569

[fix](scanner) Fix incorrect _max_thread_num in scanner context #40569

zhiqiang-hhhh commented Sep 9, 2024 •

edited

Loading

doris-robot commented Sep 9, 2024

zhiqiang-hhhh commented Sep 9, 2024

zhiqiang-hhhh commented Sep 9, 2024

doris-robot commented Sep 9, 2024

doris-robot commented Sep 9, 2024

doris-robot commented Sep 9, 2024

doris-robot commented Sep 9, 2024

zhiqiang-hhhh commented Sep 10, 2024

zhiqiang-hhhh commented Sep 10, 2024

yiguolei Sep 10, 2024

zhiqiang-hhhh Sep 10, 2024

doris-robot commented Sep 10, 2024

yiguolei Sep 10, 2024

zhiqiang-hhhh commented Sep 10, 2024

zhiqiang-hhhh commented Sep 10, 2024

doris-robot commented Sep 10, 2024

doris-robot commented Sep 10, 2024

doris-robot commented Sep 10, 2024

doris-robot commented Sep 10, 2024

github-actions bot commented Sep 11, 2024

github-actions bot commented Sep 11, 2024

yiguolei commented Sep 11, 2024

doris-robot commented Sep 11, 2024

doris-robot commented Sep 11, 2024

doris-robot commented Sep 11, 2024

wangbo left a comment

[fix](scanner) Fix incorrect _max_thread_num in scanner context #40569

[fix](scanner) Fix incorrect _max_thread_num in scanner context #40569

Conversation

zhiqiang-hhhh commented Sep 9, 2024 • edited Loading

doris-robot commented Sep 9, 2024

zhiqiang-hhhh commented Sep 9, 2024

zhiqiang-hhhh commented Sep 9, 2024

doris-robot commented Sep 9, 2024

doris-robot commented Sep 9, 2024

doris-robot commented Sep 9, 2024

doris-robot commented Sep 9, 2024

zhiqiang-hhhh commented Sep 10, 2024

zhiqiang-hhhh commented Sep 10, 2024

yiguolei Sep 10, 2024

Choose a reason for hiding this comment

zhiqiang-hhhh Sep 10, 2024

Choose a reason for hiding this comment

doris-robot commented Sep 10, 2024

yiguolei Sep 10, 2024

Choose a reason for hiding this comment

zhiqiang-hhhh commented Sep 10, 2024

zhiqiang-hhhh commented Sep 10, 2024

doris-robot commented Sep 10, 2024

doris-robot commented Sep 10, 2024

doris-robot commented Sep 10, 2024

doris-robot commented Sep 10, 2024

github-actions bot commented Sep 11, 2024

github-actions bot commented Sep 11, 2024

yiguolei commented Sep 11, 2024

doris-robot commented Sep 11, 2024

doris-robot commented Sep 11, 2024

doris-robot commented Sep 11, 2024

wangbo left a comment

Choose a reason for hiding this comment

zhiqiang-hhhh commented Sep 9, 2024 •

edited

Loading