Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FIX](agg) fix vertical_compaction_reader for agg table with array/map type #33130

Merged

Conversation

amorynan
Copy link
Contributor

@amorynan amorynan commented Apr 1, 2024

Proposed changes

fix vertical_compaction_reader for agg table with array/map type
same with this pr: #24873
Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@amorynan
Copy link
Contributor Author

amorynan commented Apr 1, 2024

run buildall

Copy link
Contributor

github-actions bot commented Apr 1, 2024

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
Copy link
Contributor

github-actions bot commented Apr 1, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39210 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e9a929c7409db66a0c069864f089553da7a4b9fc, data reload: false

------ Round 1 ----------------------------------
q1	17597	4273	4309	4273
q2	2002	190	176	176
q3	10491	1218	1426	1218
q4	10203	874	1035	874
q5	7512	3033	2972	2972
q6	213	132	135	132
q7	1123	669	621	621
q8	9436	2098	2080	2080
q9	6728	6293	6171	6171
q10	8518	3531	3537	3531
q11	417	250	239	239
q12	385	220	216	216
q13	17785	2932	2917	2917
q14	280	245	244	244
q15	530	483	483	483
q16	504	390	377	377
q17	971	937	910	910
q18	7234	6493	6486	6486
q19	2754	1567	1550	1550
q20	599	315	312	312
q21	3588	3166	3128	3128
q22	365	308	300	300
Total cold run time: 109235 ms
Total hot run time: 39210 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4085	4046	4043	4043
q2	338	225	218	218
q3	3001	2963	2978	2963
q4	1887	1854	1851	1851
q5	5272	5249	5236	5236
q6	212	127	125	125
q7	2267	1816	1832	1816
q8	3240	3301	3305	3301
q9	8563	8540	8561	8540
q10	3798	3846	3851	3846
q11	556	463	456	456
q12	712	568	525	525
q13	16790	2888	2881	2881
q14	303	267	256	256
q15	517	475	465	465
q16	459	429	416	416
q17	1728	1708	1691	1691
q18	7777	7787	7700	7700
q19	1699	1783	1729	1729
q20	2086	1865	1820	1820
q21	5325	4967	5150	4967
q22	499	417	443	417
Total cold run time: 71114 ms
Total hot run time: 55262 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.55% (8842/24870)
Line Coverage: 27.28% (72502/265775)
Region Coverage: 26.48% (37531/141731)
Branch Coverage: 23.28% (19132/82176)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e9a929c7409db66a0c069864f089553da7a4b9fc_e9a929c7409db66a0c069864f089553da7a4b9fc/report/index.html

Copy link
Contributor

github-actions bot commented Apr 2, 2024

clang-tidy review says "All clean, LGTM! 👍"

@amorynan
Copy link
Contributor Author

amorynan commented Apr 2, 2024

run buildall

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/vec/columns/column_array.h Show resolved Hide resolved
be/src/vec/columns/column_map.h Show resolved Hide resolved
be/src/vec/columns/column_string.h Show resolved Hide resolved
xiaokang
xiaokang previously approved these changes Apr 2, 2024
Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

be/src/vec/columns/column_array.h Outdated Show resolved Hide resolved
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 2, 2024
Copy link
Contributor

github-actions bot commented Apr 2, 2024

PR approved by at least one committer and no changes requested.

Copy link
Contributor

github-actions bot commented Apr 2, 2024

PR approved by anyone and no changes requested.

@xiaokang xiaokang added the p0_c label Apr 2, 2024
@amorynan
Copy link
Contributor Author

amorynan commented Apr 2, 2024

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Apr 2, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 38914 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 02f047fb10c7fea6ec7dbfc4484069c91db0cea5, data reload: false

------ Round 1 ----------------------------------
q1	19193	4268	4084	4084
q2	2006	185	190	185
q3	10484	1205	1387	1205
q4	10205	830	980	830
q5	7540	3040	3004	3004
q6	220	137	131	131
q7	1128	623	610	610
q8	9428	2068	2051	2051
q9	6749	6228	6208	6208
q10	8428	3536	3504	3504
q11	427	244	247	244
q12	376	219	209	209
q13	17773	2908	2876	2876
q14	278	244	247	244
q15	520	488	488	488
q16	479	401	388	388
q17	975	937	934	934
q18	7267	6546	6460	6460
q19	1601	1561	1542	1542
q20	593	306	302	302
q21	3557	3113	3112	3112
q22	373	303	323	303
Total cold run time: 109600 ms
Total hot run time: 38914 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4059	4085	4035	4035
q2	327	217	216	216
q3	3018	2951	2956	2951
q4	1890	1863	1831	1831
q5	5284	5238	5243	5238
q6	209	124	122	122
q7	2251	1794	1836	1794
q8	3223	3299	3292	3292
q9	8527	8516	8499	8499
q10	3749	3799	3826	3799
q11	534	452	443	443
q12	700	555	536	536
q13	12731	2915	2892	2892
q14	287	263	270	263
q15	514	472	473	472
q16	458	404	409	404
q17	1743	1684	1689	1684
q18	7649	7333	7455	7333
q19	1651	1661	1657	1657
q20	1953	1705	1731	1705
q21	5048	4792	4820	4792
q22	500	419	412	412
Total cold run time: 66305 ms
Total hot run time: 54370 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 182129 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 02f047fb10c7fea6ec7dbfc4484069c91db0cea5, data reload: false

query1	892	1118	1128	1118
query2	6474	1799	1849	1799
query3	6666	214	219	214
query4	25041	21390	21477	21390
query5	4162	431	418	418
query6	270	188	174	174
query7	4588	318	299	299
query8	232	180	184	180
query9	8454	2278	2283	2278
query10	561	262	276	262
query11	14918	14460	14491	14460
query12	150	97	95	95
query13	1647	394	388	388
query14	8534	6832	6756	6756
query15	216	180	183	180
query16	7160	284	273	273
query17	991	620	604	604
query18	1908	296	286	286
query19	212	170	159	159
query20	103	96	96	96
query21	218	135	125	125
query22	4964	4796	4805	4796
query23	33553	33031	33232	33031
query24	12581	3166	3164	3164
query25	682	385	395	385
query26	1905	175	158	158
query27	2990	321	329	321
query28	6733	1866	1844	1844
query29	1350	578	580	578
query30	295	164	164	164
query31	987	718	744	718
query32	101	61	62	61
query33	724	249	251	249
query34	1042	486	494	486
query35	831	703	704	703
query36	982	865	816	816
query37	284	78	78	78
query38	3482	3416	3399	3399
query39	1570	1540	1542	1540
query40	290	137	138	137
query41	51	46	45	45
query42	118	107	106	106
query43	440	404	396	396
query44	1087	741	719	719
query45	265	258	263	258
query46	1087	812	773	773
query47	1916	1808	1812	1808
query48	401	318	308	308
query49	1169	376	369	369
query50	793	389	389	389
query51	6856	6741	6675	6675
query52	109	105	98	98
query53	356	296	290	290
query54	330	250	250	250
query55	91	86	78	78
query56	242	229	232	229
query57	1219	1116	1127	1116
query58	261	229	226	226
query59	2408	2332	2171	2171
query60	264	245	253	245
query61	119	113	113	113
query62	695	445	464	445
query63	316	286	286	286
query64	6488	3155	3234	3155
query65	3083	3000	3003	3000
query66	1460	345	336	336
query67	15386	14921	14709	14709
query68	5212	584	580	580
query69	542	327	328	327
query70	1196	1102	1109	1102
query71	427	289	288	288
query72	6520	2765	2478	2478
query73	716	329	327	327
query74	6718	6337	6388	6337
query75	2992	2295	2318	2295
query76	3416	1088	1215	1088
query77	570	252	251	251
query78	10779	10111	10092	10092
query79	8734	543	539	539
query80	1415	422	432	422
query81	516	243	244	243
query82	727	100	104	100
query83	201	163	165	163
query84	270	93	92	92
query85	1470	290	286	286
query86	411	282	278	278
query87	3663	3590	3468	3468
query88	4229	2273	2281	2273
query89	545	370	375	370
query90	2018	182	172	172
query91	136	108	110	108
query92	56	56	56	56
query93	6967	550	538	538
query94	1130	203	185	185
query95	1096	1093	1100	1093
query96	609	274	270	270
query97	2661	2514	2484	2484
query98	265	226	214	214
query99	1352	835	871	835
Total cold run time: 293312 ms
Total hot run time: 182129 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.94 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 02f047fb10c7fea6ec7dbfc4484069c91db0cea5, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.05	0.05
query3	0.24	0.05	0.05
query4	1.68	0.06	0.06
query5	0.47	0.48	0.49
query6	1.18	0.67	0.66
query7	0.02	0.01	0.01
query8	0.06	0.04	0.04
query9	0.56	0.52	0.51
query10	0.57	0.56	0.56
query11	0.15	0.11	0.11
query12	0.14	0.11	0.11
query13	0.60	0.60	0.59
query14	0.78	0.78	0.80
query15	0.86	0.84	0.84
query16	0.35	0.36	0.35
query17	1.01	0.99	1.00
query18	0.26	0.25	0.26
query19	1.84	1.73	1.70
query20	0.01	0.01	0.01
query21	15.46	0.76	0.71
query22	2.89	5.34	1.51
query23	17.74	1.37	1.06
query24	1.51	0.20	0.32
query25	0.15	0.10	0.08
query26	0.29	0.17	0.19
query27	0.09	0.08	0.08
query28	13.60	0.95	0.94
query29	12.59	3.44	3.41
query30	0.25	0.06	0.05
query31	2.85	0.40	0.39
query32	3.26	0.48	0.47
query33	2.85	2.84	2.93
query34	15.48	4.33	4.35
query35	4.38	4.39	4.40
query36	0.68	0.48	0.46
query37	0.19	0.18	0.18
query38	0.17	0.17	0.16
query39	0.05	0.04	0.04
query40	0.18	0.15	0.18
query41	0.10	0.05	0.06
query42	0.06	0.05	0.05
query43	0.04	0.05	0.04
Total cold run time: 105.75 s
Total hot run time: 29.94 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.57% (8851/24883)
Line Coverage: 27.29% (72537/265818)
Region Coverage: 26.49% (37548/141738)
Branch Coverage: 23.29% (19134/82164)
Coverage Report: http://coverage.selectdb-in.cc/coverage/02f047fb10c7fea6ec7dbfc4484069c91db0cea5_02f047fb10c7fea6ec7dbfc4484069c91db0cea5/report/index.html

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 02f047fb10c7fea6ec7dbfc4484069c91db0cea5 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       16.2 seconds inserted 10000000 Rows, about 617K ops/s

@amorynan
Copy link
Contributor Author

amorynan commented Apr 2, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39184 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9f34eb21390bd466e1ad8985b1f3471fdb4b0a98, data reload: false

------ Round 1 ----------------------------------
q1	17690	4292	4193	4193
q2	2386	197	185	185
q3	12568	1251	1469	1251
q4	10340	900	1026	900
q5	7671	3074	3008	3008
q6	221	135	135	135
q7	1155	637	628	628
q8	9526	2106	2131	2106
q9	6757	6246	6182	6182
q10	8409	3519	3515	3515
q11	416	268	237	237
q12	384	215	214	214
q13	17803	2894	2927	2894
q14	274	242	242	242
q15	530	490	475	475
q16	489	397	374	374
q17	987	940	943	940
q18	7276	6457	6492	6457
q19	1605	1554	1542	1542
q20	603	310	315	310
q21	3531	3153	3094	3094
q22	382	302	305	302
Total cold run time: 111003 ms
Total hot run time: 39184 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4071	4068	4062	4062
q2	332	212	217	212
q3	2975	2958	2947	2947
q4	1892	1837	1854	1837
q5	5226	5235	5205	5205
q6	208	123	123	123
q7	2240	1793	1806	1793
q8	3250	3382	3325	3325
q9	8523	8492	8492	8492
q10	3750	3865	3792	3792
q11	545	472	452	452
q12	717	556	542	542
q13	13520	2866	2898	2866
q14	291	258	275	258
q15	508	471	477	471
q16	442	397	416	397
q17	1739	1666	1685	1666
q18	7722	7388	7224	7224
q19	1647	1654	1652	1652
q20	1930	1742	1720	1720
q21	5072	4709	4806	4709
q22	484	441	421	421
Total cold run time: 67084 ms
Total hot run time: 54166 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 180527 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9f34eb21390bd466e1ad8985b1f3471fdb4b0a98, data reload: false

query1	1220	394	1110	394
query2	6496	1951	1839	1839
query3	6669	219	227	219
query4	25012	21445	21417	21417
query5	4193	401	403	401
query6	283	193	194	193
query7	4600	310	305	305
query8	263	183	199	183
query9	8454	2247	2248	2247
query10	567	256	254	254
query11	14810	14363	14463	14363
query12	151	104	95	95
query13	1629	386	374	374
query14	8651	6803	6741	6741
query15	213	176	192	176
query16	7176	280	282	280
query17	990	633	601	601
query18	1914	288	296	288
query19	215	162	162	162
query20	100	94	98	94
query21	199	133	127	127
query22	5035	4820	4749	4749
query23	33694	32658	32716	32658
query24	12695	3201	3185	3185
query25	700	407	417	407
query26	1899	167	162	162
query27	3012	332	337	332
query28	6709	1870	1845	1845
query29	1392	597	601	597
query30	321	169	165	165
query31	1014	749	740	740
query32	101	65	60	60
query33	719	260	256	256
query34	1027	489	495	489
query35	849	696	701	696
query36	1007	873	877	873
query37	288	83	76	76
query38	3531	3486	3378	3378
query39	1579	1553	1543	1543
query40	292	137	138	137
query41	50	46	46	46
query42	110	105	102	102
query43	447	404	402	402
query44	1093	734	722	722
query45	279	236	264	236
query46	1060	805	766	766
query47	1934	1815	1847	1815
query48	375	319	309	309
query49	1164	382	365	365
query50	798	391	400	391
query51	6900	6707	6828	6707
query52	109	96	100	96
query53	366	292	291	291
query54	333	243	239	239
query55	85	83	81	81
query56	248	231	233	231
query57	1245	1127	1134	1127
query58	257	225	235	225
query59	2666	2529	2367	2367
query60	269	255	246	246
query61	111	108	106	106
query62	712	467	462	462
query63	314	282	280	280
query64	6455	3123	3245	3123
query65	3080	3002	2993	2993
query66	1453	349	323	323
query67	15669	14948	14871	14871
query68	8846	588	602	588
query69	554	339	322	322
query70	1234	1128	1083	1083
query71	479	278	271	271
query72	6316	2580	2402	2402
query73	923	323	327	323
query74	6877	6395	6467	6395
query75	3330	2334	2257	2257
query76	5196	1068	1221	1068
query77	612	260	252	252
query78	10714	10271	10172	10172
query79	8649	537	553	537
query80	1379	434	416	416
query81	515	246	234	234
query82	722	108	107	107
query83	201	164	165	164
query84	269	90	90	90
query85	1431	293	283	283
query86	407	279	284	279
query87	3595	3427	3449	3427
query88	3850	2266	2267	2266
query89	566	369	371	369
query90	1968	174	185	174
query91	134	103	104	103
query92	65	58	56	56
query93	6958	541	529	529
query94	1097	191	189	189
query95	432	325	329	325
query96	615	277	265	265
query97	2635	2561	2522	2522
query98	232	212	217	212
query99	1286	824	859	824
Total cold run time: 299338 ms
Total hot run time: 180527 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.92 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9f34eb21390bd466e1ad8985b1f3471fdb4b0a98, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.24	0.05	0.05
query4	1.66	0.07	0.06
query5	0.50	0.48	0.49
query6	1.15	0.65	0.66
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.57	0.52	0.52
query10	0.57	0.58	0.57
query11	0.15	0.12	0.11
query12	0.14	0.12	0.12
query13	0.61	0.59	0.59
query14	0.78	0.80	0.78
query15	0.87	0.83	0.83
query16	0.35	0.36	0.36
query17	0.96	0.98	0.97
query18	0.26	0.26	0.26
query19	1.85	1.71	1.66
query20	0.02	0.01	0.01
query21	15.43	0.72	0.70
query22	3.55	5.23	1.52
query23	17.47	1.34	1.10
query24	1.38	0.39	0.22
query25	0.14	0.10	0.09
query26	0.29	0.18	0.19
query27	0.08	0.08	0.08
query28	13.51	0.94	0.94
query29	12.74	3.47	3.35
query30	0.25	0.06	0.06
query31	2.85	0.39	0.39
query32	3.28	0.48	0.47
query33	2.86	2.89	2.85
query34	15.51	4.33	4.32
query35	4.37	4.38	4.39
query36	0.68	0.48	0.47
query37	0.19	0.17	0.17
query38	0.17	0.16	0.17
query39	0.05	0.04	0.04
query40	0.17	0.15	0.15
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 106.02 s
Total hot run time: 29.92 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 9f34eb21390bd466e1ad8985b1f3471fdb4b0a98 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       17.5 seconds inserted 10000000 Rows, about 571K ops/s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.64% (8881/24917)
Line Coverage: 27.38% (72908/266288)
Region Coverage: 26.56% (37705/141970)
Branch Coverage: 23.35% (19213/82286)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9f34eb21390bd466e1ad8985b1f3471fdb4b0a98_9f34eb21390bd466e1ad8985b1f3471fdb4b0a98/report/index.html

Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 2, 2024
Copy link
Contributor

github-actions bot commented Apr 2, 2024

PR approved by at least one committer and no changes requested.

Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@HappenLee HappenLee merged commit 8d9e12a into apache:master Apr 3, 2024
26 of 29 checks passed
@yiguolei yiguolei mentioned this pull request Apr 7, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants