Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](multi-catalog)resolve hive meta store compatibility for different version issues #32551

Merged
merged 2 commits into from
Mar 22, 2024
Merged

[fix](multi-catalog)resolve hive meta store compatibility for different version issues #32551

merged 2 commits into from
Mar 22, 2024

Conversation

wsjz
Copy link
Contributor

@wsjz wsjz commented Mar 20, 2024 •
edited by morningman
Loading

Proposed changes

issue: #31442

Fix hive list partition at different version.
Only Hive3 uses the prependCatalogToDbName() to wrap db_name.

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@wsjz
Copy link
Contributor Author

wsjz commented Mar 20, 2024

run buildall

@wsjz
Copy link
Contributor Author

wsjz commented Mar 21, 2024

run external

@wsjz wsjz changed the title [fix](multi-catalog)resolve hive listPartitions compatibility issues [fix](multi-catalog)resolve hive meta store compatibility for different version issues Mar 21, 2024
@wsjz
Copy link
Contributor Author

wsjz commented Mar 21, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37420 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c5250319431a23d9bf23e431c011938b15766a8f, data reload: false

------ Round 1 ----------------------------------
q1	17638	4473	4087	4087
q2	2024	143	137	137
q3	10616	1162	1208	1162
q4	10224	780	774	774
q5	7465	2925	2909	2909
q6	205	124	126	124
q7	1023	573	561	561
q8	9313	2001	1979	1979
q9	7021	6467	6402	6402
q10	8416	3365	3465	3365
q11	441	224	213	213
q12	397	190	191	190
q13	17789	2852	2851	2851
q14	219	216	198	198
q15	502	444	460	444
q16	444	350	346	346
q17	944	511	555	511
q18	7128	6357	6423	6357
q19	2106	1401	1449	1401
q20	542	266	248	248
q21	3491	2903	2863	2863
q22	359	298	308	298
Total cold run time: 108307 ms
Total hot run time: 37420 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4096	4047	4067	4047
q2	323	228	233	228
q3	2956	2817	2846	2817
q4	1821	1548	1575	1548
q5	5199	5246	5216	5216
q6	195	118	121	118
q7	2237	1810	1846	1810
q8	3129	3269	3260	3260
q9	8501	8517	8528	8517
q10	3674	3644	3637	3637
q11	541	449	450	449
q12	715	561	515	515
q13	16921	2873	2845	2845
q14	276	246	263	246
q15	485	446	456	446
q16	450	416	418	416
q17	1728	1472	1453	1453
q18	7432	7179	7093	7093
q19	1596	1520	1569	1520
q20	1902	1698	1702	1698
q21	4870	4649	4624	4624
q22	519	448	457	448
Total cold run time: 69566 ms
Total hot run time: 52951 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 181754 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c5250319431a23d9bf23e431c011938b15766a8f, data reload: false

query1	920	351	345	345
query2	7423	2030	2005	2005
query3	6785	206	228	206
query4	32447	20704	20766	20704
query5	4641	537	419	419
query6	1052	178	174	174
query7	5637	293	286	286
query8	225	162	169	162
query9	10006	2364	2323	2323
query10	472	254	293	254
query11	17655	14584	14505	14505
query12	144	98	95	95
query13	1670	426	413	413
query14	13819	10391	10329	10329
query15	324	205	210	205
query16	8070	253	257	253
query17	2087	553	522	522
query18	1906	276	271	271
query19	340	154	157	154
query20	93	83	83	83
query21	198	121	127	121
query22	4672	4438	4439	4438
query23	31969	31098	31072	31072
query24	10687	2850	2804	2804
query25	620	383	390	383
query26	1062	155	156	155
query27	2420	363	350	350
query28	7470	1916	1898	1898
query29	872	639	625	625
query30	297	143	143	143
query31	963	733	736	733
query32	94	57	56	56
query33	771	253	257	253
query34	1080	492	488	488
query35	825	609	613	609
query36	981	873	864	864
query37	119	79	78	78
query38	3519	3431	3415	3415
query39	1462	1355	1358	1355
query40	212	111	109	109
query41	48	46	45	45
query42	103	96	97	96
query43	492	465	471	465
query44	1087	718	718	718
query45	254	266	268	266
query46	1089	696	686	686
query47	1669	1609	1582	1582
query48	432	355	368	355
query49	1096	337	333	333
query50	757	367	374	367
query51	6724	6627	6627	6627
query52	99	87	104	87
query53	346	273	273	273
query54	311	238	242	238
query55	86	76	78	76
query56	251	217	217	217
query57	1106	1030	1017	1017
query58	224	200	196	196
query59	2888	2574	2505	2505
query60	272	242	249	242
query61	95	96	91	91
query62	581	402	390	390
query63	301	273	270	270
query64	5409	3852	3842	3842
query65	3094	3019	2985	2985
query66	793	332	337	332
query67	14922	14619	14537	14537
query68	5802	514	514	514
query69	583	373	383	373
query70	1305	1168	1141	1141
query71	473	291	286	286
query72	6400	2772	2610	2610
query73	719	320	313	313
query74	7194	6737	6712	6712
query75	3884	2923	2790	2790
query76	4276	888	975	888
query77	635	261	263	261
query78	10332	9557	9550	9550
query79	8425	510	525	510
query80	2037	402	406	402
query81	558	207	222	207
query82	1613	201	203	201
query83	315	144	148	144
query84	288	81	86	81
query85	1704	392	362	362
query86	479	287	279	279
query87	3713	3521	3567	3521
query88	4879	2322	2362	2322
query89	545	375	365	365
query90	1961	175	178	175
query91	178	133	154	133
query92	60	46	46	46
query93	6678	493	479	479
query94	1214	175	173	173
query95	432	328	315	315
query96	586	263	265	263
query97	3059	2891	2882	2882
query98	226	207	201	201
query99	1111	779	760	760
Total cold run time: 312202 ms
Total hot run time: 181754 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit c5250319431a23d9bf23e431c011938b15766a8f with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       20.8 seconds inserted 10000000 Rows, about 480K ops/s

@wsjz wsjz force-pushed the compatible_hive_get_parts branch from c525031 to ceb54d8 Compare March 22, 2024 06:57
@wsjz wsjz force-pushed the compatible_hive_get_parts branch from ceb54d8 to f119ad8 Compare March 22, 2024 07:00
@wsjz
Copy link
Contributor Author

wsjz commented Mar 22, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37576 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f119ad85ae9071b23fe43541fc9470d7eefb5804, data reload: false

------ Round 1 ----------------------------------
q1	17651	4226	4078	4078
q2	2108	149	145	145
q3	10595	1148	1156	1148
q4	10219	773	688	688
q5	7451	2995	2974	2974
q6	201	125	123	123
q7	1013	573	553	553
q8	9340	1951	2000	1951
q9	7010	6477	6442	6442
q10	8386	3359	3475	3359
q11	448	224	212	212
q12	421	202	198	198
q13	17802	2833	2893	2833
q14	227	198	206	198
q15	518	467	465	465
q16	498	376	370	370
q17	942	538	656	538
q18	7144	6524	6503	6503
q19	1992	1432	1419	1419
q20	543	253	248	248
q21	3601	2909	2839	2839
q22	344	292	292	292
Total cold run time: 108454 ms
Total hot run time: 37576 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4068	4060	4048	4048
q2	324	239	243	239
q3	2969	2880	2810	2810
q4	1810	1564	1589	1564
q5	5260	5242	5219	5219
q6	193	117	119	117
q7	2218	1868	1849	1849
q8	3157	3302	3267	3267
q9	8563	8557	8614	8557
q10	3722	3687	3668	3668
q11	541	441	455	441
q12	719	598	531	531
q13	17209	2889	2875	2875
q14	288	257	256	256
q15	511	460	455	455
q16	472	432	413	413
q17	1720	1497	1472	1472
q18	7540	7285	7095	7095
q19	5563	1485	1554	1485
q20	1924	1760	1703	1703
q21	4829	4764	4618	4618
q22	526	462	433	433
Total cold run time: 74126 ms
Total hot run time: 53115 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 181820 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f119ad85ae9071b23fe43541fc9470d7eefb5804, data reload: false

query1	959	367	364	364
query2	7504	2090	2283	2090
query3	6775	223	225	223
query4	31969	21289	21222	21222
query5	4489	433	437	433
query6	693	181	180	180
query7	5928	296	288	288
query8	457	190	180	180
query9	10411	2348	2312	2312
query10	564	248	264	248
query11	15406	14149	14058	14058
query12	136	92	85	85
query13	1648	418	421	418
query14	11386	11766	10640	10640
query15	334	198	191	191
query16	8182	263	260	260
query17	2131	570	561	561
query18	2025	294	289	289
query19	364	155	155	155
query20	93	88	95	88
query21	202	138	125	125
query22	4583	4474	4385	4385
query23	31653	32116	31124	31124
query24	9790	2887	2789	2789
query25	516	398	385	385
query26	701	162	157	157
query27	2157	350	354	350
query28	5874	1881	1918	1881
query29	849	640	585	585
query30	298	148	144	144
query31	988	748	708	708
query32	104	56	54	54
query33	636	243	252	243
query34	839	462	471	462
query35	695	600	604	600
query36	999	888	881	881
query37	107	74	75	74
query38	3598	3410	3454	3410
query39	1404	1510	1369	1369
query40	218	115	109	109
query41	49	46	45	45
query42	106	93	93	93
query43	481	456	467	456
query44	1084	755	724	724
query45	261	250	262	250
query46	1079	672	680	672
query47	1653	1591	1591	1591
query48	443	361	366	361
query49	1047	339	336	336
query50	757	371	359	359
query51	6583	6538	6734	6538
query52	109	90	90	90
query53	335	278	282	278
query54	281	240	254	240
query55	85	79	81	79
query56	245	221	226	221
query57	1065	1011	1010	1010
query58	227	202	208	202
query59	2768	2699	2572	2572
query60	353	250	250	250
query61	96	107	99	99
query62	577	387	412	387
query63	309	274	280	274
query64	4979	3811	4009	3811
query65	3035	3005	3015	3005
query66	863	368	352	352
query67	14923	14418	14431	14418
query68	5382	513	525	513
query69	565	390	382	382
query70	1162	1122	1166	1122
query71	414	297	285	285
query72	6125	2852	2644	2644
query73	706	314	313	313
query74	7745	6510	6654	6510
query75	3442	2802	2769	2769
query76	3569	888	844	844
query77	427	282	267	267
query78	10511	9643	9633	9633
query79	9322	539	516	516
query80	2274	388	373	373
query81	569	209	210	209
query82	1542	199	200	199
query83	295	140	139	139
query84	286	83	80	80
query85	1479	344	314	314
query86	494	289	310	289
query87	3777	3526	3540	3526
query88	5111	2273	2271	2271
query89	522	361	368	361
query90	2012	175	171	171
query91	165	136	135	135
query92	63	48	47	47
query93	7533	503	488	488
query94	1284	177	174	174
query95	434	321	323	321
query96	611	275	262	262
query97	3058	2882	2868	2868
query98	229	222	208	208
query99	1205	754	766	754
Total cold run time: 303771 ms
Total hot run time: 181820 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit f119ad85ae9071b23fe43541fc9470d7eefb5804 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s

@wsjz
Copy link
Contributor Author

wsjz commented Mar 22, 2024

run feut

@wsjz
Copy link
Contributor Author

wsjz commented Mar 22, 2024

run fe ut

morningman
morningman approved these changes Mar 22, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 22, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

kaka11chen
kaka11chen approved these changes Mar 22, 2024
View reviewed changes
Copy link
Contributor

@kaka11chen kaka11chen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit df56ab3 into apache:master Mar 22, 2024
27 of 30 checks passed
yiguolei pushed a commit that referenced this pull request Mar 24, 2024
@wsjz @Doris-Extras
[fix](multi-catalog)resolve hive meta store compatibility for differe…
ac8ba43 
…nt version issues (#32551)

Fix hive list partition at different version.
Only Hive3 uses the prependCatalogToDbName() to wrap db_name.
@wsjz wsjz deleted the compatible_hive_get_parts branch November 18, 2024 01:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morningman morningman morningman approved these changes

@kaka11chen kaka11chen kaka11chen approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@wsjz @doris-robot @kaka11chen @morningman