Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](trino-connector) Fix the problem of reading the DATE type from Trino too slowly #37266

Merged
merged 1 commit into from
Jul 9, 2024

Conversation

BePPPower
Copy link
Contributor

@BePPPower BePPPower commented Jul 4, 2024

Previously, we use LocalDate.parse(DateTimeUtils.printDate(block.getInt(position, 0)), DateTimeFormatter.ISO_DATE) to handle DATE type from Trino.

Now, we use LocalDate.ofEpochDay(block.getInt(position, 0)) to handle DATE type from Trino.
This will result in a 20-fold speed improvement.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@BePPPower
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39876 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 66372bfb2c126d3fe5a06f8c9f1e379a708caac4, data reload: false

------ Round 1 ----------------------------------
q1	17614	4378	4309	4309
q2	2017	193	199	193
q3	10506	1185	1102	1102
q4	10196	858	822	822
q5	7505	2654	2634	2634
q6	220	136	133	133
q7	951	592	602	592
q8	9223	2060	2081	2060
q9	8859	6503	6458	6458
q10	9003	3689	3719	3689
q11	464	245	247	245
q12	409	239	233	233
q13	17772	2981	2978	2978
q14	273	221	218	218
q15	526	484	497	484
q16	520	376	380	376
q17	959	712	669	669
q18	8134	7533	7388	7388
q19	6059	1486	1497	1486
q20	671	333	339	333
q21	4871	3141	3149	3141
q22	402	345	333	333
Total cold run time: 117154 ms
Total hot run time: 39876 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4424	4383	4303	4303
q2	371	273	275	273
q3	2990	2860	2832	2832
q4	2061	1786	1669	1669
q5	5643	5503	5485	5485
q6	220	129	130	129
q7	2228	1864	1854	1854
q8	3258	3417	3415	3415
q9	8727	8679	8877	8679
q10	4081	4002	3698	3698
q11	580	495	514	495
q12	822	651	649	649
q13	16172	3223	3217	3217
q14	298	287	308	287
q15	522	502	478	478
q16	469	427	423	423
q17	1813	1517	1503	1503
q18	8123	7989	7836	7836
q19	1739	1667	1600	1600
q20	2179	1874	1836	1836
q21	5177	5060	4855	4855
q22	617	544	556	544
Total cold run time: 72514 ms
Total hot run time: 56060 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172354 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 66372bfb2c126d3fe5a06f8c9f1e379a708caac4, data reload: false

query1	909	380	378	378
query2	6487	2394	2206	2206
query3	6630	202	205	202
query4	19040	17588	17257	17257
query5	3699	496	461	461
query6	261	170	166	166
query7	4584	297	291	291
query8	298	266	281	266
query9	8500	2432	2423	2423
query10	592	285	266	266
query11	10554	9907	10116	9907
query12	115	80	80	80
query13	1632	359	363	359
query14	10152	6906	7320	6906
query15	225	189	188	188
query16	7759	303	302	302
query17	1570	543	514	514
query18	1962	267	270	267
query19	195	145	148	145
query20	87	81	79	79
query21	210	136	127	127
query22	4584	4139	3979	3979
query23	34013	33666	33464	33464
query24	11272	2895	2830	2830
query25	601	378	386	378
query26	734	153	164	153
query27	2328	317	330	317
query28	6049	2172	2174	2172
query29	905	632	610	610
query30	259	162	160	160
query31	963	752	750	750
query32	105	56	54	54
query33	738	299	296	296
query34	1038	477	475	475
query35	737	645	675	645
query36	1151	964	967	964
query37	138	81	80	80
query38	2973	2834	2804	2804
query39	907	824	806	806
query40	208	123	120	120
query41	54	48	51	48
query42	116	97	106	97
query43	599	532	549	532
query44	1206	750	725	725
query45	190	163	160	160
query46	1071	707	698	698
query47	1861	1802	1810	1802
query48	366	304	298	298
query49	845	409	404	404
query50	764	379	378	378
query51	6755	6783	6728	6728
query52	106	91	90	90
query53	356	282	279	279
query54	879	438	439	438
query55	71	71	72	71
query56	285	258	260	258
query57	1106	1037	1055	1037
query58	263	262	266	262
query59	3408	3338	3100	3100
query60	309	297	299	297
query61	116	115	108	108
query62	651	446	455	446
query63	324	288	299	288
query64	8697	2345	1863	1863
query65	3147	3092	3140	3092
query66	760	333	342	333
query67	15444	15177	15016	15016
query68	4569	544	560	544
query69	604	370	315	315
query70	1166	1078	1147	1078
query71	427	286	275	275
query72	7584	5530	5466	5466
query73	734	323	317	317
query74	6004	5490	5507	5490
query75	3485	2591	2671	2591
query76	2796	917	924	917
query77	622	297	296	296
query78	9617	8848	8840	8840
query79	6943	504	509	504
query80	1864	480	486	480
query81	567	222	219	219
query82	1343	107	101	101
query83	330	170	167	167
query84	273	82	97	82
query85	2004	267	268	267
query86	506	297	317	297
query87	3335	3084	3081	3081
query88	5048	2384	2351	2351
query89	488	382	372	372
query90	1838	181	185	181
query91	126	98	101	98
query92	62	49	49	49
query93	5487	519	510	510
query94	1192	217	206	206
query95	401	312	321	312
query96	618	270	268	268
query97	3176	3016	2964	2964
query98	221	202	196	196
query99	1158	862	825	825
Total cold run time: 279501 ms
Total hot run time: 172354 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.73 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 66372bfb2c126d3fe5a06f8c9f1e379a708caac4, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.66	0.07	0.08
query5	0.48	0.48	0.47
query6	1.14	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.49	0.50
query10	0.54	0.55	0.55
query11	0.15	0.12	0.12
query12	0.15	0.12	0.12
query13	0.60	0.60	0.59
query14	0.77	0.76	0.78
query15	0.85	0.84	0.81
query16	0.38	0.37	0.37
query17	0.96	1.00	0.99
query18	0.22	0.26	0.25
query19	1.81	1.68	1.84
query20	0.02	0.01	0.01
query21	15.38	0.78	0.66
query22	4.51	6.87	2.12
query23	18.27	1.46	1.25
query24	2.02	0.26	0.21
query25	0.16	0.09	0.08
query26	0.26	0.18	0.17
query27	0.09	0.09	0.08
query28	13.18	1.01	1.00
query29	12.65	3.31	3.27
query30	0.26	0.06	0.06
query31	2.84	0.38	0.38
query32	3.29	0.49	0.46
query33	2.88	2.96	2.89
query34	17.21	4.47	4.41
query35	4.49	4.45	4.49
query36	0.65	0.48	0.50
query37	0.19	0.16	0.15
query38	0.16	0.15	0.15
query39	0.04	0.04	0.03
query40	0.17	0.15	0.14
query41	0.10	0.04	0.04
query42	0.05	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.58 s
Total hot run time: 30.73 s

Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 5, 2024
Copy link
Contributor

github-actions bot commented Jul 5, 2024

PR approved by at least one committer and no changes requested.

Copy link
Contributor

github-actions bot commented Jul 5, 2024

PR approved by anyone and no changes requested.

@morningman morningman merged commit 6100f91 into apache:master Jul 9, 2024
28 of 31 checks passed
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
…m Trino too slowly (#37266)

Previously, we use
`LocalDate.parse(DateTimeUtils.printDate(block.getInt(position, 0)),
DateTimeFormatter.ISO_DATE)` to handle DATE type from Trino.

Now, we use `LocalDate.ofEpochDay(block.getInt(position, 0))` to handle
DATE type from Trino.
This will result in a 20-fold speed improvement.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.1-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants