Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[bugfix](iceberg)fix datetime conversion error and data path error #35708

Merged
merged 3 commits into from
May 31, 2024
Merged

[bugfix](iceberg)fix datetime conversion error and data path error #35708

merged 3 commits into from
May 31, 2024

Conversation

wuwenchi
Copy link
Contributor

@wuwenchi wuwenchi commented May 31, 2024 •
edited
Loading

Proposed changes

Issue #31442

  1. The unit of the seventh parameter of ZonedDateTime.of is nanosecond, so we should multiply the microsecond by 1000.
  2. When writing to a non-partitioned iceberg table, the data path has an extra slash

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@wuwenchi
Copy link
Contributor Author

run buildall

1 similar comment
@wuwenchi
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 42119 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e0a13b7ee0daebda2d0d36bf6a50fc0193b94422, data reload: false

------ Round 1 ----------------------------------
q1	17624	4322	4753	4322
q2	2037	203	206	203
q3	10419	1218	1117	1117
q4	10180	835	836	835
q5	7482	2705	2764	2705
q6	217	139	141	139
q7	980	620	617	617
q8	9222	2153	2074	2074
q9	10685	6716	6726	6716
q10	9251	3932	3909	3909
q11	443	255	241	241
q12	491	231	231	231
q13	17314	3203	3240	3203
q14	280	240	215	215
q15	515	471	493	471
q16	465	386	393	386
q17	979	686	702	686
q18	8440	7741	7801	7741
q19	4254	1570	1580	1570
q20	624	345	323	323
q21	5139	4142	4068	4068
q22	400	347	348	347
Total cold run time: 117441 ms
Total hot run time: 42119 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4628	4418	4378	4378
q2	377	267	274	267
q3	3166	2943	2906	2906
q4	1867	1617	1644	1617
q5	5507	5436	5498	5436
q6	217	125	126	125
q7	2138	1851	1788	1788
q8	3208	3366	3385	3366
q9	8623	8712	8634	8634
q10	3971	3684	3771	3684
q11	597	514	502	502
q12	809	630	633	630
q13	17221	3223	3198	3198
q14	312	264	272	264
q15	516	495	497	495
q16	498	450	444	444
q17	1825	1514	1452	1452
q18	7581	7620	7394	7394
q19	1684	1574	1596	1574
q20	1975	1761	1776	1761
q21	4970	4772	4697	4697
q22	636	520	523	520
Total cold run time: 72326 ms
Total hot run time: 55132 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 168858 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e0a13b7ee0daebda2d0d36bf6a50fc0193b94422, data reload: false

query1	934	393	372	372
query2	6438	2450	2320	2320
query3	6633	200	201	200
query4	19194	17587	17384	17384
query5	4102	411	429	411
query6	247	170	152	152
query7	4586	311	289	289
query8	329	279	310	279
query9	8602	2392	2372	2372
query10	461	277	256	256
query11	10650	10094	10023	10023
query12	137	92	97	92
query13	1647	402	355	355
query14	9248	6807	6803	6803
query15	238	190	185	185
query16	7452	271	265	265
query17	1301	517	521	517
query18	1917	271	268	268
query19	192	150	149	149
query20	90	87	89	87
query21	199	128	135	128
query22	4276	3884	3861	3861
query23	33562	32911	33053	32911
query24	10835	2869	3014	2869
query25	569	355	361	355
query26	703	155	153	153
query27	2262	323	318	318
query28	5804	2044	2047	2044
query29	826	609	583	583
query30	251	151	158	151
query31	969	760	746	746
query32	98	53	54	53
query33	668	266	277	266
query34	880	472	472	472
query35	726	621	601	601
query36	1066	916	924	916
query37	101	67	65	65
query38	2854	2798	2748	2748
query39	836	798	785	785
query40	194	122	122	122
query41	55	52	55	52
query42	106	99	94	94
query43	615	554	577	554
query44	1111	722	738	722
query45	194	175	174	174
query46	1060	709	732	709
query47	1852	1764	1766	1764
query48	372	299	290	290
query49	843	370	382	370
query50	764	388	371	371
query51	6777	6654	6554	6554
query52	109	90	92	90
query53	356	284	289	284
query54	900	428	441	428
query55	75	71	69	69
query56	253	236	243	236
query57	1147	1012	1047	1012
query58	251	210	204	204
query59	3356	3382	3021	3021
query60	286	248	255	248
query61	91	86	88	86
query62	604	445	469	445
query63	315	278	285	278
query64	8521	2210	1736	1736
query65	3173	3104	3148	3104
query66	814	328	324	324
query67	15338	14807	15016	14807
query68	4522	523	519	519
query69	450	271	269	269
query70	1062	1087	1101	1087
query71	372	274	272	272
query72	7277	2733	2579	2579
query73	729	332	316	316
query74	5959	5662	5633	5633
query75	3346	2639	2618	2618
query76	2326	922	919	919
query77	395	264	264	264
query78	10138	9962	9804	9804
query79	1702	517	506	506
query80	1113	442	435	435
query81	546	231	221	221
query82	945	95	88	88
query83	251	170	185	170
query84	254	93	83	83
query85	1198	298	316	298
query86	417	309	319	309
query87	3255	3083	3201	3083
query88	3193	2392	2347	2347
query89	485	396	376	376
query90	1997	193	188	188
query91	137	107	109	107
query92	59	50	49	49
query93	1324	511	497	497
query94	1215	199	194	194
query95	410	316	320	316
query96	581	277	265	265
query97	3180	3020	3068	3020
query98	244	230	222	222
query99	1179	870	852	852
Total cold run time: 261676 ms
Total hot run time: 168858 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.95 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e0a13b7ee0daebda2d0d36bf6a50fc0193b94422, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.68	0.08	0.09
query5	0.49	0.49	0.50
query6	1.14	0.73	0.72
query7	0.02	0.01	0.01
query8	0.06	0.04	0.04
query9	0.54	0.49	0.50
query10	0.55	0.55	0.54
query11	0.16	0.11	0.12
query12	0.14	0.12	0.12
query13	0.60	0.58	0.60
query14	0.79	0.77	0.79
query15	0.82	0.81	0.81
query16	0.37	0.37	0.36
query17	0.95	1.01	0.97
query18	0.24	0.24	0.25
query19	1.74	1.73	1.67
query20	0.01	0.01	0.01
query21	15.72	0.65	0.65
query22	4.67	6.73	2.27
query23	18.31	1.46	1.28
query24	1.29	0.43	0.23
query25	0.15	0.08	0.08
query26	0.26	0.16	0.17
query27	0.08	0.08	0.08
query28	13.33	1.00	0.99
query29	13.76	3.35	3.29
query30	0.24	0.05	0.05
query31	2.88	0.39	0.37
query32	3.28	0.48	0.47
query33	2.87	2.91	2.92
query34	17.26	4.39	4.43
query35	4.48	4.52	4.66
query36	0.65	0.47	0.46
query37	0.18	0.16	0.15
query38	0.15	0.14	0.15
query39	0.04	0.04	0.04
query40	0.17	0.15	0.17
query41	0.10	0.05	0.04
query42	0.05	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.61 s
Total hot run time: 30.95 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.29% (9232/25442)
Line Coverage: 27.64% (75713/273971)
Region Coverage: 26.85% (39205/145994)
Branch Coverage: 23.60% (19894/84286)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e0a13b7ee0daebda2d0d36bf6a50fc0193b94422_e0a13b7ee0daebda2d0d36bf6a50fc0193b94422/report/index.html

@wuwenchi wuwenchi changed the title [bugfix](iceberg)datetime conversion error [bugfix](iceberg)fix datetime conversion error and data path error May 31, 2024
@morningman
Copy link
Contributor

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 41603 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9ba07f34179eae32453851c872c1cb7510ee1fa4, data reload: false

------ Round 1 ----------------------------------
q1	17639	4428	4210	4210
q2	2026	198	196	196
q3	10440	1258	1139	1139
q4	10194	865	908	865
q5	7457	2690	2710	2690
q6	226	136	142	136
q7	967	638	614	614
q8	9220	2121	2130	2121
q9	9360	6708	6669	6669
q10	9601	3902	3928	3902
q11	448	245	252	245
q12	422	246	231	231
q13	17262	3212	3274	3212
q14	266	206	222	206
q15	503	467	472	467
q16	486	386	395	386
q17	969	607	702	607
q18	8384	7699	7699	7699
q19	3753	1295	1444	1295
q20	652	303	308	303
q21	5231	4096	4121	4096
q22	394	335	314	314
Total cold run time: 115900 ms
Total hot run time: 41603 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4545	4438	4519	4438
q2	387	266	266	266
q3	3145	2925	2937	2925
q4	1939	1658	1609	1609
q5	5459	5535	5510	5510
q6	229	129	135	129
q7	2180	1846	1817	1817
q8	3283	3393	3408	3393
q9	8700	8649	8725	8649
q10	4173	3898	3965	3898
q11	604	494	507	494
q12	791	600	592	592
q13	16950	3200	3186	3186
q14	301	271	290	271
q15	517	487	503	487
q16	486	443	438	438
q17	1804	1513	1546	1513
q18	8710	8639	8006	8006
q19	1866	1535	1639	1535
q20	3148	1856	1874	1856
q21	8980	4854	4850	4850
q22	597	510	521	510
Total cold run time: 78794 ms
Total hot run time: 56372 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.29% (9234/25447)
Line Coverage: 27.64% (75730/274011)
Region Coverage: 26.85% (39198/146001)
Branch Coverage: 23.60% (19895/84288)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9ba07f34179eae32453851c872c1cb7510ee1fa4_9ba07f34179eae32453851c872c1cb7510ee1fa4/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 170306 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9ba07f34179eae32453851c872c1cb7510ee1fa4, data reload: false

query1	923	375	377	375
query2	6446	2440	2302	2302
query3	6635	212	203	203
query4	19400	17382	17268	17268
query5	4107	447	446	446
query6	245	165	157	157
query7	4583	305	294	294
query8	320	286	275	275
query9	8421	2404	2368	2368
query10	450	306	282	282
query11	10723	10317	9985	9985
query12	137	88	94	88
query13	1623	356	357	356
query14	8892	7830	7590	7590
query15	236	189	189	189
query16	8066	265	271	265
query17	1748	523	539	523
query18	2004	270	276	270
query19	205	151	154	151
query20	93	86	89	86
query21	206	136	146	136
query22	4401	3969	4182	3969
query23	33773	33042	33096	33042
query24	10724	2820	2894	2820
query25	592	354	368	354
query26	1223	167	157	157
query27	2851	323	336	323
query28	7350	2041	2045	2041
query29	882	629	644	629
query30	267	154	150	150
query31	984	769	754	754
query32	94	56	59	56
query33	800	305	277	277
query34	974	464	463	463
query35	753	623	616	616
query36	1114	935	898	898
query37	158	69	72	69
query38	2988	2763	2767	2763
query39	863	785	791	785
query40	207	125	124	124
query41	52	50	51	50
query42	122	97	97	97
query43	597	551	574	551
query44	1221	736	748	736
query45	204	167	163	163
query46	1064	734	703	703
query47	1874	1733	1770	1733
query48	379	299	290	290
query49	998	399	442	399
query50	770	385	376	376
query51	6722	6639	6656	6639
query52	108	92	87	87
query53	359	288	287	287
query54	878	460	458	458
query55	72	69	72	69
query56	287	249	263	249
query57	1127	1033	1039	1033
query58	262	244	245	244
query59	3400	3187	3139	3139
query60	301	264	270	264
query61	97	88	90	88
query62	650	447	457	447
query63	315	290	294	290
query64	8970	2243	1735	1735
query65	3178	3063	3093	3063
query66	837	348	331	331
query67	15272	14832	14748	14748
query68	4512	545	544	544
query69	452	302	303	302
query70	1155	1074	1169	1074
query71	397	291	292	291
query72	7266	2867	2704	2704
query73	769	328	328	328
query74	6103	5573	5650	5573
query75	3426	2664	2732	2664
query76	2526	881	942	881
query77	446	300	295	295
query78	10455	9860	9723	9723
query79	2262	514	516	514
query80	1900	491	483	483
query81	542	225	220	220
query82	752	109	102	102
query83	306	181	183	181
query84	264	90	91	90
query85	1921	313	311	311
query86	498	310	292	292
query87	3422	3129	3106	3106
query88	4126	2350	2348	2348
query89	476	394	383	383
query90	1831	194	191	191
query91	136	116	109	109
query92	65	51	53	51
query93	2505	520	505	505
query94	1311	208	198	198
query95	427	331	317	317
query96	596	273	265	265
query97	3204	3067	3014	3014
query98	255	223	214	214
query99	1330	865	882	865
Total cold run time: 272213 ms
Total hot run time: 170306 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.53 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9ba07f34179eae32453851c872c1cb7510ee1fa4, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.06
query4	1.67	0.09	0.10
query5	0.49	0.50	0.52
query6	1.11	0.72	0.71
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.54	0.48	0.50
query10	0.54	0.55	0.54
query11	0.16	0.12	0.11
query12	0.14	0.12	0.12
query13	0.60	0.59	0.60
query14	0.76	0.76	0.78
query15	0.82	0.82	0.81
query16	0.36	0.36	0.38
query17	1.03	1.01	0.96
query18	0.23	0.24	0.25
query19	1.89	1.80	1.67
query20	0.01	0.01	0.01
query21	15.46	0.70	0.68
query22	4.70	6.74	1.96
query23	18.28	1.38	1.25
query24	2.02	0.20	0.20
query25	0.14	0.09	0.09
query26	0.26	0.17	0.17
query27	0.08	0.08	0.08
query28	13.32	1.02	1.01
query29	13.30	3.26	3.30
query30	0.24	0.06	0.05
query31	2.87	0.39	0.38
query32	3.28	0.46	0.47
query33	2.84	2.92	2.88
query34	17.29	4.38	4.44
query35	4.47	4.52	4.70
query36	0.65	0.45	0.46
query37	0.17	0.16	0.16
query38	0.15	0.14	0.14
query39	0.05	0.03	0.03
query40	0.16	0.14	0.15
query41	0.09	0.04	0.04
query42	0.05	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 110.67 s
Total hot run time: 30.53 s

@morningman morningman added dev/2.1.x usercase Important user case type label labels May 31, 2024
morningman
morningman approved these changes May 31, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 31, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

kaka11chen
kaka11chen approved these changes May 31, 2024
View reviewed changes
Copy link
Contributor

@kaka11chen kaka11chen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yiguolei yiguolei merged commit 3beb92b into apache:master May 31, 2024
25 of 29 checks passed
yiguolei pushed a commit that referenced this pull request May 31, 2024
@wuwenchi @Doris-Extras
[bugfix](iceberg)fix datetime conversion error and data path error (#…
cb96a79 
…35708)

## Proposed changes
Issue #31442

<!--Describe your changes.-->

1. The unit of the seventh parameter of `ZonedDateTime.of` is
nanosecond, so we should multiply the microsecond by 1000.
2. When writing to a non-partitioned iceberg table, the data path has an
extra slash
@yiguolei yiguolei mentioned this pull request Jun 1, 2024
Closed
@wuwenchi wuwenchi mentioned this pull request Jun 4, 2024
Merged
dataroaring pushed a commit that referenced this pull request Jun 4, 2024
@wuwenchi @dataroaring
[bugfix](iceberg)fix datetime conversion error and data path error (#…
9c36bdd 
…35708)

## Proposed changes
Issue #31442

<!--Describe your changes.-->

1. The unit of the seventh parameter of `ZonedDateTime.of` is
nanosecond, so we should multiply the microsecond by 1000.
2. When writing to a non-partitioned iceberg table, the data path has an
extra slash
seawinde pushed a commit to seawinde/doris that referenced this pull request Jun 5, 2024
@wuwenchi @seawinde
[bugfix](iceberg)fix datetime conversion error and data path error (a…
36720b2 
…pache#35708)

## Proposed changes
Issue apache#31442

<!--Describe your changes.-->

1. The unit of the seventh parameter of `ZonedDateTime.of` is
nanosecond, so we should multiply the microsecond by 1000.
2. When writing to a non-partitioned iceberg table, the data path has an
extra slash
morningman pushed a commit that referenced this pull request Jun 6, 2024
@wuwenchi
[bugfix](iceberg)fix datetime conversion error for 2.0 (#35831)
ceee062 
bp:  #35708
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@wuwenchi
[bugfix](iceberg)fix datetime conversion error for 2.0 (apache#35831)
9bc870f 
bp:  apache#35708
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morningman morningman morningman approved these changes

@kaka11chen kaka11chen kaka11chen approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.12-merged dev/2.1.4-merged dev/3.0.0-merged reviewed usercase Important user case type label
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@wuwenchi @doris-robot @morningman @kaka11chen @yiguolei @dataroaring