Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix](partial-update) Fix partial update fail when the datetime default value is 'current_time' #32926 #33209

Merged
merged 3 commits into from
Apr 3, 2024

Conversation

Yukang-Lian
Copy link
Collaborator

Problem:
When partially updating columns without specifying the auto-increment column, and the imported data contains new keys, an error stating the auto-increment column could not be found occurs.

Reason:
The logic for partial column updates does not account for new keys in auto-increment columns. Since auto-increment columns can be generated by the system, it's possible to omit this column data during import. However, partial column updates treat this as a regular column, expecting it to be nullable or have a default value for automatic filling, overlooking the fact that auto-increment columns can also be auto-filled. This oversight leads to the error.

Solution:
Incorporate a check for auto-increment columns into the partial column update logic, and include the logic for generating auto-increment column values in the process of completing partial updates.

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at [email protected] by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/exec/tablet_info.h Show resolved Hide resolved
be/src/exec/tablet_info.h Show resolved Hide resolved
@Yukang-Lian
Copy link
Collaborator Author

run buildall

…des auto increment column (apache#31725)

Problem:
When partially updating columns without specifying the auto-increment column, and the imported data contains new keys, an error stating the auto-increment column could not be found occurs.

Reason:
The logic for partial column updates does not account for new keys in auto-increment columns. Since auto-increment columns can be generated by the system, it's possible to omit this column data during import. However, partial column updates treat this as a regular column, expecting it to be nullable or have a default value for automatic filling, overlooking the fact that auto-increment columns can also be auto-filled. This oversight leads to the error.

Solution:
Incorporate a check for auto-increment columns into the partial column update logic, and include the logic for generating auto-increment column values in the process of completing partial updates.
@Yukang-Lian
Copy link
Collaborator Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 50370 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e38893c858769d4c0fbdf3aad3c4ab76a35344ea, data reload: false

------ Round 1 ----------------------------------
q1	18428	4805	4581	4581
q2	2676	157	145	145
q3	11021	2162	1985	1985
q4	10114	1261	1335	1261
q5	8488	3969	3983	3969
q6	234	123	122	122
q7	2009	1587	1590	1587
q8	9266	2729	2727	2727
q9	11046	10544	10429	10429
q10	8747	3579	3527	3527
q11	410	237	239	237
q12	454	295	302	295
q13	18369	3975	4035	3975
q14	352	318	317	317
q15	505	452	463	452
q16	702	592	588	588
q17	1140	970	995	970
q18	7210	6831	6939	6831
q19	1680	1616	1549	1549
q20	513	337	332	332
q21	4473	4211	4086	4086
q22	492	405	406	405
Total cold run time: 118329 ms
Total hot run time: 50370 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4361	4303	4271	4271
q2	327	220	219	219
q3	4139	4119	4117	4117
q4	2760	2722	2773	2722
q5	7302	7246	7190	7190
q6	234	119	117	117
q7	3256	2854	2813	2813
q8	4363	4489	4525	4489
q9	17143	17093	17023	17023
q10	4226	4257	4249	4249
q11	750	695	687	687
q12	1018	838	860	838
q13	7129	3681	3746	3681
q14	445	417	428	417
q15	497	457	465	457
q16	771	696	710	696
q17	3834	3871	3880	3871
q18	8725	8771	8917	8771
q19	1716	1721	1658	1658
q20	2353	2133	2118	2118
q21	8515	8534	8573	8534
q22	1027	947	999	947
Total cold run time: 84891 ms
Total hot run time: 79885 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 201505 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e38893c858769d4c0fbdf3aad3c4ab76a35344ea, data reload: false

query1	907	398	392	392
query2	6530	2140	2122	2122
query3	6926	210	214	210
query4	21301	18046	17984	17984
query5	19739	6519	6492	6492
query6	300	216	222	216
query7	4166	292	307	292
query8	254	243	238	238
query9	3166	2696	2629	2629
query10	402	303	288	288
query11	11294	10714	10700	10700
query12	118	72	71	71
query13	5584	646	652	646
query14	17993	13570	13811	13570
query15	367	234	232	232
query16	6464	275	264	264
query17	1720	1441	853	853
query18	2352	403	401	401
query19	197	140	150	140
query20	83	74	79	74
query21	187	94	92	92
query22	5223	4999	5003	4999
query23	32260	31913	32007	31913
query24	6938	6599	6499	6499
query25	524	439	419	419
query26	615	161	155	155
query27	2025	294	295	294
query28	6084	2289	2236	2236
query29	2849	2600	2766	2600
query30	246	168	161	161
query31	912	722	746	722
query32	70	59	56	56
query33	385	248	250	248
query34	855	454	478	454
query35	1105	925	924	924
query36	1469	1151	1256	1151
query37	95	60	66	60
query38	3080	2887	2960	2887
query39	1359	1322	1313	1313
query40	225	100	93	93
query41	40	33	31	31
query42	79	81	82	81
query43	744	603	569	569
query44	1112	723	743	723
query45	238	227	230	227
query46	1232	984	978	978
query47	1970	1725	1756	1725
query48	955	679	669	669
query49	617	364	366	364
query50	872	597	638	597
query51	4760	4713	4675	4675
query52	85	79	73	73
query53	441	312	308	308
query54	2622	2452	2507	2452
query55	79	84	79	79
query56	210	201	205	201
query57	1264	1132	1017	1017
query58	210	194	175	175
query59	3616	3050	3114	3050
query60	199	199	187	187
query61	89	80	80	80
query62	827	488	478	478
query63	477	329	328	328
query64	2607	1504	1335	1335
query65	3624	3540	3574	3540
query66	805	374	355	355
query67	15470	15353	15938	15353
query68	9166	655	660	655
query69	567	334	362	334
query70	1651	1355	1314	1314
query71	406	299	310	299
query72	6392	3382	3410	3382
query73	730	324	325	324
query74	6332	5886	5846	5846
query75	5478	3685	3780	3685
query76	5618	1185	1195	1185
query77	934	257	251	251
query78	12688	11748	11796	11748
query79	8394	633	635	633
query80	1189	383	387	383
query81	490	232	235	232
query82	1645	98	94	94
query83	160	130	131	130
query84	261	67	70	67
query85	864	277	273	273
query86	336	285	288	285
query87	3210	3056	3035	3035
query88	5018	2393	2364	2364
query89	384	287	303	287
query90	1946	208	208	208
query91	149	119	114	114
query92	56	50	48	48
query93	5944	591	599	591
query94	701	199	200	199
query95	1125	1071	1049	1049
query96	650	334	338	334
query97	6490	6456	6332	6332
query98	193	177	175	175
query99	2904	952	909	909
Total cold run time: 313684 ms
Total hot run time: 201505 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.93 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e38893c858769d4c0fbdf3aad3c4ab76a35344ea, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.24	0.05	0.04
query4	1.80	0.05	0.06
query5	0.53	0.53	0.52
query6	1.28	0.61	0.62
query7	0.01	0.01	0.01
query8	0.03	0.02	0.03
query9	0.52	0.47	0.49
query10	0.55	0.54	0.52
query11	0.11	0.09	0.09
query12	0.12	0.09	0.09
query13	0.63	0.62	0.62
query14	0.80	0.77	0.81
query15	0.78	0.77	0.76
query16	0.36	0.36	0.37
query17	1.00	1.01	1.05
query18	0.22	0.26	0.27
query19	1.95	1.87	1.84
query20	0.02	0.01	0.00
query21	15.50	0.55	0.55
query22	2.16	2.78	2.01
query23	17.34	0.98	0.96
query24	4.90	0.75	1.10
query25	0.36	0.06	0.07
query26	0.59	0.16	0.17
query27	0.05	0.03	0.04
query28	8.76	0.72	0.73
query29	12.62	2.31	2.28
query30	0.58	0.52	0.52
query31	2.82	0.38	0.36
query32	3.40	0.50	0.49
query33	3.10	3.11	3.05
query34	15.25	4.80	4.80
query35	4.84	4.87	4.84
query36	1.06	1.02	1.02
query37	0.06	0.05	0.05
query38	0.04	0.02	0.02
query39	0.02	0.02	0.02
query40	0.16	0.14	0.13
query41	0.06	0.01	0.01
query42	0.03	0.01	0.01
query43	0.02	0.01	0.02
Total cold run time: 104.76 s
Total hot run time: 30.93 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit e38893c858769d4c0fbdf3aad3c4ab76a35344ea with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       20.3 seconds inserted 10000000 Rows, about 492K ops/s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.82% (8056/21301)
Line Coverage: 29.48% (65799/223161)
Region Coverage: 28.95% (33854/116956)
Branch Coverage: 24.80% (17381/70098)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e38893c858769d4c0fbdf3aad3c4ab76a35344ea_e38893c858769d4c0fbdf3aad3c4ab76a35344ea/report/index.html

@xiaokang xiaokang merged commit 9e50bd8 into apache:branch-2.0 Apr 3, 2024
23 of 26 checks passed
@xiaokang xiaokang changed the title [Cherry-pick](branch-2.0) Pick "[Fix](partial update) Fix partial update load false when schema includes auto increment column (#31725)" [Fix](partial-update) Fix partial update fail when the datetime default value is 'current_time' #32926 Apr 3, 2024
@xiaokang
Copy link
Contributor

xiaokang commented Apr 3, 2024

#32926

mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants