Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[chore](be) Acquire and check MD5 digest of the file to download #37420

Merged
merged 3 commits into from
Jul 26, 2024
Merged

[chore](be) Acquire and check MD5 digest of the file to download #37420

merged 3 commits into from
Jul 26, 2024

Conversation

w41ter
Copy link
Contributor

@w41ter w41ter commented Jul 8, 2024 •
edited
Loading

Cherry-pick #35807, #36621, #36726

Required to upgrade curl 7.79.0 -> 8.2.1

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@w41ter w41ter changed the base branch from master to branch-2.0 July 8, 2024 02:49
@w41ter
Copy link
Contributor Author

w41ter commented Jul 8, 2024

run buildall

@w41ter
Copy link
Contributor Author

w41ter commented Jul 8, 2024

run buildall

github-actions[bot]
github-actions bot reviewed Jul 8, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/test/http/http_client_test.cpp
@@ -17,8 +17,11 @@

#include "http/http_client.h"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: 'http/http_client.h' file not found [clang-diagnostic-error]

#include "http/http_client.h"
         ^

@w41ter w41ter mentioned this pull request Jul 16, 2024
Merged
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@w41ter
[chore](third) Upgrade curl 7.79.0 -> 8.2.1 (#37893)
aca7ce9 
ref #23414.

It was acquired by #37420.
@dataroaring
Copy link
Contributor

run buildall

1 similar comment
@w41ter
Copy link
Contributor Author

w41ter commented Jul 26, 2024

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.81% (8118/21468)
Line Coverage: 29.48% (66571/225841)
Region Coverage: 28.96% (34319/118501)
Branch Coverage: 24.84% (17632/70996)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a10a374f3359e45a3651783cb8ceb3ec40658ee1_a10a374f3359e45a3651783cb8ceb3ec40658ee1/report/index.html

@w41ter
Copy link
Contributor Author

w41ter commented Jul 26, 2024

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.81% (8118/21468)
Line Coverage: 29.48% (66570/225841)
Region Coverage: 28.96% (34316/118501)
Branch Coverage: 24.83% (17630/70996)
Coverage Report: http://coverage.selectdb-in.cc/coverage/6a39e8d3409995b02e34d80a164faf2bfad9d53b_6a39e8d3409995b02e34d80a164faf2bfad9d53b/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 50435 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6a39e8d3409995b02e34d80a164faf2bfad9d53b, data reload: false

------ Round 1 ----------------------------------
q1	18959	4465	4366	4366
q2	2036	156	151	151
q3	10445	1917	1953	1917
q4	11669	1276	1340	1276
q5	8473	4006	4036	4006
q6	242	132	130	130
q7	2149	1669	1694	1669
q8	9554	2781	2765	2765
q9	11299	10864	10513	10513
q10	10178	3561	3558	3558
q11	434	254	252	252
q12	471	313	312	312
q13	20469	3971	4025	3971
q14	359	331	330	330
q15	512	456	469	456
q16	669	564	560	560
q17	1129	967	957	957
q18	7150	6837	6952	6837
q19	1771	1672	1610	1610
q20	537	324	298	298
q21	4392	4097	4073	4073
q22	529	438	428	428
Total cold run time: 123426 ms
Total hot run time: 50435 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4352	4388	4309	4309
q2	321	226	223	223
q3	4160	4125	4139	4125
q4	2751	2726	2723	2723
q5	7253	7112	7095	7095
q6	237	122	117	117
q7	3263	2855	2802	2802
q8	4341	4462	4487	4462
q9	17367	17135	17091	17091
q10	4205	4265	4224	4224
q11	756	711	659	659
q12	1021	877	861	861
q13	6825	3702	3725	3702
q14	459	426	417	417
q15	501	457	443	443
q16	724	668	687	668
q17	3775	3847	3836	3836
q18	8928	8758	8803	8758
q19	1730	1727	1676	1676
q20	2348	2118	2078	2078
q21	8496	8456	8431	8431
q22	1033	975	994	975
Total cold run time: 84846 ms
Total hot run time: 79675 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 203589 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6a39e8d3409995b02e34d80a164faf2bfad9d53b, data reload: false

query1	916	419	378	378
query2	6525	2719	2478	2478
query3	6922	214	202	202
query4	21012	18003	18031	18003
query5	19747	6508	6598	6508
query6	286	227	239	227
query7	4174	302	309	302
query8	438	400	476	400
query9	3123	2708	2642	2642
query10	407	297	295	295
query11	11325	10750	10813	10750
query12	133	75	75	75
query13	5607	718	697	697
query14	17462	13273	13770	13273
query15	365	234	268	234
query16	6490	284	262	262
query17	1715	1449	870	870
query18	2318	404	417	404
query19	214	152	148	148
query20	81	77	80	77
query21	191	94	94	94
query22	5257	5052	5068	5052
query23	32570	32047	31808	31808
query24	6937	6502	6485	6485
query25	519	430	419	419
query26	520	164	160	160
query27	1884	291	294	291
query28	6152	2406	2358	2358
query29	2878	2767	2899	2767
query30	250	168	165	165
query31	896	721	726	721
query32	68	63	63	63
query33	418	268	260	260
query34	856	468	487	468
query35	1103	945	935	935
query36	1191	1157	1077	1077
query37	87	62	59	59
query38	3109	2943	2896	2896
query39	1376	1330	1327	1327
query40	201	95	94	94
query41	47	43	45	43
query42	87	90	77	77
query43	735	693	587	587
query44	1117	724	721	721
query45	242	236	237	236
query46	1244	970	972	970
query47	1786	1717	1929	1717
query48	1022	702	722	702
query49	631	374	373	373
query50	865	601	612	601
query51	4910	4672	4635	4635
query52	89	88	85	85
query53	456	328	323	323
query54	2661	2427	2463	2427
query55	87	75	84	75
query56	226	215	198	198
query57	1237	1130	1108	1108
query58	211	208	196	196
query59	4095	4108	3976	3976
query60	219	205	214	205
query61	102	96	98	96
query62	810	477	453	453
query63	483	341	342	341
query64	2606	1530	1400	1400
query65	3625	3508	3613	3508
query66	778	380	387	380
query67	15835	15649	15368	15368
query68	10042	659	679	659
query69	582	334	351	334
query70	1812	1669	1372	1372
query71	415	323	319	319
query72	6563	3465	3513	3465
query73	736	323	326	323
query74	6367	5904	5901	5901
query75	5389	3737	3700	3700
query76	6445	1172	1115	1115
query77	1099	251	254	251
query78	12661	11928	11897	11897
query79	7754	647	669	647
query80	1185	398	407	398
query81	495	233	246	233
query82	1545	101	97	97
query83	167	134	132	132
query84	264	71	71	71
query85	895	335	341	335
query86	324	309	292	292
query87	3262	3052	3030	3030
query88	4543	2322	2323	2322
query89	404	287	294	287
query90	1987	210	195	195
query91	185	140	143	140
query92	59	52	54	52
query93	6096	625	566	566
query94	688	211	213	211
query95	1126	1050	1040	1040
query96	645	315	318	315
query97	6550	6402	6354	6354
query98	187	176	179	176
query99	3032	941	835	835
Total cold run time: 315433 ms
Total hot run time: 203589 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.78 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6a39e8d3409995b02e34d80a164faf2bfad9d53b, data reload: false

query1	0.02	0.03	0.02
query2	0.06	0.02	0.02
query3	0.25	0.05	0.05
query4	1.83	0.06	0.06
query5	0.56	0.53	0.53
query6	1.27	0.61	0.62
query7	0.01	0.01	0.01
query8	0.04	0.03	0.02
query9	0.55	0.47	0.49
query10	0.54	0.54	0.53
query11	0.12	0.08	0.08
query12	0.12	0.09	0.08
query13	0.63	0.62	0.61
query14	0.78	0.78	0.78
query15	0.77	0.76	0.75
query16	0.37	0.37	0.38
query17	1.05	1.02	1.00
query18	0.24	0.20	0.29
query19	1.95	1.81	1.83
query20	0.01	0.02	0.02
query21	15.46	0.55	0.56
query22	2.17	2.24	1.62
query23	16.87	1.04	0.97
query24	3.43	2.29	1.98
query25	0.39	0.09	0.04
query26	0.54	0.15	0.15
query27	0.05	0.04	0.04
query28	7.44	0.71	0.75
query29	12.72	2.36	2.39
query30	0.54	0.51	0.54
query31	2.80	0.38	0.38
query32	3.37	0.49	0.49
query33	3.06	3.07	3.08
query34	15.27	4.84	4.78
query35	4.88	4.87	4.84
query36	1.06	1.02	1.02
query37	0.05	0.05	0.05
query38	0.03	0.02	0.02
query39	0.02	0.02	0.02
query40	0.16	0.13	0.14
query41	0.07	0.02	0.01
query42	0.02	0.01	0.02
query43	0.02	0.01	0.01
Total cold run time: 101.59 s
Total hot run time: 31.78 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 6a39e8d3409995b02e34d80a164faf2bfad9d53b with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.5 seconds inserted 10000000 Rows, about 465K ops/s

@w41ter w41ter merged commit 7fc4ddc into apache:branch-2.0 Jul 26, 2024
21 of 24 checks passed
@w41ter w41ter deleted the pick/md5_check_to_2.0 branch July 26, 2024 06:03
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@w41ter
[chore](third) Upgrade curl 7.79.0 -> 8.2.1 (apache#37893)
3a6f6a7 
ref apache#23414.

It was acquired by apache#37420.
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@w41ter @Baymine
[chore](be) Acquire and check MD5 digest of the file to download (apa…
8939fe7 
…che#37420)

Cherry-pick apache#35807, apache#36621, apache#36726
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@platoneko platoneko Awaiting requested review from platoneko

@gavinchou gavinchou Awaiting requested review from gavinchou

@dataroaring dataroaring Awaiting requested review from dataroaring

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@w41ter @doris-robot @dataroaring