Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feature](http) Support to acquire md5 digest of the file to download #35807

Merged
merged 1 commit into from
Jun 4, 2024
Merged

[feature](http) Support to acquire md5 digest of the file to download #35807

merged 1 commit into from
Jun 4, 2024

Conversation

w41ter
Copy link
Contributor

@w41ter w41ter commented Jun 3, 2024

Proposed changes

Issue Number: close #xxx

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@w41ter
Copy link
Contributor Author

w41ter commented Jun 3, 2024

run buildall

github-actions[bot]
github-actions bot reviewed Jun 3, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/test/http/http_client_test.cpp
@@ -17,8 +17,11 @@

#include "http/http_client.h"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: 'http/http_client.h' file not found [clang-diagnostic-error]

#include "http/http_client.h"
         ^

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.11% (9194/25462)
Line Coverage: 27.46% (75326/274321)
Region Coverage: 26.74% (39093/146215)
Branch Coverage: 23.50% (19834/84388)
Coverage Report: http://coverage.selectdb-in.cc/coverage/5d8f5fe78e5546437a716889a9a6a41bf98e7c1e_5d8f5fe78e5546437a716889a9a6a41bf98e7c1e/report/index.html

gavinchou
gavinchou reviewed Jun 3, 2024
View reviewed changes
be/src/http/utils.cpp
if (is_acquire_md5) {
Md5Digest md5;

void* buf = mmap(nullptr, file_size, PROT_READ, MAP_SHARED, fd, 0);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need to check return values of mmap() and munmap()

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jun 3, 2024

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 40768 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5d8f5fe78e5546437a716889a9a6a41bf98e7c1e, data reload: false

------ Round 1 ----------------------------------
q1	17606	4319	4308	4308
q2	2007	196	207	196
q3	10442	1218	1149	1149
q4	10192	874	687	687
q5	7477	2729	2714	2714
q6	222	142	134	134
q7	982	617	613	613
q8	9213	2113	2089	2089
q9	9111	6664	6693	6664
q10	9389	3888	3868	3868
q11	449	249	245	245
q12	471	251	236	236
q13	18649	3340	3066	3066
q14	269	236	242	236
q15	521	485	470	470
q16	528	398	386	386
q17	985	696	634	634
q18	8349	7725	7799	7725
q19	5798	1310	1435	1310
q20	656	332	326	326
q21	5150	3368	3948	3368
q22	392	347	344	344
Total cold run time: 118858 ms
Total hot run time: 40768 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4741	4413	4373	4373
q2	363	267	265	265
q3	3138	2975	2827	2827
q4	1920	1660	1704	1660
q5	5492	5492	5471	5471
q6	224	131	133	131
q7	2157	1840	1835	1835
q8	3190	3392	3338	3338
q9	8675	8717	8742	8717
q10	4068	3837	3675	3675
q11	585	508	488	488
q12	789	637	637	637
q13	16584	3101	3147	3101
q14	288	275	261	261
q15	524	491	482	482
q16	478	428	429	428
q17	1776	1508	1475	1475
q18	7907	7518	7283	7283
q19	1680	1495	1437	1437
q20	2072	1792	1772	1772
q21	4949	4668	4871	4668
q22	714	538	541	538
Total cold run time: 72314 ms
Total hot run time: 54862 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 169827 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5d8f5fe78e5546437a716889a9a6a41bf98e7c1e, data reload: false

query1	934	381	359	359
query2	6457	2485	2520	2485
query3	6645	206	206	206
query4	22012	17294	17635	17294
query5	4148	470	449	449
query6	255	158	153	153
query7	4575	292	290	290
query8	299	277	281	277
query9	8640	2421	2393	2393
query10	446	304	303	303
query11	10585	9972	9982	9972
query12	130	87	89	87
query13	1631	365	360	360
query14	10125	6851	7634	6851
query15	229	189	177	177
query16	7812	263	265	263
query17	1651	516	526	516
query18	1950	274	271	271
query19	203	161	158	158
query20	91	82	85	82
query21	210	128	129	128
query22	4369	4187	4230	4187
query23	33532	33049	33018	33018
query24	11005	2853	2705	2705
query25	624	346	356	346
query26	1452	156	154	154
query27	2919	316	349	316
query28	7210	2071	2063	2063
query29	953	613	609	609
query30	278	149	151	149
query31	957	722	730	722
query32	94	52	54	52
query33	754	284	280	280
query34	941	469	469	469
query35	751	624	622	622
query36	1105	974	1001	974
query37	168	71	74	71
query38	2873	2708	2715	2708
query39	921	774	789	774
query40	199	125	120	120
query41	54	54	49	49
query42	120	96	101	96
query43	603	549	568	549
query44	1199	732	748	732
query45	194	176	169	169
query46	1073	741	678	678
query47	1870	1757	1789	1757
query48	362	294	290	290
query49	1011	408	410	408
query50	787	380	384	380
query51	6920	6708	6731	6708
query52	110	92	94	92
query53	359	289	289	289
query54	906	463	445	445
query55	73	76	73	73
query56	279	283	255	255
query57	1109	1044	1037	1037
query58	286	248	237	237
query59	3556	3576	3295	3295
query60	294	265	258	258
query61	91	87	88	87
query62	634	441	450	441
query63	319	299	303	299
query64	10026	2263	1740	1740
query65	3188	3090	3111	3090
query66	1373	325	329	325
query67	15453	14853	15158	14853
query68	4557	543	543	543
query69	569	444	386	386
query70	1176	1103	1144	1103
query71	424	291	291	291
query72	6880	2719	2518	2518
query73	755	319	321	319
query74	5900	5508	5624	5508
query75	3388	2644	2687	2644
query76	2624	998	910	910
query77	461	305	304	304
query78	10179	9721	9668	9668
query79	1531	517	508	508
query80	792	461	450	450
query81	571	218	218	218
query82	731	101	101	101
query83	192	170	168	168
query84	253	88	85	85
query85	1417	274	270	270
query86	463	334	306	306
query87	3300	3053	3076	3053
query88	3893	2349	2367	2349
query89	498	391	397	391
query90	1831	194	191	191
query91	141	109	110	109
query92	70	51	121	51
query93	1627	508	498	498
query94	1217	186	183	183
query95	398	301	315	301
query96	585	273	271	271
query97	3220	3050	3048	3048
query98	232	224	207	207
query99	1226	847	830	830
Total cold run time: 273483 ms
Total hot run time: 169827 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.34 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5d8f5fe78e5546437a716889a9a6a41bf98e7c1e, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.23	0.06	0.04
query4	1.67	0.10	0.09
query5	0.53	0.49	0.50
query6	1.12	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.53	0.48	0.49
query10	0.54	0.55	0.55
query11	0.16	0.11	0.10
query12	0.14	0.12	0.12
query13	0.60	0.59	0.59
query14	0.78	0.77	0.81
query15	0.83	0.81	0.82
query16	0.37	0.37	0.37
query17	1.03	1.01	0.95
query18	0.24	0.23	0.25
query19	1.81	1.70	1.80
query20	0.01	0.01	0.01
query21	15.74	0.66	0.64
query22	4.18	8.03	1.60
query23	18.28	1.37	1.30
query24	1.26	0.43	0.26
query25	0.14	0.10	0.08
query26	0.27	0.18	0.18
query27	0.08	0.08	0.06
query28	13.34	1.01	0.99
query29	13.82	3.29	3.27
query30	0.24	0.07	0.06
query31	2.83	0.39	0.39
query32	3.27	0.47	0.45
query33	2.89	2.93	2.92
query34	17.11	4.46	4.45
query35	4.59	4.51	4.74
query36	0.65	0.45	0.46
query37	0.18	0.16	0.15
query38	0.15	0.14	0.14
query39	0.04	0.04	0.03
query40	0.16	0.15	0.14
query41	0.08	0.04	0.04
query42	0.05	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.16 s
Total hot run time: 30.34 s

dataroaring
dataroaring approved these changes Jun 4, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 585662b into apache:master Jun 4, 2024
28 of 32 checks passed
dataroaring pushed a commit that referenced this pull request Jun 4, 2024
@w41ter @dataroaring
[improvement](http) Support to acquire md5 digest of the file to down…
2b9d769 
…load (#35807)
seawinde pushed a commit to seawinde/doris that referenced this pull request Jun 5, 2024
@w41ter @seawinde
[improvement](http) Support to acquire md5 digest of the file to down…
75f64c4 
…load (apache#35807)
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Jul 8, 2024
@w41ter
[improvement](http) Support to acquire md5 digest of the file to down…
caa1869 
…load

Cherry-pick apache#35807
@w41ter w41ter mentioned this pull request Jul 8, 2024
Merged
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Jul 8, 2024
@w41ter
[improvement](http) Support to acquire md5 digest of the file to down…
c1b600d 
…load

Cherry-pick apache#35807
@w41ter w41ter mentioned this pull request Jul 8, 2024
Merged
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Jul 8, 2024
@w41ter
[improvement](http) Support to acquire md5 digest of the file to down…
e06cc2a 
…load

Cherry-pick apache#35807
@w41ter w41ter deleted the feature/md5_of_downloading_file branch July 8, 2024 03:20
dataroaring pushed a commit that referenced this pull request Jul 8, 2024
@w41ter
[chore](be) Acquire and check MD5 digest of the file to download (#37418
5280e27 
)

Cherry-pick #35807, #36621, #36726
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Jul 26, 2024
@w41ter
[improvement](http) Support to acquire md5 digest of the file to down…
f89f45a 
…load

Cherry-pick apache#35807
w41ter added a commit that referenced this pull request Jul 26, 2024
@w41ter
[chore](be) Acquire and check MD5 digest of the file to download (#37420
7fc4ddc 
)

Cherry-pick #35807, #36621, #36726
@w41ter w41ter removed the dev/2.0.x label Jul 26, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@w41ter @Baymine
[chore](be) Acquire and check MD5 digest of the file to download (apa…
8939fe7 
…che#37420)

Cherry-pick apache#35807, apache#36621, apache#36726
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@dataroaring dataroaring dataroaring approved these changes

@gavinchou gavinchou gavinchou approved these changes

Assignees
No one assigned
Labels
dev/2.0.14-merged dev/2.1.5-merged dev/3.0.0-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@w41ter @doris-robot @gavinchou @dataroaring