Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[enhancement](cloud) support BE http action: list_cache and clear #41037

Merged
merged 8 commits into from
Nov 7, 2024

Conversation

yagagagaga
Copy link
Contributor

@yagagagaga yagagagaga commented Sep 20, 2024

Proposed changes

Add a http action which is useful when you debug.

API

GET /api/file_cache

request parameter

request parameter1

param type desc require
op string the value must be list_cache, other value you can refer to #40831 #37484 yes
value string the segment file name yes

request parameter2

param type desc require
op string the value must be clear, other value you can refer to #40831 #37484 yes
value string the segment file name yes
sync bool clean local cache in sync no

response

response1

if success

param type desc
array return the segment file cache in local path

if fail

param type desc
array empty array

response2

if success

param type desc
status string
msg string

example

case 1

curl  '172.100.0.4:8040/api/file_cache?op=list_cache&value=0200000000000001bf42c14374fff491ffb7c89a1a65c5bb_0.dat'

return

["/opt/doris/be/file_cache/c6a/c6a599f453f67f0949f80ad9990fa3dd/0"]

case 2

curl '127.0.0.1:8040/api/file_cache?op=clear&sync=true&value=0200000000000001284b68fea3dcfe8a83e65cd88426b081_0.dat'

return

{
    "status": "OK",
    "msg": "OK"
}

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@yagagagaga yagagagaga marked this pull request as draft September 20, 2024 03:54
be/src/http/action/file_cache_action.cpp Outdated Show resolved Hide resolved
be/src/io/cache/block_file_cache_factory.h Outdated Show resolved Hide resolved
be/src/io/cache/block_file_cache.cpp Outdated Show resolved Hide resolved
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@yagagagaga yagagagaga marked this pull request as ready for review October 22, 2024 12:54
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@yagagagaga
Copy link
Contributor Author

run buidlall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@yagagagaga yagagagaga changed the title [enhancement](cloud) support BE http action: hash_path and clear [enhancement](cloud) support BE http action: list_cache and clear Oct 22, 2024
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@yagagagaga
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 41499 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 166281bcefca5e6980e799a69f9f12e17b0ba255, data reload: false

------ Round 1 ----------------------------------
q1	17685	7511	7267	7267
q2	2014	290	306	290
q3	11590	1082	1181	1082
q4	10593	891	754	754
q5	7752	3137	3142	3137
q6	244	155	152	152
q7	1055	642	610	610
q8	9357	1974	2027	1974
q9	6677	6548	6453	6453
q10	7091	2457	2466	2457
q11	447	247	234	234
q12	410	215	217	215
q13	17754	2994	3018	2994
q14	238	213	213	213
q15	584	520	522	520
q16	642	602	598	598
q17	998	535	528	528
q18	7273	6799	6836	6799
q19	1372	992	926	926
q20	483	192	191	191
q21	4124	3340	3104	3104
q22	1104	1008	1001	1001
Total cold run time: 109487 ms
Total hot run time: 41499 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7305	7295	7249	7249
q2	327	229	232	229
q3	3061	3036	2876	2876
q4	2100	1841	1799	1799
q5	5779	5791	5781	5781
q6	241	146	152	146
q7	2263	1866	1796	1796
q8	3401	3593	3467	3467
q9	9020	9002	8898	8898
q10	3587	3602	3570	3570
q11	605	491	503	491
q12	820	635	663	635
q13	9941	3179	3149	3149
q14	302	280	300	280
q15	578	532	536	532
q16	686	647	631	631
q17	1876	1613	1615	1613
q18	8365	7805	7654	7654
q19	1746	1511	1603	1511
q20	2123	1923	1899	1899
q21	5625	5423	5458	5423
q22	1162	1035	1027	1027
Total cold run time: 70913 ms
Total hot run time: 60656 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.46% (9707/25915)
Line Coverage: 28.73% (80529/280319)
Region Coverage: 28.15% (41634/147906)
Branch Coverage: 24.72% (21151/85574)
Coverage Report: http://coverage.selectdb-in.cc/coverage/166281bcefca5e6980e799a69f9f12e17b0ba255_166281bcefca5e6980e799a69f9f12e17b0ba255/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 191663 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 166281bcefca5e6980e799a69f9f12e17b0ba255, data reload: false

query1	967	420	406	406
query2	6262	2088	2030	2030
query3	8688	197	214	197
query4	34200	23562	23627	23562
query5	3608	490	510	490
query6	289	169	184	169
query7	4201	294	283	283
query8	285	227	225	225
query9	9542	2713	2710	2710
query10	472	276	266	266
query11	17982	15193	15356	15193
query12	153	104	100	100
query13	1550	422	426	422
query14	9407	7476	6981	6981
query15	263	175	186	175
query16	8036	472	466	466
query17	1641	623	592	592
query18	2164	320	327	320
query19	367	161	179	161
query20	137	115	122	115
query21	213	108	103	103
query22	4941	4659	4738	4659
query23	35332	34067	33825	33825
query24	10854	2805	2724	2724
query25	626	425	409	409
query26	1138	162	165	162
query27	2271	280	283	280
query28	7415	2453	2429	2429
query29	837	445	417	417
query30	272	152	153	152
query31	1154	752	804	752
query32	104	56	54	54
query33	769	289	310	289
query34	936	527	514	514
query35	871	779	767	767
query36	1132	949	962	949
query37	159	98	93	93
query38	4070	3879	3927	3879
query39	1473	1423	1414	1414
query40	213	105	100	100
query41	49	49	47	47
query42	130	99	103	99
query43	529	472	489	472
query44	1250	825	806	806
query45	196	165	170	165
query46	1160	711	704	704
query47	1946	1835	1854	1835
query48	424	324	320	320
query49	918	441	431	431
query50	801	391	383	383
query51	7091	7047	6946	6946
query52	100	89	90	89
query53	252	181	180	180
query54	1183	436	440	436
query55	78	80	75	75
query56	314	295	284	284
query57	1330	1136	1162	1136
query58	227	236	238	236
query59	3219	2968	2948	2948
query60	303	275	266	266
query61	108	100	101	100
query62	896	668	665	665
query63	218	188	175	175
query64	4013	627	597	597
query65	3275	3187	3238	3187
query66	833	296	303	296
query67	16240	15671	15629	15629
query68	4550	565	557	557
query69	466	287	308	287
query70	1192	1109	1127	1109
query71	382	294	277	277
query72	7213	4184	3974	3974
query73	779	354	349	349
query74	10102	9093	8968	8968
query75	3408	2670	2684	2670
query76	2881	992	937	937
query77	417	297	298	297
query78	10435	9650	9642	9642
query79	1302	588	584	584
query80	2074	441	451	441
query81	565	243	238	238
query82	732	141	139	139
query83	280	139	132	132
query84	272	70	69	69
query85	1310	292	312	292
query86	368	302	310	302
query87	4519	4313	4398	4313
query88	3008	2192	2152	2152
query89	405	285	288	285
query90	2196	184	184	184
query91	152	103	103	103
query92	65	48	52	48
query93	1061	525	530	525
query94	1053	291	285	285
query95	345	239	254	239
query96	611	279	285	279
query97	3262	3190	3147	3147
query98	216	198	194	194
query99	1573	1319	1304	1304
Total cold run time: 299271 ms
Total hot run time: 191663 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.03 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 166281bcefca5e6980e799a69f9f12e17b0ba255, data reload: false

query1	0.03	0.04	0.03
query2	0.05	0.03	0.03
query3	0.23	0.07	0.07
query4	1.67	0.10	0.10
query5	0.51	0.51	0.50
query6	1.15	0.72	0.72
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.55	0.49	0.50
query10	0.57	0.55	0.54
query11	0.14	0.12	0.11
query12	0.14	0.11	0.10
query13	0.61	0.62	0.60
query14	2.84	2.86	2.73
query15	0.89	0.83	0.83
query16	0.38	0.40	0.38
query17	1.00	1.00	1.03
query18	0.20	0.20	0.20
query19	1.87	1.87	2.04
query20	0.01	0.01	0.01
query21	15.35	0.60	0.56
query22	2.75	2.76	1.33
query23	17.07	0.89	0.94
query24	2.49	1.46	2.20
query25	0.18	0.17	0.19
query26	0.41	0.15	0.15
query27	0.05	0.05	0.05
query28	9.63	1.10	1.06
query29	12.55	3.36	3.36
query30	0.25	0.07	0.06
query31	2.87	0.38	0.39
query32	3.28	0.46	0.46
query33	3.01	3.00	3.05
query34	16.98	4.47	4.47
query35	4.49	4.52	4.51
query36	0.67	0.50	0.47
query37	0.08	0.06	0.06
query38	0.05	0.04	0.03
query39	0.04	0.02	0.02
query40	0.16	0.13	0.12
query41	0.08	0.02	0.03
query42	0.04	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 105.41 s
Total hot run time: 33.03 s

@gavinchou
Copy link
Contributor

list what has been cleared, a file list with size

Copy link
Contributor

@freemandealer freemandealer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

github-actions bot commented Nov 5, 2024

PR approved by anyone and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 5, 2024
Copy link
Contributor

github-actions bot commented Nov 5, 2024

PR approved by at least one committer and no changes requested.

@gavinchou gavinchou merged commit 99d0748 into apache:master Nov 7, 2024
25 of 27 checks passed
yagagagaga added a commit to yagagagaga/doris that referenced this pull request Nov 7, 2024
…ache#41037)

## Proposed changes

Add a http action which is useful when you debug.

### API
```http
GET /api/file_cache
```

### request parameter

#### request parameter1

|param|type|desc|require|
|:---|:---|:---|:---|
|op|string|the value must be `list_cache`, other value you can refer to
apache#40831 apache#37484 |yes|
|value|string|the segment file name |yes|

#### request parameter2

|param|type|desc|require|
|:---|:---|:---|:---|
|op|string|the value must be `clear`, other value you can refer to
apache#40831 apache#37484 |yes|
|value|string|the segment file name |yes|
|sync|bool|clean local cache in sync |no|

### response

#### response1

if success
|param|type|desc|
|:---|:---|:---|
||array|return the segment file cache in local path|

if fail
|param|type|desc|
|:---|:---|:---|
||array|empty array|

#### response2

if success
|param|type|desc|
|:---|:---|:---|
|status|string||
|msg|string||

### example

#### case 1

```bash
curl  '172.100.0.4:8040/api/file_cache?op=list_cache&value=0200000000000001bf42c14374fff491ffb7c89a1a65c5bb_0.dat'
```

return
```json
["/opt/doris/be/file_cache/c6a/c6a599f453f67f0949f80ad9990fa3dd/0"]
```

#### case 2

```bash
curl '127.0.0.1:8040/api/file_cache?op=clear&sync=true&value=0200000000000001284b68fea3dcfe8a83e65cd88426b081_0.dat'
```

return
```json
{
    "status": "OK",
    "msg": "OK"
}
```

(cherry picked from commit 99d0748)
gavinchou pushed a commit that referenced this pull request Nov 14, 2024
…and clear (#41037) (#43412)

## Proposed changes

Add a http action which is useful when you debug.

### API
```http
GET /api/file_cache
```

### request parameter

#### request parameter1

|param|type|desc|require|
|:---|:---|:---|:---|
|op|string|the value must be `list_cache`, other value you can refer to
#40831 #37484 |yes|
|value|string|the segment file name |yes|

#### request parameter2

|param|type|desc|require|
|:---|:---|:---|:---|
|op|string|the value must be `clear`, other value you can refer to
#40831 #37484 |yes|
|value|string|the segment file name |yes|
|sync|bool|clean local cache in sync |no|

### response

#### response1

if success
|param|type|desc|
|:---|:---|:---|
||array|return the segment file cache in local path|

if fail
|param|type|desc|
|:---|:---|:---|
||array|empty array|

#### response2

if success
|param|type|desc|
|:---|:---|:---|
|status|string||
|msg|string||

### example

#### case 1

```bash
curl  '172.100.0.4:8040/api/file_cache?op=list_cache&value=0200000000000001bf42c14374fff491ffb7c89a1a65c5bb_0.dat'
```

return
```json
["/opt/doris/be/file_cache/c6a/c6a599f453f67f0949f80ad9990fa3dd/0"]
```

#### case 2

```bash
curl '127.0.0.1:8040/api/file_cache?op=clear&sync=true&value=0200000000000001284b68fea3dcfe8a83e65cd88426b081_0.dat'
```

return
```json
{
    "status": "OK",
    "msg": "OK"
}
```

(cherry picked from commit 99d0748)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.3-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants