Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(api): support metric API Prometheus format & add statistic metric api #2286

Merged
merged 16 commits into from
Oct 8, 2023
Merged

feat(api): support metric API Prometheus format & add statistic metric api #2286

merged 16 commits into from
Oct 8, 2023

Conversation

SunnyBoy-WYH
Copy link
Contributor

@SunnyBoy-WYH SunnyBoy-WYH commented Aug 16, 2023 •
edited by imbajin
Loading

fix #1656 #1356 (will enhance it later)

术语：
统计指标：指【请求总数、成功数、失败数、平均响应时间、最大响应时间】五个指标

基础指标：指serverReporter中原有的gauges、counters、histograms等

思路：
image

1.增加原有Metric接口的prom格式返回：

# HELP hugegraph_info
# TYPE hugegraph_info untyped
hugegraph_info{version="0.69",} 1.0
# HELP org_apache_hugegraph_backend_cache_Cache_edge_hugegraph_capacity
# TYPE org_apache_hugegraph_backend_cache_Cache_edge_hugegraph_capacity gauge
org_apache_hugegraph_backend_cache_Cache_edge_hugegraph_capacity 1000000
# HELP org_apache_hugegraph_backend_cache_Cache_edge_hugegraph_expire
# TYPE org_apache_hugegraph_backend_cache_Cache_edge_hugegraph_expire gauge
org_apache_hugegraph_backend_cache_Cache_edge_hugegraph_expire 600000
# HELP org_apache_hugegraph_backend_cache_Cache_edge_hugegraph_hits

2.增加statistics接口，返回【请求总数、成功数、失败数、平均响应时间、最大响应时间】五个指标，同时支持prop格式返回
json：

{
"metrics/GET":{
"FAILED_REQUEST":0,
"MEAN_RESPONSE_TIME":10.047648352712768,
"MAX_RESPONSE_TIME":27,
"SUCCESS_REQUEST":12,
"TOTAL_REQUEST":12
},
"metrics/statistics/GET":{
"FAILED_REQUEST":0,
"MEAN_RESPONSE_TIME":0.6234681965773761,
"MAX_RESPONSE_TIME":2,
"SUCCESS_REQUEST":3,
"TOTAL_REQUEST":3
}
}
prom格式：

# HELP hugegraph_info
# TYPE hugegraph_info untyped
hugegraph_info{version="0.69",} 1.0
# HELP metrics_GET
# TYPE metrics_GET gauge
metrics_GET{name=FAILED_REQUEST,} 0
metrics_GET{name=MEAN_RESPONSE_TIME,} 10.047648352712768
metrics_GET{name=MAX_RESPONSE_TIME,} 27
metrics_GET{name=SUCCESS_REQUEST,} 12
metrics_GET{name=TOTAL_REQUEST,} 12

3.需要讨论一下 statistics接口的数据是否要合并在Metric接口中，目前没有合并是因为，原有指标结果JSON中只包含counter、summary等五种标准指标 ；

另外分开还有个好处是，如果后续想要按照图、接口查询统计指标，比较方便定制

@SunnyBoy-WYH SunnyBoy-WYH changed the title Feat：metric api support promtheus return format Feat：metric api support promtheus return format & add statistic metric api Aug 16, 2023
JackyYangPassion
JackyYangPassion reviewed Aug 16, 2023
View reviewed changes
hugegraph-api/src/main/java/org/apache/hugegraph/api/filter/AccessLogFilter.java Show resolved Hide resolved
hugegraph-api/src/main/java/org/apache/hugegraph/api/API.java Outdated Show resolved Hide resolved
hugegraph-api/src/main/java/org/apache/hugegraph/api/API.java Outdated Show resolved Hide resolved
hugegraph-api/src/main/java/org/apache/hugegraph/api/API.java Outdated
private static final Meter EXPECTED_ERROR_METER =
MetricsUtil.registerMeter(API.class, "expected-error");
MetricsUtil.registerMeter(API.class, "expected-error");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

hugegraph-api/src/main/java/org/apache/hugegraph/api/API.java Outdated
private static final Meter UNKNOWN_ERROR_METER =
MetricsUtil.registerMeter(API.class, "unknown-error");
MetricsUtil.registerMeter(API.class, "unknown-error");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

hugegraph-api/src/main/java/org/apache/hugegraph/api/API.java Outdated Show resolved Hide resolved
hugegraph-api/src/main/java/org/apache/hugegraph/api/API.java Outdated
@@ -97,7 +95,7 @@ public static <R> R commit(HugeGraph g, Callable<R> callable) {
SUCCEED_METER.mark();
return result;
} catch (IllegalArgumentException | NotFoundException |
ForbiddenException e) {
ForbiddenException e) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not addressed comment

hugegraph-api/src/main/java/org/apache/hugegraph/api/API.java Outdated Show resolved Hide resolved
hugegraph-test/src/main/java/org/apache/hugegraph/api/MetricsApiTest.java Outdated
@@ -165,8 +190,8 @@ public void testMetricsBackend() {
String key = (String) e.getKey();
value = e.getValue();
Assert.assertTrue(String.format(
"Expect map value for key %s but got %s",
key, value),
"Expect map value for key %s but got %s",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

prefer to keep the origin alignment

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not addressed: prefer to keep the origin alignment

@imbajin imbajin changed the title Feat：metric api support promtheus return format & add statistic metric api feat(api): support metric API Prometheus format & add statistic metric api Aug 16, 2023
@imbajin imbajin added this to the 1.5.0 milestone Aug 23, 2023
@imbajin
Copy link
Member

imbajin commented Aug 23, 2023

@SunnyBoy-WYH seems CI failed? could check the CI status in your personal repo (submit a PR to your repo, u could check/test CI there)

@SunnyBoy-WYH
Copy link
Contributor Author

@SunnyBoy-WYH seems CI failed? could check the CI status in your personal repo (submit a PR to your repo, u could check/test CI there)

Nice idea! ( when i push this pr, the local test passed, and a few days before i am looking for the ci problem,but ci info not enough.) i will try it again.

@codecov
Copy link

codecov bot commented Sep 10, 2023 •
edited
Loading

Codecov Report

Merging #2286 (6216976) into master (4ceef1a) will decrease coverage by 3.23%.
The diff coverage is 95.25%.

@@             Coverage Diff              @@
##             master    #2286      +/-   ##
============================================
- Coverage     68.26%   65.04%   -3.23%     
  Complexity      989      989              
============================================
  Files           500      503       +3     
  Lines         41516    41801     +285     
  Branches       5786     5804      +18     
============================================
- Hits          28342    27189    -1153     
- Misses        10415    11920    +1505     
+ Partials       2759     2692      -67
Files Changed Coverage Δ
...pi/src/main/java/org/apache/hugegraph/api/API.java 70.00% <ø> (ø)
...java/org/apache/hugegraph/metrics/MetricsKeys.java 90.90% <90.90%> (ø)
...java/org/apache/hugegraph/metrics/MetricsUtil.java 91.26% <93.81%> (+28.76%) ⬆️
...a/org/apache/hugegraph/api/metrics/MetricsAPI.java 89.79% <95.78%> (+24.00%) ⬆️
...g/apache/hugegraph/api/filter/AccessLogFilter.java 100.00% <100.00%> (ø)
...va/org/apache/hugegraph/api/filter/PathFilter.java 100.00% <100.00%> (ø)

... and 69 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

javeme
javeme reviewed Sep 10, 2023
View reviewed changes
hugegraph-api/src/main/java/org/apache/hugegraph/metrics/MetricsUtil.java Outdated Show resolved Hide resolved
hugegraph-api/src/main/java/org/apache/hugegraph/metrics/MetricsUtil.java Outdated Show resolved Hide resolved
hugegraph-api/src/main/java/org/apache/hugegraph/metrics/MetricsUtil.java Outdated Show resolved Hide resolved
hugegraph-api/src/main/java/org/apache/hugegraph/api/filter/AccessLogFilter.java Outdated Show resolved Hide resolved
hugegraph-api/src/main/java/org/apache/hugegraph/api/filter/AccessLogFilter.java Outdated Show resolved Hide resolved
hugegraph-api/src/main/java/org/apache/hugegraph/api/filter/PathFilter.java Show resolved Hide resolved
hugegraph-api/src/main/java/org/apache/hugegraph/api/filter/AccessLogFilter.java Outdated Show resolved Hide resolved
hugegraph-api/src/main/java/org/apache/hugegraph/api/filter/AccessLogFilter.java Outdated Show resolved Hide resolved
hugegraph-api/src/main/java/org/apache/hugegraph/api/filter/AccessLogFilter.java Outdated Show resolved Hide resolved
hugegraph-api/src/main/java/org/apache/hugegraph/api/filter/PathFilter.java Outdated Show resolved Hide resolved
javeme
javeme reviewed Sep 13, 2023
View reviewed changes
hugegraph-api/src/main/java/org/apache/hugegraph/api/filter/PathFilter.java Outdated Show resolved Hide resolved
hugegraph-api/src/main/java/org/apache/hugegraph/api/filter/PathFilter.java Outdated Show resolved Hide resolved
hugegraph-api/src/main/java/org/apache/hugegraph/api/metrics/MetricsAPI.java Outdated Show resolved Hide resolved
hugegraph-api/src/main/java/org/apache/hugegraph/api/metrics/MetricsAPI.java Outdated Show resolved Hide resolved
hugegraph-api/src/main/java/org/apache/hugegraph/api/metrics/MetricsAPI.java Outdated Show resolved Hide resolved
hugegraph-api/src/main/java/org/apache/hugegraph/api/metrics/MetricsAPI.java Outdated Show resolved Hide resolved
hugegraph-api/src/main/java/org/apache/hugegraph/metrics/MetricsKeys.java Outdated Show resolved Hide resolved
hugegraph-api/src/main/java/org/apache/hugegraph/metrics/MetricsUtil.java Outdated Show resolved Hide resolved
hugegraph-test/src/main/java/org/apache/hugegraph/api/MetricsApiTest.java Outdated Show resolved Hide resolved
javeme
javeme reviewed Sep 16, 2023
View reviewed changes
Copy link
Contributor

@javeme javeme left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there are still some comments to be addressed, thanks

hugegraph-api/src/main/java/org/apache/hugegraph/api/metrics/MetricsAPI.java Outdated Show resolved Hide resolved
hugegraph-api/src/main/java/org/apache/hugegraph/api/metrics/MetricsAPI.java Outdated Show resolved Hide resolved
hugegraph-api/src/main/java/org/apache/hugegraph/metrics/MetricsKeys.java Outdated Show resolved Hide resolved
hugegraph-api/src/main/java/org/apache/hugegraph/metrics/MetricsUtil.java Show resolved Hide resolved
hugegraph-api/src/main/java/org/apache/hugegraph/metrics/MetricsUtil.java Outdated Show resolved Hide resolved
hugegraph-api/src/main/java/org/apache/hugegraph/metrics/MetricsUtil.java Outdated Show resolved Hide resolved
hugegraph-test/src/main/java/org/apache/hugegraph/api/MetricsApiTest.java Outdated Show resolved Hide resolved
hugegraph-test/src/main/java/org/apache/hugegraph/api/MetricsApiTest.java Outdated Show resolved Hide resolved
javeme
javeme reviewed Sep 17, 2023
View reviewed changes
hugegraph-api/src/main/java/org/apache/hugegraph/api/metrics/MetricsAPI.java Outdated Show resolved Hide resolved
hugegraph-api/src/main/java/org/apache/hugegraph/metrics/MetricsUtil.java Outdated Show resolved Hide resolved
hugegraph-api/src/main/java/org/apache/hugegraph/api/API.java Outdated Show resolved Hide resolved
hugegraph-api/src/main/java/org/apache/hugegraph/api/filter/AccessLogFilter.java Outdated Show resolved Hide resolved
hugegraph-api/src/main/java/org/apache/hugegraph/api/metrics/MetricsAPI.java Outdated Show resolved Hide resolved
hugegraph-api/src/main/java/org/apache/hugegraph/api/metrics/MetricsAPI.java Outdated Show resolved Hide resolved
hugegraph-api/src/main/java/org/apache/hugegraph/metrics/MetricsUtil.java Outdated Show resolved Hide resolved
hugegraph-api/src/main/java/org/apache/hugegraph/metrics/MetricsUtil.java Outdated Show resolved Hide resolved
hugegraph-test/src/main/java/org/apache/hugegraph/api/MetricsApiTest.java Outdated
@@ -165,8 +190,8 @@ public void testMetricsBackend() {
String key = (String) e.getKey();
value = e.getValue();
Assert.assertTrue(String.format(
"Expect map value for key %s but got %s",
key, value),
"Expect map value for key %s but got %s",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not addressed: prefer to keep the origin alignment

hugegraph-api/src/main/java/org/apache/hugegraph/api/API.java Outdated
@@ -97,7 +95,7 @@ public static <R> R commit(HugeGraph g, Callable<R> callable) {
SUCCEED_METER.mark();
return result;
} catch (IllegalArgumentException | NotFoundException |
ForbiddenException e) {
ForbiddenException e) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not addressed comment

javeme
javeme approved these changes Sep 22, 2023
View reviewed changes
Copy link
Contributor

@javeme javeme left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@JackyYangPassion JackyYangPassion merged commit fc9bc28 into apache:master Oct 8, 2023
17 of 21 checks passed
@SunnyBoy-WYH SunnyBoy-WYH mentioned this pull request Nov 6, 2023
Merged
VGalaxies pushed a commit to VGalaxies/incubator-hugegraph that referenced this pull request Nov 10, 2023
@SunnyBoy-WYH @VGalaxies
feat(api): support metric API Prometheus format & add statistic metri…
b05215e 
…c api (apache#2286)
imbajin pushed a commit that referenced this pull request Nov 10, 2023
@SunnyBoy-WYH @imbajin
feat(api): support metric API Prometheus format & add statistic metri…
176637c 
…c api (#2286)
@qwtsc qwtsc mentioned this pull request Nov 18, 2023
Open
1 task
@JackyYangPassion JackyYangPassion mentioned this pull request Jul 1, 2024
Closed
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@JackyYangPassion JackyYangPassion JackyYangPassion approved these changes

@javeme javeme javeme approved these changes

Assignees
No one assigned
Labels
None yet
Projects
HugeGraph Tasks
Status: ✅ Done
Milestone
1.5.0
4 participants
@SunnyBoy-WYH @imbajin @javeme @JackyYangPassion