Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Support record slow-query-log in HugeServer #2325

Open
SunnyBoy-WYH opened this issue Oct 12, 2023 · 3 comments
Open

[Feature] Support record slow-query-log in HugeServer #2325

SunnyBoy-WYH opened this issue Oct 12, 2023 · 3 comments
Labels
feature New feature
Milestone

Comments

@SunnyBoy-WYH
Copy link
Contributor

SunnyBoy-WYH commented Oct 12, 2023

Feature Description (功能描述)

as you know slow query log is important for db,like mysql.
image
record query time, when using api to query graph info,because i think the api total time is the real query time for user.

some questions maybe need discuss @javeme @JackyYangPassion @imbajin @liuxiaocs7 @Radeity @simon824 @z7658329 @VGalaxies

  1. do we need store slow log to file or just print it ? just use slf4j Logger?
  2. hg support gremlin api & params api such as EdgeAPI/VertexAPI, they are different, do we need transform it when we record it( gremlin query language and params are different)?
  3. i think we need set time threold、enabled、file path?like neo4j:

threold:
image

enable:
image

@SunnyBoy-WYH SunnyBoy-WYH added the feature New feature label Oct 12, 2023
@SunnyBoy-WYH SunnyBoy-WYH changed the title [Feature] support hg slow query log record [Feature] support record hugegraph slow query log in server api Oct 12, 2023
@imbajin imbajin changed the title [Feature] support record hugegraph slow query log in server api [Feature] Support record slow-log in HugeServer Oct 13, 2023
@imbajin imbajin added this to the 1.5.0 milestone Oct 13, 2023
@imbajin imbajin moved this to 🆕 New in HugeGraph Tasks Oct 13, 2023
@imbajin imbajin changed the title [Feature] Support record slow-log in HugeServer [Feature] Support record slow-query-log in HugeServer Oct 13, 2023
@SunnyBoy-WYH
Copy link
Contributor Author

SunnyBoy-WYH commented Oct 15, 2023

@javeme like this:
image

@imbajin
Copy link
Member

imbajin commented Oct 16, 2023

  1. do we need store slow log to file or just print it ? just use slf4j Logger?
  2. hg support gremlin api & params api such as EdgeAPI/VertexAPI, they are different, do we need transform it when we record it( gremlin query language and params are different)?
  3. i think we need set time threold、enabled、file path?like neo4j
  1. I think we could try log to specified slow-log-file first, then consider store in DB file next
  2. TBD
  3. Fine to set them
  4. we need more refer info from distributed database like TiDB/HBase/NB to compare the pros & cons

@SunnyBoy-WYH
Copy link
Contributor Author

SunnyBoy-WYH commented Oct 16, 2023

  1. do we need store slow log to file or just print it ? just use slf4j Logger?
  2. hg support gremlin api & params api such as EdgeAPI/VertexAPI, they are different, do we need transform it when we record it( gremlin query language and params are different)?
  3. i think we need set time threold、enabled、file path?like neo4j
  1. I think we could try log to specified slow-log-file first, then consider store in DB file next
  2. TBD
  3. Fine to set them
  4. we need more refer info from distributed database like TiDB/HBase/NB to compare the pros & cons

TIDB :

  1. TiDB 将执行时间超过 tidb_slow_log_threshold(默认值为 300 毫秒)的语句输出到慢查询文件(默认值为 “tidb-slow.log”)。

2.TiDB 默认启用慢查询日志。您可以通过修改系统变量tidb_enable_slow_log来启用或禁用该功能。

record items(intro see ref https://docs.pingcap.com/tidb/stable/identify-slow-queries):
image

HBase:(http://svn.apache.org/repos/asf/hbase/hbase.apache.org/trunk/0.94/book/ops.monitoring.html)
HBase 慢查询日志由可解析的 JSON 结构组成,这些结构描述那些客户端操作(获取、放置、删除等)的属性,这些操作要么运行时间过长,要么生成过多的输出。“运行时间太长”和“输出太多”的阈值是可配置的:

1.hbase.ipc.warn.response.time在不记录查询的情况下可以运行查询的最大毫秒数。默认值为 10000 或 10 秒。可以设置为 -1 以按时间禁用日志记录。
2.hbase.ipc.warn.response.size查询在不记录的情况下可以返回的响应的最大字节大小。默认为 100 MB。可以设置为 -1 以按大小禁用日志记录。

示例:
{
"tables":{
"riley2":{
"puts":[
{
"totalColumns":11,
"families":{
"actions":[
{
"timestamp":1315501284459,
"qualifier":"0",
"vlen":9667580
},
{
"timestamp":1315501284459,
"qualifier":"1",
"vlen":10122412
},
{
"timestamp":1315501284459,
"qualifier":"2",
"vlen":11104617
},
{
"timestamp":1315501284459,
"qualifier":"3",
"vlen":13430635
}
]
},
"row":"cfcd208495d565ef66e7dff9f98764da:0"
}
],
"families":[
"actions"
]
}
},
"processingtimems":956,
"client":"10.47.34.63:33623",
"starttimems":1315501284456,
"queuetimems":0,
"totalPuts":1,
"class":"HRegionServer",
"responsesize":0,
"method":"multiPut"
}

and like this, i use a java pojo transfer to json. like hbase

image

2023-10-17 01:45:49 [grizzly-http-server-20] [INFO] o.a.h.a.f.AccessLogFilter - slow query log: {"executeTime":12,"rawQuery":"{"gremlin":"g.V()","bindings":{},"language":"gremlin-groovy","aliases":{"g":"__g_hugegraph"}}","method":"POST","threshold":0,"path":"gremlin"}

2023-10-17 01:45:49 [grizzly-http-server-21] [INFO] o.a.h.a.f.AccessLogFilter - slow query log: {"executeTime":1,"rawQuery":"{}","method":"GET","threshold":0,"path":"graphs/hugegraph/graph/edges"}

2023-10-17 01:45:49 [grizzly-http-server-22] [INFO] o.a.h.a.f.AccessLogFilter - slow query log: {"executeTime":1,"rawQuery":"{}","method":"GET","threshold":0,"path":"graphs/hugegraph/graph/vertices"}

@imbajin imbajin moved this from 🆕 New to 🏗 In progress in HugeGraph Tasks Apr 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature New feature
Projects
Status: 🏗 In progress
Development

No branches or pull requests

2 participants