Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

count() 查询报错 #37

Closed
fisherinbox opened this issue Aug 27, 2018 · 3 comments
Closed

count() 查询报错 #37

fisherinbox opened this issue Aug 27, 2018 · 3 comments
Labels

Comments

@fisherinbox
Copy link

千万级别的点数据,以hbase作为存储,使用 g.V().count()查询会报超时,改了超时配置之后还是会报hbase相关的错误
java.io.InterruptedIOException: null
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:214) ~[hbase-client-2.0.0.jar:2.0.0]
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58) ~[hbase-client-2.0.0.jar:2.0.0]
at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithoutRetries(RpcRetryingCallerImpl.java:192) ~[hbase-client-2.0.0.jar:2.0.0]
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:269) ~[hbase-client-2.0.0.jar:2.0.0]
at org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:437) ~[hbase-client-2.0.0.jar:2.0.0]
at org.apache.hadoop.hbase.client.ClientScanner.nextWithSyncCache(ClientScanner.java:312) ~[hbase-client-2.0.0.jar:2.0.0]
at org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:597) ~[hbase-client-2.0.0.jar:2.0.0]
at org.apache.hadoop.hbase.client.ResultScanner$1.hasNext(ResultScanner.java:53) ~[hbase-client-2.0.0.jar:2.0.0]
at com.baidu.hugegraph.backend.store.hbase.HbaseSessions$RowIterator.hasNext(HbaseSessions.java:474) ~[hugegraph-hbase-0.7.4.jar:?]
at com.baidu.hugegraph.backend.serializer.BinaryEntryIterator.fetch(BinaryEntryIterator.java:71) ~[hugegraph-core-0.7.4.jar:0.7.4.0]
at com.baidu.hugegraph.backend.store.BackendEntryIterator.hasNext(BackendEntryIterator.java:55) ~[hugegraph-core-0.7.4.jar:0.7.4.0]
at com.baidu.hugegraph.iterator.ExtendableIterator.fetch(ExtendableIterator.java:89) ~[hugegraph-common-1.4.9.jar:1.4.9.0]
at com.baidu.hugegraph.iterator.WrappedIterator.hasNext(WrappedIterator.java:41) ~[hugegraph-common-1.4.9.jar:1.4.9.0]
at com.baidu.hugegraph.iterator.MapperIterator.fetch(MapperIterator.java:42) ~[hugegraph-common-1.4.9.jar:1.4.9.0]
at com.baidu.hugegraph.iterator.WrappedIterator.hasNext(WrappedIterator.java:41) ~[hugegraph-common-1.4.9.jar:1.4.9.0]
at com.baidu.hugegraph.iterator.FilterIterator.fetch(FilterIterator.java:42) ~[hugegraph-common-1.4.9.jar:1.4.9.0]

@javeme
Copy link
Contributor

javeme commented Aug 30, 2018

从错误日志来看应该是超时了。
目前大部分NoSQL数据库都无法支持快速的count操作,hugegraph所依赖的后端存储包括Cassandra、HBase、RocksDB等NoSQL数据库,所以count操作实际上需要遍历整张表,一般来说数据量大的时候容易导致超时。
我们也在考虑使用其它的方法来解决这个问题,目前还没有一个比较完美的方案。如果有任何好的想法欢迎与我们分享。

@javeme
Copy link
Contributor

javeme commented Sep 16, 2020

@fisherinbox 目前已优化了count语句,千万级别可以尝试直接同步执行(可适当调大rest和gremlin超时),或者直接使用异步gremlin任务。

调整超时配置(gremlin涉及到3个超时参数,需同时调整):
gremlin-server.yaml文件
scriptEvaluationTimeout

rest-server.properties文件
restserver.request_timeout
gremlinserver.timeout

具体参考:http://hugegraph.baidu.com/config/config-option.html

zhoney pushed a commit to zhoney/hugegraph that referenced this issue Nov 9, 2021
* add computer-dis rest api for call k8s task

* set graph load_from_local_config with true

* upgrade etcd endpoint key name

Co-authored-by: guoyonggang <[email protected]>
@github-actions
Copy link

Due to the lack of activity, the current issue is marked as stale and will be closed after 20 days, any update will remove the stale label

imbajin pushed a commit that referenced this issue Oct 19, 2022
Change-Id: Ib24ede9c20bb2c23a3f06fe72c53be2342295fd4
imbajin pushed a commit that referenced this issue Nov 2, 2022
Change-Id: Ib24ede9c20bb2c23a3f06fe72c53be2342295fd4
imbajin pushed a commit that referenced this issue Nov 7, 2022
Change-Id: Ib24ede9c20bb2c23a3f06fe72c53be2342295fd4
imbajin pushed a commit that referenced this issue Nov 9, 2022
Change-Id: Ib24ede9c20bb2c23a3f06fe72c53be2342295fd4
VGalaxies pushed a commit that referenced this issue Aug 3, 2024
Change-Id: I36b460d1a95b4f063ccd329d2058e4e80ebb71c0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants