This repository has been archived by the owner on May 10, 2022. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 25
feat: handle scan iteration exceed limit and refactor code #96
Merged
neverchanje
merged 8 commits into
XiaoMi:thrift-0.11.0-inlined
from
hycdong:abnormal_scan
Apr 28, 2020
Merged
feat: handle scan iteration exceed limit and refactor code #96
neverchanje
merged 8 commits into
XiaoMi:thrift-0.11.0-inlined
from
hycdong:abnormal_scan
Apr 28, 2020
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
neverchanje
reviewed
Apr 26, 2020
src/main/java/com/xiaomi/infra/pegasus/client/PegasusScanner.java
Outdated
Show resolved
Hide resolved
src/main/java/com/xiaomi/infra/pegasus/client/PegasusScanner.java
Outdated
Show resolved
Hide resolved
src/main/java/com/xiaomi/infra/pegasus/client/PegasusScanner.java
Outdated
Show resolved
Hide resolved
src/main/java/com/xiaomi/infra/pegasus/client/PegasusTable.java
Outdated
Show resolved
Hide resolved
src/main/java/com/xiaomi/infra/pegasus/client/PegasusTable.java
Outdated
Show resolved
Hide resolved
} else if (response.error | ||
== 1) { // rocksDB error kNotFound, that scan context has been removed | ||
_context = CONTEXT_ID_NOT_EXIST; | ||
_contextId = CONTEXT_ID_NOT_EXIST; | ||
} else if (response.error == 7) { // rocksDB error kIncomplete |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What will happen if the server with scan-limit meets with the client of the previous version? Will the client throw an exception when it accepts rocksdb::Status::kIncomplete
or just continue?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In current version, client will throw an exception when it accepts rocksdb::Status::kIncomplete
immediately and can not get the result of the scanner. In this version, client will also throw an exception but will return scan result.
neverchanje
reviewed
Apr 26, 2020
src/main/java/com/xiaomi/infra/pegasus/client/PegasusTable.java
Outdated
Show resolved
Hide resolved
neverchanje
reviewed
Apr 26, 2020
src/main/java/com/xiaomi/infra/pegasus/client/PegasusTable.java
Outdated
Show resolved
Hide resolved
neverchanje
approved these changes
Apr 28, 2020
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
As issue pegasus issue #486 and pegasus pull request #500 shows, server limit long time rocksdb iteration, this pull request aims to handle it and refactor scan code.
When response.error = 7 means that client got rocksdb incomplete error, old client will throw exception. However, we should return data have already got and then throw exception.
Test
It is hard to construct scan timeout in unit test. As a result, I test this function manually in onebox.
I started onebox with different
iteration_threshold_time
, wrote 10000 kv-pairs in the same hashkey, and testsortKeyCount
,multiget
, scanner interface, all results are as expected.Compatibility
old client -> old server ok
old client -> new server ok
new client -> old server ok
new client -> new server ok
Those situations above have already been tested working well. If old client access new server, it will throw exception immediately when got rocksdb incomplete error and this exception is hard to read, it won't lead any error and compatibility problem.