Skip to content
This repository has been archived by the owner on May 10, 2022. It is now read-only.

feat: handle scan iteration exceed limit and refactor code #96

Merged
merged 8 commits into from
Apr 28, 2020

Conversation

hycdong
Copy link
Contributor

@hycdong hycdong commented Mar 23, 2020

As issue pegasus issue #486 and pegasus pull request #500 shows, server limit long time rocksdb iteration, this pull request aims to handle it and refactor scan code.

When response.error = 7 means that client got rocksdb incomplete error, old client will throw exception. However, we should return data have already got and then throw exception.

Test
It is hard to construct scan timeout in unit test. As a result, I test this function manually in onebox.
I started onebox with different iteration_threshold_time, wrote 10000 kv-pairs in the same hashkey, and test sortKeyCount, multiget, scanner interface, all results are as expected.

Compatibility

old client -> old server ok
old client -> new server ok
new client -> old server ok
new client -> new server ok

Those situations above have already been tested working well. If old client access new server, it will throw exception immediately when got rocksdb incomplete error and this exception is hard to read, it won't lead any error and compatibility problem.

} else if (response.error
== 1) { // rocksDB error kNotFound, that scan context has been removed
_context = CONTEXT_ID_NOT_EXIST;
_contextId = CONTEXT_ID_NOT_EXIST;
} else if (response.error == 7) { // rocksDB error kIncomplete

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What will happen if the server with scan-limit meets with the client of the previous version? Will the client throw an exception when it accepts rocksdb::Status::kIncomplete or just continue?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In current version, client will throw an exception when it accepts rocksdb::Status::kIncomplete immediately and can not get the result of the scanner. In this version, client will also throw an exception but will return scan result.

@neverchanje neverchanje merged commit 2d65cc6 into XiaoMi:thrift-0.11.0-inlined Apr 28, 2020
@hycdong hycdong deleted the abnormal_scan branch April 30, 2020 02:57
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants