Support for single-primary, multi-secondary instances #4899

riversand963 · 2019-01-18T18:51:19Z

This PR allows RocksDB to run in single-primary, multi-secondary process mode.
The writer is a regular RocksDB (e.g. an DBImpl) instance playing the role of a primary.
Multiple DBImplSecondary processes (secondaries) share the same set of SST files, MANIFEST, WAL files with the primary. Secondaries tail the MANIFEST of the primary and apply updates to their own in-memory state of the file system, e.g. VersionStorageInfo.

This PR has several components:

(Originally in Add PathNotFound subcode to IOError #4745). Add a PathNotFound subcode to IOError to denote the failure when a secondary tries to open a file which has been deleted by the primary.
(Similar to Support non-blocking, retryable record reading #4602). Add FragmentBufferedReader to handle partially-read, trailing record at the end of a log from where future read can continue.
(Originally in Add support for MANIFEST tailing #4710 and Add unit test for DBImplSecondary #4820). Add implementation of the secondary, i.e. DBImplSecondary.
3.1 Tail the primary's MANIFEST during recovery.
3.2 Tail the primary's MANIFEST during normal processing by calling ReadAndApply.
3.3 Tailing WAL will be in a future PR.
Add an example in 'examples/multi_processes_example.cc' to demonstrate the usage of secondary RocksDB instance in a multi-process setting. Instructions to run the example can be found at the beginning of the source code.

Test plan
A general check of all tests.

$make clean && make -j32 all
$make check

Specifically, if you are interested in testing log reader related changes, run the following

$./log_test

To test the basic functionality of DBImplSecondary, run the following

./db_secondary_test

All tests must pass. It's also highly recommended to repeat the tests with ASAN.

db/db_impl_secondary.cc

db/version_set.cc

db/log_reader.cc

db/version_set.cc

siying

It's awesome to see this complicated feature to close to the final line!

facebook-github-bot · 2019-03-23T00:37:49Z

@riversand963 has updated the pull request. Re-import the pull request

facebook-github-bot

@riversand963 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot · 2019-03-25T20:31:40Z

@riversand963 has updated the pull request. Re-import the pull request

facebook-github-bot

@riversand963 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

include/rocksdb/db.h

facebook-github-bot · 2019-03-26T22:18:06Z

@riversand963 has updated the pull request. Re-import the pull request

facebook-github-bot

@riversand963 has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.

facebook-github-bot · 2019-03-27T00:09:11Z

@riversand963 merged this pull request in 9358178.

rockeet · 2019-03-27T08:09:41Z

as in 3.3: Tailing WAL will be in a future PR.

@riversand963 How long does this future mean? We are eager to using this feature.

riversand963 · 2019-03-28T23:03:55Z

Thanks @rockeet for the interest. We plan to start this next week. Since we also have other items, the new feature can be ready anytime between next month or middle of this year depending on our task prioritization.

rockeet · 2019-03-29T06:37:53Z

Thanks @riversand963, This is a very great feature!

Summary: This PR allows RocksDB to run in single-primary, multi-secondary process mode. The writer is a regular RocksDB (e.g. an `DBImpl`) instance playing the role of a primary. Multiple `DBImplSecondary` processes (secondaries) share the same set of SST files, MANIFEST, WAL files with the primary. Secondaries tail the MANIFEST of the primary and apply updates to their own in-memory state of the file system, e.g. `VersionStorageInfo`. This PR has several components: 1. (Originally in facebook#4745). Add a `PathNotFound` subcode to `IOError` to denote the failure when a secondary tries to open a file which has been deleted by the primary. 2. (Similar to facebook#4602). Add `FragmentBufferedReader` to handle partially-read, trailing record at the end of a log from where future read can continue. 3. (Originally in facebook#4710 and facebook#4820). Add implementation of the secondary, i.e. `DBImplSecondary`. 3.1 Tail the primary's MANIFEST during recovery. 3.2 Tail the primary's MANIFEST during normal processing by calling `ReadAndApply`. 3.3 Tailing WAL will be in a future PR. 4. Add an example in 'examples/multi_processes_example.cc' to demonstrate the usage of secondary RocksDB instance in a multi-process setting. Instructions to run the example can be found at the beginning of the source code. Pull Request resolved: facebook#4899 Differential Revision: D14510945 Pulled By: riversand963 fbshipit-source-id: 4ac1c5693e6012ad23f7b4b42d3c374fecbe8886

Summary: PR #4899 implemented the general framework for RocksDB secondary instances. This PR adds the support for WAL tailing in `OpenAsSecondary`, which means after the `OpenAsSecondary` call, the secondary is now able to see primary's writes that are yet to be flushed. The secondary can see primary's writes in the WAL up to the moment of `OpenAsSecondary` call starts. Differential Revision: D15059905 Pulled By: miasantreble fbshipit-source-id: 44f71f548a30b38179a7940165e138f622de1f10

Summary: This PR allows RocksDB to run in single-primary, multi-secondary process mode. The writer is a regular RocksDB (e.g. an `DBImpl`) instance playing the role of a primary. Multiple `DBImplSecondary` processes (secondaries) share the same set of SST files, MANIFEST, WAL files with the primary. Secondaries tail the MANIFEST of the primary and apply updates to their own in-memory state of the file system, e.g. `VersionStorageInfo`. This PR has several components: 1. (Originally in facebook#4745). Add a `PathNotFound` subcode to `IOError` to denote the failure when a secondary tries to open a file which has been deleted by the primary. 2. (Similar to facebook#4602). Add `FragmentBufferedReader` to handle partially-read, trailing record at the end of a log from where future read can continue. 3. (Originally in facebook#4710 and facebook#4820). Add implementation of the secondary, i.e. `DBImplSecondary`. 3.1 Tail the primary's MANIFEST during recovery. 3.2 Tail the primary's MANIFEST during normal processing by calling `ReadAndApply`. 3.3 Tailing WAL will be in a future PR. 4. Add an example in 'examples/multi_processes_example.cc' to demonstrate the usage of secondary RocksDB instance in a multi-process setting. Instructions to run the example can be found at the beginning of the source code. Pull Request resolved: facebook#4899 Differential Revision: D14510945 Pulled By: riversand963 fbshipit-source-id: 4ac1c5693e6012ad23f7b4b42d3c374fecbe8886

Summary: PR facebook#4899 implemented the general framework for RocksDB secondary instances. This PR adds the support for WAL tailing in `OpenAsSecondary`, which means after the `OpenAsSecondary` call, the secondary is now able to see primary's writes that are yet to be flushed. The secondary can see primary's writes in the WAL up to the moment of `OpenAsSecondary` call starts. Differential Revision: D15059905 Pulled By: miasantreble fbshipit-source-id: 44f71f548a30b38179a7940165e138f622de1f10

riversand963 self-assigned this Jan 18, 2019

riversand963 requested review from siying and miasantreble January 18, 2019 18:51

facebook-github-bot added the CLA Signed label Jan 18, 2019

This was referenced Jan 18, 2019

Add unit test for DBImplSecondary #4820

Closed

Add PathNotFound subcode to IOError #4745

Closed

Add support for MANIFEST tailing #4710

Closed

Support non-blocking, retryable record reading #4602

Closed

riversand963 force-pushed the multiproc branch 2 times, most recently from e923fdb to c00aeb3 Compare January 29, 2019 22:36

riversand963 closed this Jan 29, 2019

riversand963 changed the title ~~Support for single-primary, multi-secondary processes~~ [ABANDONED] Support for single-primary, multi-secondary processes Jan 29, 2019

riversand963 reopened this Jan 29, 2019

riversand963 changed the title ~~[ABANDONED] Support for single-primary, multi-secondary processes~~ Support for single-primary, multi-secondary processes Jan 29, 2019

riversand963 force-pushed the multiproc branch from c00aeb3 to 31fb7e3 Compare February 1, 2019 01:50

siying reviewed Feb 1, 2019

View reviewed changes

db/version_set.cc Outdated Show resolved Hide resolved

db/log_reader.cc Show resolved Hide resolved

db/version_set.cc Outdated Show resolved Hide resolved

siying reviewed Feb 1, 2019

View reviewed changes

riversand963 force-pushed the multiproc branch from 31fb7e3 to c494c57 Compare February 2, 2019 00:06

riversand963 force-pushed the multiproc branch 4 times, most recently from 3a0cd70 to 6d1864a Compare February 20, 2019 17:34

riversand963 force-pushed the multiproc branch 6 times, most recently from 95abeec to eaa0001 Compare March 4, 2019 05:05

riversand963 requested a review from siying March 4, 2019 21:57

riversand963 added 2 commits March 22, 2019 17:19

Refactor unit test for db secondary

164a57d

Add comments to explain the usage of new API

a24be88

facebook-github-bot reviewed Mar 23, 2019

View reviewed changes

riversand963 added 2 commits March 24, 2019 19:50

Allow secondary to open a strict subset of column families

79d773b

Add an example

d21071d

facebook-github-bot reviewed Mar 25, 2019

View reviewed changes

miasantreble reviewed Mar 26, 2019

View reviewed changes

include/rocksdb/db.h Show resolved Hide resolved

miasantreble reviewed Mar 26, 2019

View reviewed changes

include/rocksdb/db.h Outdated Show resolved Hide resolved

Add more comments

d46f9c5

facebook-github-bot reviewed Mar 26, 2019

View reviewed changes

facebook-github-bot closed this in 9358178 Mar 26, 2019

facebook-github-bot added the Merged label Mar 27, 2019

riversand963 deleted the multiproc branch March 27, 2019 00:26

miasantreble mentioned this pull request Apr 8, 2019

secondary instance: add support for WAL tailing on OpenAsSecondary #5161

Closed

ajkr mentioned this pull request Apr 16, 2019

[wip] roachtest: use check-store in roachtest cockroachdb/cockroach#36758

Closed

ryoqun mentioned this pull request May 24, 2020

Support opening an in-use rocksdb as secondary solana-labs/solana#10209

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support for single-primary, multi-secondary instances #4899

Support for single-primary, multi-secondary instances #4899

riversand963 commented Jan 18, 2019 •

edited

Loading

siying left a comment

facebook-github-bot commented Mar 23, 2019

facebook-github-bot left a comment

facebook-github-bot commented Mar 25, 2019

facebook-github-bot left a comment

facebook-github-bot commented Mar 26, 2019

facebook-github-bot left a comment

facebook-github-bot commented Mar 27, 2019

rockeet commented Mar 27, 2019 •

edited

Loading

riversand963 commented Mar 28, 2019

rockeet commented Mar 29, 2019

Support for single-primary, multi-secondary instances #4899

Support for single-primary, multi-secondary instances #4899

Conversation

riversand963 commented Jan 18, 2019 • edited Loading

siying left a comment

Choose a reason for hiding this comment

facebook-github-bot commented Mar 23, 2019

facebook-github-bot left a comment

Choose a reason for hiding this comment

facebook-github-bot commented Mar 25, 2019

facebook-github-bot left a comment

Choose a reason for hiding this comment

facebook-github-bot commented Mar 26, 2019

facebook-github-bot left a comment

Choose a reason for hiding this comment

facebook-github-bot commented Mar 27, 2019

rockeet commented Mar 27, 2019 • edited Loading

riversand963 commented Mar 28, 2019

rockeet commented Mar 29, 2019

riversand963 commented Jan 18, 2019 •

edited

Loading

rockeet commented Mar 27, 2019 •

edited

Loading