Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(online_migration): part3 - support ingestion_behind for pegasus #862

Merged
merged 5 commits into from
Jan 6, 2022

Conversation

hycdong
Copy link
Contributor

@hycdong hycdong commented Dec 31, 2021

As #851 show, we decide to support online migration.

This pull request support it for pegasus, including:

  1. Parse allow_ingest_behind from app->envs, and set rocksdb allow_ingest_behind, reference function start and function update_allow_ingest_behind.
  2. Support ingest_behind for ingestion, reference ingest_files functions.
  3. Update start_bulk_load cmd to support ingest_behind
  4. Add bulk_load_ingest_behind function test

@@ -180,9 +180,12 @@ int rocksdb_wrapper::write_batch_delete(int64_t decree, dsn::string_view raw_key

void rocksdb_wrapper::clear_up_write_batch() { _write_batch->Clear(); }

int rocksdb_wrapper::ingestion_files(int64_t decree, const std::vector<std::string> &sst_file_list)
int rocksdb_wrapper::ingestion_files(int64_t decree,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about name it ingest_files?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

{
const auto &iter = envs.find(ROCKSDB_ALLOW_INGEST_BEHIND);
if (iter == envs.end()) {
_allow_ingest_behind = false;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you can set _db_opts.allow_ingest_behind directly

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Smityz
Smityz previously approved these changes Jan 5, 2022
@hycdong hycdong merged commit fa0ec7a into apache:master Jan 6, 2022
@hycdong hycdong deleted the support_ingest_behind branch January 6, 2022 07:25
acelyc111 pushed a commit that referenced this pull request Jun 23, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants