Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(bulkload): fix rocksDB parameter allow_ingest_behind lose after replica migration #1651

Merged
merged 12 commits into from
Nov 29, 2023

Conversation

ninsmiracle
Copy link
Contributor

What problem does this PR solve?

#1650

What is changed and how does it work?

When reopen a replica(open db with envs args = 0,envs.size = 0),put the last allow_ingest_behind value to the current option.

In current replica open logic,ONLY reopen situation can start with envs.size = 0.

Tests
  • Manual test

@github-actions github-actions bot added the cpp label Oct 19, 2023
src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved
src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved
src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved
src/server/pegasus_server_impl.cpp Outdated Show resolved Hide resolved
src/server/pegasus_server_impl.h Show resolved Hide resolved
src/server/pegasus_server_impl.h Outdated Show resolved Hide resolved
src/server/pegasus_server_impl.cpp Show resolved Hide resolved
acelyc111
acelyc111 previously approved these changes Oct 19, 2023
@acelyc111
Copy link
Member

@ninsmiracle Please fix the build failure, thanks!

@ninsmiracle
Copy link
Contributor Author

@ninsmiracle Please fix the build failure, thanks!

I can not build master branch now on my local system(which is debian 10),however I could build 2.4.1 here. What shoud I do to deal with this problem?

@acelyc111
Copy link
Member

@ninsmiracle Please fix the build failure, thanks!

I can not build master branch now on my local system(which is debian 10),however I could build 2.4.1 here. What shoud I do to deal with this problem?

See the failure details, https://github.com/apache/incubator-pegasus/actions/runs/6574709367/job/17860337517?pr=1651#step:9:789, it's clear. Is there anything forget to push?

[ 92%] Building CXX object src/server/test/CMakeFiles/pegasus_unit_test.dir/__/pegasus_server_impl.cpp.o
/__w/incubator-pegasus/incubator-pegasus/src/server/pegasus_server_impl.cpp: In member function 'void pegasus::server::pegasus_server_impl::reset_rocksdb_options(const rocksdb::ColumnFamilyOptions&, const rocksdb::DBOptions&, const std::map<std::__cxx11::basic_string<char>, std::__cxx11::basic_string<char> >&, rocksdb::ColumnFamilyOptions*, rocksdb::DBOptions*)':
/__w/incubator-pegasus/incubator-pegasus/src/server/pegasus_server_impl.cpp:3152:5: error: 'target_opts' was not declared in this scope; did you mean 'target_cf_opts'?
 3152 |     target_opts->num_levels = base_opts.num_levels;
      |     ^~~~~~~~~~~
      |     target_cf_opts
/__w/incubator-pegasus/incubator-pegasus/src/server/pegasus_server_impl.cpp:3152:31: error: 'base_opts' was not declared in this scope; did you mean 'base_cf_opts'?
 3152 |     target_opts->num_levels = base_opts.num_levels;
      |                               ^~~~~~~~~
      |                               base_cf_opts
At global scope:
cc1plus: note: unrecognized command-line option '-Wno-implicit-float-conversion' may have been intended to silence earlier diagnostics
cc1plus: note: unrecognized command-line option '-Wno-inconsistent-missing-override' may have been intended to silence earlier diagnostics
make[2]: *** [src/server/test/CMakeFiles/pegasus_unit_test.dir/build.make:300: src/server/test/CMakeFiles/pegasus_unit_test.dir/__/pegasus_server_impl.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/Makefile2:2923: src/server/test/CMakeFiles/pegasus_unit_test.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

@ninsmiracle
Copy link
Contributor Author

@ninsmiracle Please fix the build failure, thanks!

I can not build master branch now on my local system(which is debian 10),however I could build 2.4.1 here. What shoud I do to deal with this problem?

See the failure details, https://github.com/apache/incubator-pegasus/actions/runs/6574709367/job/17860337517?pr=1651#step:9:789, it's clear. Is there anything forget to push?

[ 92%] Building CXX object src/server/test/CMakeFiles/pegasus_unit_test.dir/__/pegasus_server_impl.cpp.o
/__w/incubator-pegasus/incubator-pegasus/src/server/pegasus_server_impl.cpp: In member function 'void pegasus::server::pegasus_server_impl::reset_rocksdb_options(const rocksdb::ColumnFamilyOptions&, const rocksdb::DBOptions&, const std::map<std::__cxx11::basic_string<char>, std::__cxx11::basic_string<char> >&, rocksdb::ColumnFamilyOptions*, rocksdb::DBOptions*)':
/__w/incubator-pegasus/incubator-pegasus/src/server/pegasus_server_impl.cpp:3152:5: error: 'target_opts' was not declared in this scope; did you mean 'target_cf_opts'?
 3152 |     target_opts->num_levels = base_opts.num_levels;
      |     ^~~~~~~~~~~
      |     target_cf_opts
/__w/incubator-pegasus/incubator-pegasus/src/server/pegasus_server_impl.cpp:3152:31: error: 'base_opts' was not declared in this scope; did you mean 'base_cf_opts'?
 3152 |     target_opts->num_levels = base_opts.num_levels;
      |                               ^~~~~~~~~
      |                               base_cf_opts
At global scope:
cc1plus: note: unrecognized command-line option '-Wno-implicit-float-conversion' may have been intended to silence earlier diagnostics
cc1plus: note: unrecognized command-line option '-Wno-inconsistent-missing-override' may have been intended to silence earlier diagnostics
make[2]: *** [src/server/test/CMakeFiles/pegasus_unit_test.dir/build.make:300: src/server/test/CMakeFiles/pegasus_unit_test.dir/__/pegasus_server_impl.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/Makefile2:2923: src/server/test/CMakeFiles/pegasus_unit_test.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

Sorry,I misunderstanding the meaning of CI prompt message. I notice the IWYU also failed,and I thought at first it make every jobs failed under this job.
But I'm still can not build this version of Pegasus on my local system,and the error called /fmt/core.h:624:28: error: ‘fmt::v5::basic_format_context<std::back_insert_iterator<fmt::v5::internal::basic_buffer<char> >, char>::formatter_type<dsn::rpc_address>::type’ {aka ‘struct fmt::v5::formatter<dsn::rpc_address>’} has no member named ‘parse’,should I ingore it ?

@acelyc111
Copy link
Member

@ninsmiracle Please fix the build failure, thanks!

I can not build master branch now on my local system(which is debian 10),however I could build 2.4.1 here. What shoud I do to deal with this problem?

See the failure details, https://github.com/apache/incubator-pegasus/actions/runs/6574709367/job/17860337517?pr=1651#step:9:789, it's clear. Is there anything forget to push?

[ 92%] Building CXX object src/server/test/CMakeFiles/pegasus_unit_test.dir/__/pegasus_server_impl.cpp.o
/__w/incubator-pegasus/incubator-pegasus/src/server/pegasus_server_impl.cpp: In member function 'void pegasus::server::pegasus_server_impl::reset_rocksdb_options(const rocksdb::ColumnFamilyOptions&, const rocksdb::DBOptions&, const std::map<std::__cxx11::basic_string<char>, std::__cxx11::basic_string<char> >&, rocksdb::ColumnFamilyOptions*, rocksdb::DBOptions*)':
/__w/incubator-pegasus/incubator-pegasus/src/server/pegasus_server_impl.cpp:3152:5: error: 'target_opts' was not declared in this scope; did you mean 'target_cf_opts'?
 3152 |     target_opts->num_levels = base_opts.num_levels;
      |     ^~~~~~~~~~~
      |     target_cf_opts
/__w/incubator-pegasus/incubator-pegasus/src/server/pegasus_server_impl.cpp:3152:31: error: 'base_opts' was not declared in this scope; did you mean 'base_cf_opts'?
 3152 |     target_opts->num_levels = base_opts.num_levels;
      |                               ^~~~~~~~~
      |                               base_cf_opts
At global scope:
cc1plus: note: unrecognized command-line option '-Wno-implicit-float-conversion' may have been intended to silence earlier diagnostics
cc1plus: note: unrecognized command-line option '-Wno-inconsistent-missing-override' may have been intended to silence earlier diagnostics
make[2]: *** [src/server/test/CMakeFiles/pegasus_unit_test.dir/build.make:300: src/server/test/CMakeFiles/pegasus_unit_test.dir/__/pegasus_server_impl.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/Makefile2:2923: src/server/test/CMakeFiles/pegasus_unit_test.dir/all] Error 2
make: *** [Makefile:136: all] Error 2

Sorry,I misunderstanding the meaning of CI prompt message. I notice the IWYU also failed,and I thought at first it make every jobs failed under this job. But I'm still can not build this version of Pegasus on my local system,and the error called /fmt/core.h:624:28: error: ‘fmt::v5::basic_format_context<std::back_insert_iterator<fmt::v5::internal::basic_buffer<char> >, char>::formatter_type<dsn::rpc_address>::type’ {aka ‘struct fmt::v5::formatter<dsn::rpc_address>’} has no member named ‘parse’,should I ingore it ?

According to the error message fmt::v5::formatter..., it seems the libfmt is too old in your develop enviroment, you can try to resolve it by running ./run.sh build -c --clear_thirdparty.

@empiredan empiredan merged commit 16caf01 into apache:master Nov 29, 2023
73 of 77 checks passed
@acelyc111
Copy link
Member

@ninsmiracle Thanks for the contribution!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants