Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

List the host for pending remote bootstrap #6996

Open
tedyu opened this issue Jan 26, 2021 · 0 comments
Open

List the host for pending remote bootstrap #6996

tedyu opened this issue Jan 26, 2021 · 0 comments

Comments

@tedyu
Copy link
Contributor

tedyu commented Jan 26, 2021

From https://detective-gcp.dev.yugabyte.com/job/github-yugabyte-db-phabricator%2F73645%2Fartifact%2Fbuild%2Fasan-clang-dynamic-ninja%2Fyb-test-logs%2Ftests-client__snapshot-txn-test%2FSnapshotTxnTest_RemoteBootstrapOnStart.log?class=SnapshotTxnTest&max_lines=3000&name=RemoteBootstrapOnStart&start_line=21001 :

21238 |   | F | 50.310855 | 5315 | ts_tablet_manager.cc:1599 | P PEER_B: Waited for 30.000sms. Still had 1 pending remote bootstraps
-- | -- | -- | -- | -- | -- | --
21239 | Fatal failure details written to /nfusr/centos-gcp-cloud/jenkins-worker-4b0/jenkins/jenkins-github-yugabyte-db-phabricator-73645/build/asan-clang-dynamic-ninja/yb-test-logs/tests-client__snapshot-txn-test/SnapshotTxnTest_RemoteBootstrapOnStart.fatal_failure_details.2021-01-23T01_39_12.pid5315.txt
21240 | F20210123 01:39:12 ../../src/yb/tserver/ts_tablet_manager.cc:1599] P PEER_B: Waited for 30.000sms. Still had 1 pending remote bootstraps
21241 | @ 0x7f47b08f9c34 yb::LogFatalHandlerSink::send(int, char const*, char const*, int, tm const*, char const*, unsigned long) (src/yb/util/logging.cc:474)
21242 | @ 0x7f47af1fa9c2
21243 | @ 0x7f47af1caf31
21244 | @ 0x7f47af1ceb7c
21245 | @ 0x7f47af1ce15d
21246 | @ 0x7f47c762c2ac yb::tserver::TSTabletManager::StartShutdown() (src/yb/tserver/ts_tablet_manager.cc:1599)
21247 | @ 0x7f47c75494cc yb::tserver::TabletServer::Shutdown() (src/yb/tserver/tablet_server.cc:366)
21248 | @ 0x7f47c74d5d49 yb::tserver::MiniTabletServer::Shutdown() (src/yb/tserver/mini_tablet_server.cc:184)
21249 | @ 0x5642fb yb::client::SnapshotTxnTest::TestRemoteBootstrap() (src/yb/client/snapshot-txn-test.cc:770)
21250 | @ 0x7f47b18d1421 void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20201207195011-bbf28cf4f1-centos7-linuxbrew/src/gmock-1.8.0/googletest/src/gtest.cc:2402)
21251 | @ 0x7f47b18d1421 void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20201207195011-bbf28cf4f1-centos7-linuxbrew/src/gmock-1.8.0/googletest/src/gtest.cc:2438)
21252 | @ 0x7f47b184a3db testing::Test::Run() (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20201207195011-bbf28cf4f1-centos7-linuxbrew/src/gmock-1.8.0/googletest/src/gtest.cc:2474)
21253 | @ 0x7f47b184d189 testing::TestInfo::Run() (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20201207195011-bbf28cf4f1-centos7-linuxbrew/src/gmock-1.8.0/googletest/src/gtest.cc:2656)
21254 | @ 0x7f47b184fc07 testing::TestCase::Run() (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20201207195011-bbf28cf4f1-centos7-linuxbrew/src/gmock-1.8.0/googletest/src/gtest.cc:2774)
21255 | @ 0x7f47b1875817 testing::internal::UnitTestImpl::RunAllTests() (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20201207195011-bbf28cf4f1-centos7-linuxbrew/src/gmock-1.8.0/googletest/src/gtest.cc:4649)
21256 | @ 0x7f47b18da0fc bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20201207195011-bbf28cf4f1-centos7-linuxbrew/src/gmock-1.8.0/googletest/src/gtest.cc:2402)
21257 | @ 0x7f47b18da0fc bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20201207195011-bbf28cf4f1-centos7-linuxbrew/src/gmock-1.8.0/googletest/src/gtest.cc:2438)
21258 | @ 0x7f47b187448b testing::UnitTest::Run() (/opt/yb-build/thirdparty/yugabyte-db-thirdparty-v20201207195011-bbf28cf4f1-centos7-linuxbrew/src/gmock-1.8.0/googletest/src/gtest.cc:4257)
21259 | @ 0x7f47cf0ec92b main (src/yb/util/test_main.cc:104)

Listing the pending remote bootstrap would give us more information in troubleshooting.
@tedyu tedyu changed the title List the pending remote bootstrap List the host for pending remote bootstrap Jan 26, 2021
tedyu added a commit that referenced this issue Jan 28, 2021
@tedyu
[#6996] ybase: List the host for pending remote bootstrap
03b8b67 
Summary:
When pending remote bootstrap assertion is raised, currently there is not much information on which host(s) is pending remote bootstrap:
```
21238 |   | F | 50.310855 | 5315 | ts_tablet_manager.cc:1599 | P PEER_B: Waited for 30.000sms. Still had 1 pending remote bootstraps
-- | -- | -- | -- | -- | -- | --
21239 | Fatal failure details written to /nfusr/centos-gcp-cloud/jenkins-worker-4b0/jenkins/jenkins-github-yugabyte-db-phabricator-73645/build/asan-clang-dynamic-ninja/yb-test-logs/tests-client__snapshot-txn-test/SnapshotTxnTest_RemoteBootstrapOnStart.fatal_failure_details.2021-01-23T01_39_12.pid5315.txt
21240 | F20210123 01:39:12 ../../src/yb/tserver/ts_tablet_manager.cc:1599] P PEER_B: Waited for 30.000sms. Still had 1 pending remote bootstraps
21241 | @ 0x7f47b08f9c34 yb::LogFatalHandlerSink::send(int, char const*, char const*, int, tm const*, char const*, unsigned long) (src/yb/util/logging.cc:474)
21242 | @ 0x7f47af1fa9c2
21243 | @ 0x7f47af1caf31
21244 | @ 0x7f47af1ceb7c
21245 | @ 0x7f47af1ce15d
21246 | @ 0x7f47c762c2ac yb::tserver::TSTabletManager::StartShutdown() (src/yb/tserver/ts_tablet_manager.cc:1599)
21247 | @ 0x7f47c75494cc yb::tserver::TabletServer::Shutdown() (src/yb/tserver/tablet_server.cc:366)
```
This revision adds host information (source address) so that troubleshooting is easier.

Test Plan: Run test suite via Jenkins

Reviewers: bogdan

Reviewed By: bogdan

Subscribers: ybase

Differential Revision: https://phabricator.dev.yugabyte.com/D10454
polarweasel pushed a commit to lizayugabyte/yugabyte-db that referenced this issue Mar 9, 2021
@tedyu
[yugabyte#6996] ybase: List the host for pending remote bootstrap
5fdc789 
Summary:
When pending remote bootstrap assertion is raised, currently there is not much information on which host(s) is pending remote bootstrap:
```
21238 |   | F | 50.310855 | 5315 | ts_tablet_manager.cc:1599 | P PEER_B: Waited for 30.000sms. Still had 1 pending remote bootstraps
-- | -- | -- | -- | -- | -- | --
21239 | Fatal failure details written to /nfusr/centos-gcp-cloud/jenkins-worker-4b0/jenkins/jenkins-github-yugabyte-db-phabricator-73645/build/asan-clang-dynamic-ninja/yb-test-logs/tests-client__snapshot-txn-test/SnapshotTxnTest_RemoteBootstrapOnStart.fatal_failure_details.2021-01-23T01_39_12.pid5315.txt
21240 | F20210123 01:39:12 ../../src/yb/tserver/ts_tablet_manager.cc:1599] P PEER_B: Waited for 30.000sms. Still had 1 pending remote bootstraps
21241 | @ 0x7f47b08f9c34 yb::LogFatalHandlerSink::send(int, char const*, char const*, int, tm const*, char const*, unsigned long) (src/yb/util/logging.cc:474)
21242 | @ 0x7f47af1fa9c2
21243 | @ 0x7f47af1caf31
21244 | @ 0x7f47af1ceb7c
21245 | @ 0x7f47af1ce15d
21246 | @ 0x7f47c762c2ac yb::tserver::TSTabletManager::StartShutdown() (src/yb/tserver/ts_tablet_manager.cc:1599)
21247 | @ 0x7f47c75494cc yb::tserver::TabletServer::Shutdown() (src/yb/tserver/tablet_server.cc:366)
```
This revision adds host information (source address) so that troubleshooting is easier.

Test Plan: Run test suite via Jenkins

Reviewers: bogdan

Reviewed By: bogdan

Subscribers: ybase

Differential Revision: https://phabricator.dev.yugabyte.com/D10454
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@tedyu